Methods, systems, terminal devices, storage media, and products for distributed parallel historical data archiving

Abstract

This invention provides a method, system, terminal device, storage medium, and product for distributed parallel historical data archiving, relating to the field of computer systems. This invention allows for dynamic configuration of the number of days data is retained, flexibly and effectively controlling the amount of data in the data tables. It employs a publish-subscribe model to decouple the functions of each role in the cluster, enabling the system to achieve distributed parallel processing, rapid expansion, and high availability. Different data tables are triggered for data backup and cleanup based on the business's final state time. While ensuring the correctness of business data and preventing order omissions, cleanup is only triggered when all related data tables for the same order have reached the business's final state, avoiding one-sided data issues.

Description

Technical Field

[0001] This invention relates to the field of computer systems, and more specifically, to a method, system, terminal device, storage medium, and product for distributed parallel historical data archiving. Background Technology

[0002] During peak business hours, the clearing platform needs to complete the reconciliation and clearing of millions of orders within three hours. In order to complete the reconciliation and clearing of this number of orders in the shortest possible time and ensure that merchants' T-1 trading day orders are cleared and transferred every morning, various methods are used to improve the system's processing speed from different angles.

[0003] A distributed data archiving system is an auxiliary system designed to enable faster reconciliation and settlement systems. The purpose of this system is to minimize the number of orders stored in the dedicated reconciliation and settlement table, thereby improving the speed of reconciliation and settlement by reducing the amount of existing data.

[0004] Problems that exist without a data archiving system:

[0005] 1. As the number of merchants using the government-funded payment system increases and daily transaction volume grows, historical orders accumulate rapidly. This leads to a significant increase in the data volume in the database's main order reconciliation and settlement tables, sub-order reconciliation and settlement tables, and settlement detail reconciliation and settlement tables. The daily reconciliation and settlement process performed by the clearing platform is extremely slow, resulting in merchants not receiving yesterday's transaction amounts in a timely manner. This severely impacts the user experience for connected customers.

[0006] 2. A dedicated table is used for reconciliation and settlement, separate from the actual business data table. Order reconciliation and settlement transfer information and results are stored in this dedicated table and are not synchronized with the actual business data table, resulting in incomplete data in the actual business data table.

[0007] Data archiving systems are designed to solve the above problems. Summary of the Invention

[0008] This invention provides a method, system, terminal device, storage medium, and product for distributed parallel historical data archiving.

[0009] A method for distributed parallel historical data archiving, the specific steps of which include data backup, data synchronization and data cleaning;

[0010] Data backup, this step includes task publishing, task consumption, and task input parameters;

[0011] The task publishing process refers to the control center scanning the acquiring statement download, parsing, clearing, and transfer control table every 5 minutes for merchants whose data synchronization status is "pending" and whose operation date is the current day. The control center then publishes the data backup and synchronization task for these merchants to the task scheduling center and updates the data synchronization status of these merchants to "in progress." While publishing tasks, the control center checks the number of concurrent tasks already in progress; if more than 200 (configurable) tasks are already running, no new tasks will be published.

[0012] The term "task consumption" refers to the fact that six servers were registered as task consumers in the production environment.

[0013] The task input parameters refer to inputting the merchant number and date information into the control center;

[0014] Data synchronization includes master order data processing, sub-order data processing, order settlement details processing, and downloading, parsing, and updating the settlement and transfer control table of the acquiring statement.

[0015] The master order data processing includes two steps. The first step is to back up the master order reconciliation and settlement data in batches. Orders in the master order reconciliation and settlement data table whose reconciliation status is balanced and whose reconciliation date is equal to the task input date are backed up in batches to the master order reconciliation and settlement history table, and the backup status in the master order reconciliation and settlement data table is updated to "backed up". The second step is to synchronize the master order reconciliation status in batches. Specifically, the orders in the master order reconciliation and settlement data table that have been backed up in the first step are updated in batches to update the reconciliation status and reconciliation-related information in the master order business table, and the synchronization status in the master order reconciliation and settlement data table is updated to "synchronized".

[0016] The sub-order data processing includes synchronizing sub-orders that need to be settled and synchronizing sub-orders that do not need to be settled. Synchronizing sub-orders that need to be settled involves two steps. The first step is to back up the sub-order reconciliation and settlement data in batches. Specifically, orders in the sub-order reconciliation and settlement data table with a settlement status of "settled" (business final state) and a settlement date equal to the task input parameter date are backed up in batches to the sub-order reconciliation and settlement history table, and the backup status in the sub-order reconciliation and settlement data table is updated to "backed up". The second step is to synchronize the sub-order reconciliation status and settlement status in batches. Specifically, for the sub-orders that have been backed up in the first step, the payment system reconciliation status, settlement status, and reconciliation and settlement related information in the sub-order business table are updated in batches, and the synchronization status in the sub-order reconciliation and settlement data table is updated to "synchronized".

[0017] Synchronizing sub-orders that do not require settlement involves two steps. The first step is to back up the sub-order reconciliation and settlement data in batches. This step backs up orders in the sub-order reconciliation and settlement history table in batches, where the settlement status in the sub-order reconciliation and settlement table is "no settlement required" and the payment system reconciliation status is "no reconciliation required" or "reconciled (final state of business)," and the reconciliation date is equal to the date of the input parameter in the task. At the same time, the backup status in the sub-order reconciliation and settlement table is updated to "backed up." The second step is to synchronize the sub-order reconciliation and settlement status in batches. This step updates the payment system reconciliation status, settlement status, and reconciliation and settlement related information in the sub-order business table in batches for the sub-orders that have been backed up in the first step in the sub-order reconciliation and settlement table, and updates the synchronization status in the sub-order reconciliation and settlement table to "synchronized."

[0018] The order settlement details processing includes two steps. The first step involves backing up the order settlement details reconciliation and settlement dedicated table data in batches. Orders in the order settlement details reconciliation and settlement dedicated table with a settlement status of "settled and transferred" and a settlement transfer date equal to the task input parameter date are backed up in batches to the order settlement details reconciliation and settlement history table. The backup status in the order settlement details reconciliation and settlement dedicated table is then updated to "backed up."

[0019] The second step is to synchronize the order settlement details and settlement status in batches. This involves updating the payment system reconciliation status, settlement status, and related information in the sub-order business table in batches for the sub-orders that were backed up in the first step of the sub-order reconciliation and settlement dedicated table. The synchronization status in the sub-order reconciliation and settlement dedicated table will then be updated to "synchronized."

[0020] The update of the acquiring statement download, parsing, clearing and transfer control table specifically updates the data synchronization status of the corresponding merchant transaction day in the acquiring statement download, parsing, clearing and transfer control table to "executed".

[0021] Data cleanup is a step that involves obtaining the configured number of days to retain data and calculating the deletion date.

[0022] Furthermore: a distributed parallel historical data archiving system, including a controller module and a consumer module;

[0023] The controller module is the control center, which is responsible for the creation and release of archiving tasks and the scheduling of consumer nodes;

[0024] The consumer module is responsible for executing archiving tasks;

[0025] The controller module also includes a data backup module, a data synchronization module, and a data cleanup module. The data backup module is used for task publishing, task consumption, and task input parameters.

[0026] The data synchronization module is used for processing master order data, sub-order data, order settlement details, and downloading, parsing, and updating the settlement and transfer control table of the acquiring statement.

[0027] The data cleanup module is used to obtain the configured number of days to retain data and calculate the deletion date.

[0028] Furthermore: a computer program product comprising a computer program that, when executed by a processor, implements the method described above.

[0029] Furthermore, the terminal device may include a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions that can be executed by the processor. When the terminal device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the deep learning model training method as described in the foregoing embodiments.

[0030] Further: a storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described above.

[0031] The beneficial effects of this invention are: This invention allows for dynamic configuration of the number of days data is retained, flexibly and effectively controlling the amount of data stored in the data tables. By adopting a publish-subscribe model, the functions of each role in the cluster are decoupled, enabling the system to achieve distributed parallel processing, rapid expansion, and high availability. Different data tables are triggered for data backup and cleanup based on the business's final state time. This ensures the correctness of business data, prevents missed orders, and ensures that cleanup is only triggered when all related data tables for the same order have reached the business's final state, thus avoiding one-sided data issues. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating the method of the present invention is shown.

[0034] Figure 2 A schematic diagram of the data backup process in the method of the present invention is shown.

[0035] Figure 3 A schematic diagram of the data synchronization process in the method of the present invention is shown.

[0036] Figure 4 A schematic diagram of the system of the present invention is shown.

[0037] Figure 5 A schematic diagram of the composition of the terminal device of the present invention is shown. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in the present invention are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of the present invention. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the present invention. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0039] Furthermore, the embodiments described herein are merely some, not all, of the embodiments of the invention. The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0040] It should be noted that the term "comprising" will be used in the embodiments of the present invention to indicate the presence of a feature subsequently declared, but does not preclude the addition of other features. It should also be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of the present invention, it should also be noted that the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0041] In this scheme, the distributed data archiving system mainly includes two roles: controller and consumer.

[0042] The controller is the control center, responsible for creating and publishing archiving tasks and scheduling consumer nodes, while the consumer is responsible for executing archiving tasks. Archiving tasks are divided into two types: data backup and synchronization, and data cleanup.

[0043] The controller and consumer roles are functionally decoupled through a publish-subscribe pattern, allowing any node in the cluster to temporarily act as either a controller or a consumer. Even if only one server remains in the cluster, it can still function normally; the failure of a single node will not cause the entire cluster to become inoperable. This ensures high system availability and also enables rapid cluster expansion.

[0044] Every day at midnight, the controller scans the merchant configuration table and generates a control record for each merchant in the acquiring reconciliation statement download, parsing, clearing, and transfer control table, with a transaction date of T-1. The download, parsing, clearing, and transfer control table mainly includes the merchant number, transaction date, operation date, download status, parsing status, clearing status, transfer status, reconciliation rules, reconciliation master task number, data synchronization (backup) status, and data cleanup status. Initially, all statuses are pending execution.

[0045] The following is in conjunction with the instruction manual appendix. Figure 1-3 The above archiving tasks will be explained in further detail.

[0046] Regarding data backup and synchronization

[0047] Task Release: Starting at 2 PM daily (the afternoon is chosen to avoid overlap with the reconciliation and settlement tasks executed in the early morning, thus alleviating server and database pressure), the control center scans the merchant reconciliation statements, downloads and parses the settlement and transfer control table, and records merchants whose data synchronization status is "Pending" and whose operation date is the current day. The control center then releases the data backup and synchronization task for these merchants to the task scheduling center (a product developed in-house), and updates the merchant's data synchronization status to "In Progress." While releasing tasks, the control center checks the number of concurrent tasks already in progress; if more than 200 (configurable) tasks are running, no new tasks will be released.

[0048] Task consumption: Six servers were registered as task consumers in the production environment.

[0049] Task input parameters: Merchant ID, Date.

[0050] Master Order Data Processing: The first step is to back up the master order reconciliation and settlement table data in batches. Orders in the master order reconciliation and settlement table with a reconciliation status of "even" (final business state) and a reconciliation date equal to the task input parameter date are backed up in batches to the master order reconciliation and settlement history table. Then, the backup status in the master order reconciliation and settlement table is updated to "backed up."

[0051] The second step is to synchronize the master order receiving reconciliation status in batches. This involves updating the receiving reconciliation status and related information in the master order business table in batches for the orders that were backed up in the first step of the master order reconciliation and settlement special table. Finally, the synchronization status in the master order reconciliation and settlement special table is updated to "synchronized".

[0052] Sub-order data processing:

[0053] (1) Synchronize the sub-orders that need to be cleared.

[0054] The first step is to back up the sub-order reconciliation and settlement data in batches. Orders in the sub-order reconciliation and settlement table with a settlement status of "settled" (final business state) and a settlement date equal to the task input parameter date are backed up in batches to the sub-order reconciliation and settlement history table. Then, update the backup status in the order reconciliation and settlement table to "backed up".

[0055] The second step is to synchronize the reconciliation and settlement status of sub-orders in batches. This involves updating the payment system reconciliation status, settlement status, and related information in the sub-order business table in batches for the sub-orders that were backed up in the first step of the sub-order reconciliation and settlement dedicated table. Finally, the synchronization status in the sub-order reconciliation and settlement dedicated table will be updated to "synchronized."

[0056] (2) Synchronize sub-orders that do not require settlement

[0057] The first step is to back up the sub-order reconciliation and settlement data in batches. Orders in the sub-order reconciliation and settlement table whose settlement status is "no settlement required" and whose payment system reconciliation status is "no reconciliation required" or "reconciled (final state)", and whose reconciliation date equals the date entered in the task, are backed up in batches to the sub-order reconciliation and settlement history table. Then, update the backup status in the sub-order reconciliation and settlement table to "backed up".

[0058] The second step is to synchronize the reconciliation and settlement status of sub-orders in batches. This involves updating the payment system reconciliation status, settlement status, and related information in the sub-order business table in batches for the sub-orders that were backed up in the first step of the sub-order reconciliation and settlement dedicated table. Finally, the synchronization status in the sub-order reconciliation and settlement dedicated table will be updated to "synchronized."

[0059] Order settlement details processing:

[0060] The first step is to back up the order settlement details reconciliation and settlement dedicated table data in batches. Orders in the order settlement details reconciliation and settlement dedicated table whose settlement status is "settled and transferred" (final business state) and whose settlement and transfer date equals the task input parameter date are backed up in batches to the order settlement details reconciliation and settlement history table. Then, update the backup status in the order settlement details reconciliation and settlement dedicated table to "backed up".

[0061] The second step is to synchronize the order settlement details and settlement status in batches. This involves updating the payment system reconciliation status, settlement status, and related information in the sub-order business table in batches for the sub-orders that were backed up in the first step of the sub-order reconciliation and settlement dedicated table. Finally, the synchronization status in the sub-order reconciliation and settlement dedicated table is updated to "synchronized".

[0062] Acquiring statement download, parsing, clearing and transfer control sheet update:

[0063] The data synchronization status for the corresponding merchant's transaction day in the updated acquiring statement download, parsing, clearing, and transfer control table has been updated to "executed".

[0064] Data Cleaning

[0065] Task Release: Starting at 3 PM daily (the afternoon is chosen to avoid overlap with the reconciliation and settlement execution time in the early morning), the control center scans the acquiring reconciliation statements every 5 minutes. Merchants whose data cleanup status is "Pending" and whose operation date is the current day are listed in the clearing and transfer control table. The control center then releases the data cleanup task for these merchants to the task scheduling center (this scheduling component is a self-developed product), updating the merchant's data cleanup status to "In Progress." While releasing tasks, the control center checks the number of concurrent tasks already in progress; if more than 200 (configurable) tasks are already running, no new tasks will be released.

[0066] Task Consumption: Six servers were registered as task consumers in the production environment. Task Input Parameters: Merchant ID, Date. The cleanup task mainly involves the main order reconciliation and settlement table, the sub-order reconciliation and settlement table, and the settlement details reconciliation and settlement table.

[0067] The first step is to obtain the configured number of days to retain data and calculate the deletion date. For example, if the task input date is 20221225 and the configured number of days to retain data is 20, then the calculated deletion date will be 20221205.

[0068] like Figure 4 As shown, a distributed parallel historical data archiving system includes a controller module and a consumer module.

[0069] The controller module is the control center, which is responsible for the creation and release of archiving tasks and the scheduling of consumer nodes;

[0070] The consumer module is responsible for executing archiving tasks;

[0071] The controller module also includes a data backup module, a data synchronization module, and a data cleanup module. The data backup module is used for task publishing, task consumption, and task input parameters.

[0072] The data synchronization module is used for processing master order data, sub-order data, order settlement details, and downloading, parsing, and updating the settlement and transfer control table of the acquiring statement.

[0073] The data cleanup module is used to obtain the configured number of days to retain data and calculate the deletion date.

[0074] like Figure 5As shown, the terminal device 6 may include a processor 601, a storage medium 602, and a bus 603. The storage medium 602 stores machine-readable instructions executable by the processor 601. When the terminal device is running, the processor 601 communicates with the storage medium 602 via the bus 603. The processor 601 executes the machine-readable instructions to perform the steps of the deep learning model training method described in the foregoing embodiments. The specific implementation and technical effects are similar and will not be repeated here.

[0075] For ease of explanation, only one processor is described in the terminal device described above. However, it should be noted that in some embodiments, the terminal device of the present invention may also include multiple processors, and therefore the steps performed by one processor described in the present invention may also be performed jointly by multiple processors or individually.

[0076] For ease of explanation, the present invention also includes a computer program product comprising a computer program that, when executed by a processor, implements the methods described above.

[0077] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for distributed parallel historical data archiving, characterized in that, The specific steps include data backup, data synchronization and data cleaning; Data backup, which includes task publishing, task consumption and task input parameters; Data synchronization, which includes main order data processing, sub-order data processing, order clearing details processing and acquirer reconciliation bill download analysis clearing transfer control table updating; The main order data processing includes two steps. The first step is to backup the main order reconciliation clearing special table data in batches. The orders in the main order reconciliation clearing special table whose acquirer reconciliation state is balanced and whose reconciliation date is equal to the task input parameter date are backed up in batches to the main order reconciliation clearing history table, and the backup state in the main order reconciliation clearing special table is updated to backed up; The second step is to synchronize the main order acquirer reconciliation state in batches. Specifically, the orders that have completed backup in the first step in the main order reconciliation clearing special table are updated in batches to the acquirer reconciliation state, reconciliation related information in the main order business table, and the synchronization state in the main order reconciliation clearing special table is updated to synchronized; The sub-order data processing includes synchronizing the sub-orders that need to be cleared and the sub-orders that do not need to be cleared; The order clearing details processing includes two steps. The first step is to backup the order clearing details reconciliation clearing special table data in batches. The orders in the order clearing details reconciliation clearing special table whose clearing state is cleared and transferred and whose clearing and transfer date is equal to the task input parameter date are backed up in batches to the order clearing details reconciliation clearing history table, and the backup state in the order clearing details reconciliation clearing special table is updated to backed up; The second step is to synchronize the order clearing details clearing state in batches. The sub-orders that have completed backup in the first step in the sub-order reconciliation clearing special table are updated in batches to the payment system reconciliation state, clearing state, reconciliation clearing related information in the sub-order business table, and the synchronization state in the sub-order reconciliation clearing special table is updated to synchronized; The acquirer reconciliation bill download analysis clearing transfer control table updating specifically updates the data synchronization state corresponding to the merchant transaction date in the acquirer reconciliation bill download analysis clearing transfer control table to executed; Data cleaning, which calculates the deletion date by obtaining the configured retention data days.

2. The method of claim 1, wherein, The task publishing refers to the control center scanning the merchants whose data synchronization state in the acquirer reconciliation bill download analysis clearing transfer control table is to be executed and whose operation date is the current day every 5 minutes, publishing the data backup and synchronization tasks of the merchants to the task scheduling center, and updating the data synchronization state of the merchants to executing. The control center publishes tasks while checking the number of concurrent tasks in execution. If the number of concurrent tasks in execution exceeds the set number, the control center will not publish new tasks; the task consumption refers to registering several servers in the production environment as task consumers; and the task input parameters refer to inputting the merchant number and date information to the control center.

3. The method of claim 1, wherein, The synchronization of the sub-order requiring clearing includes two steps. The first step is to backup the sub-order reconciliation clearing special table data in batches. Specifically, the orders in the sub-order reconciliation clearing special table whose clearing state is cleared and whose clearing date is equal to the task input date are backed up in batches to the sub-order reconciliation clearing history table, and the backup state in the order reconciliation clearing special table is updated to be backed up. The second step is to synchronize the sub-order reconciliation state and the clearing state. Specifically, the sub-orders in the sub-order reconciliation clearing special table which have been backed up in the first step are updated in batches in the payment system reconciliation state, the clearing state, the reconciliation clearing related information in the sub-order business table, and the synchronization state in the sub-order reconciliation clearing special table is updated to be synchronized.

4. The method of claim 1, wherein, The synchronization of the sub-order not requiring clearing includes two steps. The first step is to backup the sub-order reconciliation clearing special table data in batches. Specifically, the orders in the sub-order reconciliation clearing special table whose clearing state is not required to be cleared and whose payment system reconciliation state is not required to be reconciled or has been reconciled, and whose reconciliation date is equal to the task input date are backed up in batches to the sub-order reconciliation clearing history table, and the backup state in the sub-order reconciliation clearing special table is updated to be backed up. The second step is to synchronize the sub-order reconciliation state and the clearing state. Specifically, the sub-orders in the sub-order reconciliation clearing special table which have been backed up in the first step are updated in batches in the payment system reconciliation state, the clearing state, the reconciliation clearing related information in the sub-order business table, and the synchronization state in the sub-order reconciliation clearing special table is updated to be synchronized.

5. A system for distributed parallel historical data archiving, characterized by, The system is used to execute the method of any one of claims 1-4, comprising a controller module and a consumer module; The controller module is a control center responsible for creating and publishing archiving tasks and scheduling consumer nodes; The consumer module is responsible for executing archiving tasks; The controller module further comprises a data backup module, a data synchronization module, and a data cleaning module. The data backup module is used for task publishing, task consumption, and task input parameters. The data synchronization module is used for main order data processing, sub-order data processing, order clearing detail processing, and acquirer reconciliation bill download analysis clearing transfer control table updating. The data cleaning module is used to obtain the configured retention data days and calculate the deletion date.

6. A terminal device, comprising: Comprising: A processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the terminal device runs, the processor communicates with the storage medium through the bus. The processor executes the machine-readable instructions to execute the steps of the method of any one of claims 1-4.

7. A storage medium, characterized by The storage medium stores a computer program. When the computer program is executed by the processor, the steps of the method of any one of claims 1-4 are executed.

8. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1-4.

Images