A distributed data processing method and system

Abstract

Embodiments of the present application relate to the technical field of computer, and particularly relate to a distributed data processing method and system, which are used to reduce the amount of data transmitted to the total node and reduce the processing pressure of the total node. Embodiments of the present application include: a sub-node generates record data according to a processing flow of the sub-node on request data; the sub-node is any sub-node of N sub-nodes of a distributed system; the sub-node performs preliminary pairing verification on the record data to obtain preliminary verification success data and preliminary verification unsuccessful data; the sub-node preliminarily aggregates the preliminary verification success data to obtain preliminary aggregation data; and the sub-node sends the preliminary aggregation data and the preliminary verification unsuccessful data to a total node of the distributed system, so that the total node performs re-pairing verification on the preliminary verification unsuccessful data of the N sub-nodes to obtain re-verification success data, and re-aggregates the re-verification success data and the preliminary aggregation data.

Description

Technical Field

[0001] This invention relates to the field of computer technology in financial technology (Fintech), and more particularly to a distributed data processing method and system. Background Technology

[0002] With the continuous development of fintech, especially internet fintech, more and more technologies (such as distributed systems, blockchain, artificial intelligence, etc.) are being applied in the financial field. However, the financial industry is also placing higher demands on technology, such as the data processing of distributed systems.

[0003] In the current centralized banking core system, all transactions are conducted on a single data processing node, and the various interactive data generated by the transactions reflect the internal processing of the node.

[0004] As business continues to grow, transaction volume expands dramatically. The performance and capacity of a single node are limited, necessitating the deployment of a distributed system within the bank's core system to meet data processing demands. However, in existing distributed systems, individual nodes send their generated data directly to the central node, which then aggregates it. Consequently, the central node receives a massive amount of data, placing significant pressure on it and resulting in lengthy processing times. Summary of the Invention

[0005] This application provides a distributed data processing method and system to reduce the amount of data transmitted to the central node and reduce the processing pressure on the central node.

[0006] An embodiment of the present invention provides a distributed data processing method, comprising:

[0007] The sub-node generates record data according to the sub-node's processing flow for request data; the sub-node is any one of the N sub-nodes in the distributed system, where N≥1;

[0008] The sub-nodes perform preliminary pairwise verification on the recorded data to obtain data that has been preliminarily verified successfully and data that has not been preliminarily verified successfully.

[0009] The sub-nodes perform preliminary aggregation of the initial verification success data to obtain preliminary aggregated data;

[0010] The sub-nodes send the preliminary aggregated data and the preliminary unsuccessful verification data to the master node of the distributed system, so that the master node performs pairwise verification on the preliminary unsuccessful verification data of the N sub-nodes to obtain the re-verified successful data, and then aggregates the re-verified successful data with the preliminary aggregated data again.

[0011] In an optional embodiment, after the sub-node generates record data based on its processing of the request data, and before the sub-node performs preliminary pairwise verification on the record data in the sub-node, the method further includes:

[0012] For each record, the sub-node parses the record data to determine the configuration information of the record data;

[0013] The sub-node determines the data type of the recorded data based on the relationship between configuration information and data type;

[0014] The distributed data processing method further includes:

[0015] The sub-node sends the data type of the recorded data to the main node.

[0016] In one optional embodiment, the sub-node performs preliminary pairwise verification on the recorded data to obtain data that has been preliminarily verified successfully and data that has been preliminarily verified unsuccessfully, including:

[0017] The sub-node determines whether there is a second record data that is paired with the first record data in all the record data of the sub-node; the first record data is any record data in all the record data.

[0018] If it exists, the sub-node will determine the first record data and the second record data as the preliminary verification success data;

[0019] If it does not exist, the sub-node will determine the first record data as the data from the initial verification that failed.

[0020] In one optional embodiment, the configuration information of the recorded data includes a request identifier corresponding to the recorded data;

[0021] After the sub-node generates record data based on its processing of the request data, and before the sub-node performs preliminary pairwise verification on the record data in the sub-node, the process further includes:

[0022] The sub-nodes group the record data according to the request identifier corresponding to the record data;

[0023] The sub-node performs preliminary pairwise verification on the recorded data in the sub-node, including:

[0024] For any group, the sub-node performs preliminary pairwise verification on the record data in the group;

[0025] The sub-nodes will perform a preliminary summary of the initial successful verification data, including:

[0026] For any given group, the sub-node performs a preliminary summary of the initial successful verification data in that group.

[0027] In an optional embodiment, the sub-node performs a preliminary aggregation of the initial verification success data to obtain preliminary aggregated data, further comprising:

[0028] The sub-nodes perform preliminary aggregation of the initial verification success data at a predetermined frequency.

[0029] A distributed data processing system includes: a master node and N branch nodes, where N≥1;

[0030] The sub-node is used to generate record data according to the processing flow of the request data of the sub-node; the sub-node is any one of the N sub-nodes; the record data is initially paired and verified to obtain initially verified successful data and initially verified unsuccessful data; the initially verified successful data is initially summarized to obtain initially summarized data; the initially summarized data and the initially verified unsuccessful data are sent to the main node;

[0031] The main node is used to receive the preliminary summary data and the preliminary verification failure data sent by N sub-nodes; to perform pairwise verification again on the preliminary verification failure data of the N sub-nodes to obtain the verification success data, and to summarize the verification success data again with the preliminary summary data.

[0032] In an optional embodiment, the sub-node is further configured to:

[0033] For each record, the record data is parsed to determine the configuration information of the record data;

[0034] The data type of the recorded data is determined based on the relationship between the configuration information and the data type.

[0035] The data type of the recorded data is sent to the master node.

[0036] In one optional embodiment, the sub-node is specifically used for:

[0037] Determine whether there exists a second record data that is paired with the first record data among all the record data of the sub-node; the first record data is any record data among all the record data.

[0038] If it exists, the first record data and the second record data are determined as the preliminary verification success data;

[0039] If it does not exist, the first recorded data is determined to be the data from the initial verification that failed.

[0040] In one optional embodiment, the configuration information of the recorded data includes a request identifier corresponding to the recorded data;

[0041] The sub-node is also used for:

[0042] The record data is grouped according to the request identifier corresponding to the record data;

[0043] For any given group, perform a preliminary pairwise verification on the recorded data within that group;

[0044] For any given group, perform a preliminary summary of the initial successful verification data within that group.

[0045] In an optional embodiment, the sub-node is further configured to:

[0046] The preliminary verification success data will be initially summarized according to the predetermined frequency.

[0047] This invention also provides an electronic device, comprising:

[0048] At least one processor; and,

[0049] A memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0051] This invention also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the method described above.

[0052] In this embodiment of the invention, the sub-nodes in the distributed system generate record data according to their own processing flow for request data, and preprocess the record data in each sub-node. Specifically, the sub-nodes perform preliminary pairwise verification on the record data to obtain preliminary verification successful data and preliminary verification unsuccessful data. The sub-nodes perform preliminary aggregation on the preliminary verification successful data to obtain preliminary aggregated data. Then, each sub-node sends the preliminary aggregated data and the preliminary verification unsuccessful data to the master node. The master node receives the data from each sub-node in the system, performs pairwise verification again on the preliminary verification unsuccessful data to obtain second-verification successful data, and aggregates the second-verification successful data with the preliminary aggregated data again. Since the request data in the distributed system generates cross-node record data, each sub-node can first verify and perform preliminary aggregation on the record data generated during the execution of request data involving only its own node, and then send the processed data and cross-node data to the master node. Thus, the master node does not need to process the processed data, but only the preliminary verification unsuccessful data, which greatly reduces the data processing load of the master node. On the other hand, the data volume of each sub-node is relatively small, and each sub-node can process data simultaneously, thus reducing the overall data processing time. In addition, the branch nodes perform preliminary aggregation of some data before sending it to the main node. The amount of data after preliminary aggregation is greatly reduced, thereby reducing the amount of data transmitted between the branch nodes and the main node and alleviating the communication pressure. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of a possible system architecture provided in an embodiment of the present invention;

[0055] Figure 2 This is a flowchart illustrating a distributed data processing method provided in an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of the structure of a distributed data processing device provided in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0059] like Figure 1 As shown, an embodiment of the present invention provides a system architecture including a main node 101 and N sub-nodes 102. The structure and parameters of any two sub-nodes are identical; therefore, each sub-node can perform the same function.

[0060] The master node 101 and / or the branch node 102 can be network devices such as computers, and can be a single device or a server cluster formed by multiple servers. Preferably, the master node 101 and / or the branch node 102 can use cloud computing technology for information processing.

[0061] The main node 101 can communicate with the sub-node 102 via the Internet, or via mobile communication systems such as GSM and LTE. The main node 101 can also remotely call the sub-node 102 using RMB, or via other methods such as RPC.

[0062] Specifically, sub-node 102 is used to generate record data according to the processing flow of request data; perform preliminary pairwise verification on the record data to obtain preliminary verification successful data and preliminary verification unsuccessful data; perform preliminary aggregation on the preliminary verification successful data to obtain preliminary aggregation data; and send the preliminary aggregation data and preliminary verification unsuccessful data to the main node.

[0063] The main node 101 is used to receive the preliminary summary data and the preliminary verification failure data sent by N sub-nodes; to perform pairwise verification on the preliminary verification failure data of N sub-nodes to obtain the verification success data, and to summarize the verification success data with the preliminary summary data again.

[0064] For ease of understanding, the terms that may be involved in the embodiments of the present invention are defined and explained below.

[0065] Mini-ledger: A subsystem responsible for processing accounting entries for core accounts. All DCNs (Data Service Nodes) synchronize accounting records to this system.

[0066] Accounting Transaction Records: Accounting transaction records generated by transactions, including key information dimensions such as date, accounting scope, institution, currency, accounting entity type, and balance attributes, are generated in each DCN of the distributed system and then aggregated in the small general ledger system.

[0067] Accrual Flow: The accrual of interest payable generated by each DCN at the end of the day. It is a type of accounting flow for accrual of interest receivable.

[0068] Total and Sub-stream Transaction Flow: This is an accounting transaction flow that summarizes the daily closing balances of all customer accounts and internal accounts within the current DCN according to their respective accounts during the daily batch processing.

[0069] Transaction matching: A business transaction generates paired accounting entries. These paired entries share the same primary transaction record but differ in their debit and credit directions. Due to the distributed system, these paired entries may be generated across different DCNs, but they will eventually be synchronized to the general ledger system. To prevent data loss, the process of reconciling paired debit and credit entries in the general ledger, based on the primary transaction record, is called transaction matching. If one side of the transaction is matched, it indicates a failure to reconcile, suggesting a possible anomaly in the transaction's accounting entry generation or a loss of data during the synchronization process.

[0070] Based on the above architecture, embodiments of the present invention provide a cross-node data processing method, such as... Figure 2 As shown, the cross-node data processing method provided in this embodiment of the invention includes the following steps:

[0071] Step 201: The sub-node generates record data according to the sub-node's processing flow for the requested data; the sub-node is any one of the N sub-nodes in the distributed system, where N≥1.

[0072] In the specific implementation process, after receiving the request data sent by the external system, the sub-node processes the request data. After the processing is completed, record data is generated. The sub-node stores the record data and sends the record data to the master node so that the master node can summarize the record data of all sub-nodes.

[0073] Step 202: The sub-nodes perform preliminary pairwise verification on the recorded data to obtain data that has been successfully verified and data that has not been successfully verified.

[0074] In practice, a request for data will generate a pair of record data. The pair of record data has the same main request identifier, but the processing direction is opposite. To prevent data loss, it is necessary to find the pair of record data for verification. If a pair of record data can be found, the verification is successful; if only a single record data is found, the verification is considered unsuccessful, and an anomaly or data loss during the synchronization process may occur.

[0075] Step 203: The sub-nodes perform preliminary aggregation of the preliminary verification success data to obtain preliminary aggregated data.

[0076] Specifically, the sub-nodes aggregate the initially verified successful data, which can be aggregated based on different dimensions, and then synchronize the aggregated total data to the master node. This effectively reduces the amount of data sent to the master node, and the master node only needs to aggregate the data again based on the initially aggregated data, thereby reducing the data processing pressure on the master node.

[0077] Step 204: The sub-nodes send the preliminary summary data and the preliminary verification failure data to the master node of the distributed system, so that the master node performs pairwise verification on the preliminary verification failure data of the N sub-nodes to obtain the verification success data again, and summarizes the verification success data again with the preliminary summary data.

[0078] In this embodiment of the invention, the sub-nodes in the distributed system generate record data according to their own processing flow for request data, and preprocess the record data in each sub-node. Specifically, the sub-nodes perform preliminary pairwise verification on the record data to obtain preliminary verification successful data and preliminary verification unsuccessful data. The sub-nodes perform preliminary aggregation on the preliminary verification successful data to obtain preliminary aggregated data. Then, each sub-node sends the preliminary aggregated data and the preliminary verification unsuccessful data to the master node. The master node receives the data from each sub-node in the system, performs pairwise verification again on the preliminary verification unsuccessful data to obtain second-verification successful data, and aggregates the second-verification successful data with the preliminary aggregated data again. Since the request data in the distributed system generates cross-node record data, each sub-node can first verify and perform preliminary aggregation on the record data generated during the execution of request data involving only its own node, and then send the processed data and cross-node data to the master node. Thus, the master node does not need to process the processed data, but only the preliminary verification unsuccessful data, which greatly reduces the data processing load of the master node. On the other hand, the data volume of each sub-node is relatively small, and each sub-node can process data simultaneously, thus reducing the overall data processing time. In addition, the branch nodes perform preliminary aggregation of some data before sending it to the main node. The amount of data after preliminary aggregation is greatly reduced, thereby reducing the amount of data transmitted between the branch nodes and the main node and alleviating the communication pressure.

[0079] The following section details the data processing process in the real-time example of this invention by setting the bank's core system as a distributed system. For ease of understanding, the technical terms used in the banking system will be consistently matched with the terms mentioned above. Specifically, DNC stands for Sub-Node; Sub-ledger Node is the Header Node; Transaction Processing Request is the requested data; Transaction-corresponding Accounting Flow is the recorded data, which includes accounting flow, accrual flow, and total / subtotal flow. Since accounting flow involves customer accounts and involves a massive amount of data, matching and summarizing operations are performed on accounting flow, while parsing operations are performed on all three types of flow. For ease of description, all are referred to as accounting flow; Flow matching operation is a pairwise verification; Flow type is a data type, including asset, liability, and internal account accounting types.

[0080] For accounting records generated from transactions, the following states are defined: 0 - Registered, 1 - Parsed, 2 - Matched, 3 - Summarized, 4 - Preliminary Matching and Summarizing. In existing technologies, accounting records synchronized from the DNC to the small ledger node are all in the 0 - Registered state. The small ledger node of the core system performs parsing, matching, and summarizing operations on accounting records; therefore, in existing technologies, the states of accounting records include 0 - Registered, 1 - Parsed, 2 - Matched, and 3 - Summarized. In this embodiment of the invention, to alleviate the pressure on the small ledger node, preliminary matching and summarizing operations are performed in the DNC; therefore, the 4 - Preliminary Matching and Summarizing state is added.

[0081] Furthermore, the parsing of accounting records can also be performed in DNC. After the sub-node generates record data based on its processing of the requested data, and before the sub-node performs preliminary pairwise verification of the record data in the sub-node, the process also includes:

[0082] For each record, the sub-node parses the record data to determine the configuration information of the record data;

[0083] The sub-node determines the data type of the recorded data based on the relationship between configuration information and data type;

[0084] The method further includes:

[0085] The sub-node sends the data type of the recorded data to the main node.

[0086] In practice, the recorded data may contain multiple data types. The main node needs to perform different subsequent operations on the recorded data based on the different data types, thus requiring the parsing of the recorded data data types. In this embodiment of the invention, the data type parsing process is also completed in the sub-nodes, and then the sub-nodes synchronize the parsing results to the main node. For example, the parsing of accounting records is divided into three types according to different accounting standards: asset type, liability type, and internal account accounting type. Three configuration tables are designed to set the corresponding account numbers for each accounting standard, with the following names:

[0087] gl_liab_accting_def -- Definition of Liability Accounting

[0088] gl_ia_accting_def -- Internal account accounting definition

[0089] gl_asset_accting_def -- Asset accounting definition

[0090] The three configuration tables above contain static data of the configuration parameter type, which doesn't change much. To enable the DCN to perform accounting subject parsing when generating paired accounting transactions, corresponding backup data for these three tables is set up in each DCN. The account number parsing process is as follows: For each accounting transaction, the records in the three configuration tables are matched based on the accounting scope, accounting entity type, and balance attribute on the accounting transaction to determine the account number of the accounting ledger to which the accounting transaction belongs. A new account number field, gl_code, is added to the accounting transaction register table gl_accting_detl to store the parsed account number. Thus, when the DNC synchronizes the accounting transactions to the small ledger node, it can write the parsing result along with other fields of the accounting transactions into a synchronization file and send it to the small ledger node. The small ledger node does not need to perform transaction parsing operations after receiving the accounting transactions.

[0091] In existing technologies, the accounting transaction status synchronized from the DCN to the small ledger node is 0 - registered. After parsing the accounting transaction, the small ledger node changes the status to 1 - parsed. In this embodiment of the invention, the DCN has already completed the parsing operation for the accounting transaction. Therefore, when synchronized to the small ledger node, the accounting transaction status is already 1 - parsed, and the small ledger node does not need to perform a parsing operation. Thus, the parsing operation previously concentrated in the small ledger node is distributed to various DCNs for processing. Since the number of DCNs in a distributed system is large and can be continuously increased as business changes occur, and parsing operations can be performed simultaneously, distributing the operations in the small ledger node to various DCNs can reduce the processing pressure on the small ledger node, save overall processing time, and improve efficiency.

[0092] Furthermore, the sub-nodes perform preliminary pairwise verification on the recorded data to obtain data that has been preliminarily verified successfully and data that has been preliminarily verified unsuccessfully, including:

[0093] The sub-node determines whether there is a second record data that is paired with the first record data in all the record data of the sub-node; the first record data is any record data in all the record data.

[0094] If it exists, the sub-node will determine the first record data and the second record data as the preliminary verification success data;

[0095] If it does not exist, the sub-node will determine the first record data as the data from the initial verification that failed.

[0096] In practical implementation within a distributed system, if the sub-processes requesting data are processed on the same node, their corresponding record data is stored on that node. Conversely, if the sub-processes requesting data are processed on different nodes, their corresponding record data is stored on separate nodes. Therefore, when a node performs initial pairwise verification on the record data, it obtains initially successful and initially unsuccessful verification data. Initially successful verification data indicates that the sub-processes requesting data were processed on the same node, while initially unsuccessful verification data may be due to anomalies or the data being stored on a different node. Therefore, the node does not process the initially unsuccessful verification data; instead, it aggregates the data to the central node, which then performs further verification.

[0097] To cope with the surge in data volume, the nodes adopt a grouping approach to store and process the record data. The configuration information of the record data includes the request identifier corresponding to that record data.

[0098] After the sub-node generates record data based on its processing of the request data, and before the sub-node performs preliminary pairwise verification on the record data in the sub-node, the process further includes:

[0099] The sub-nodes group the record data according to the request identifier corresponding to the record data.

[0100] In practical implementation, it is recommended that each DCN have a set of sub-tables with the exact same structure as the accounting transaction log table gl_accting_detl. Each sub-table corresponds to a group of accounting transactions. For the accounting transaction log, the sub-table name can be: gl_accting_detl_part_$, where the $ symbol is an integer variable, ranging from 1 to N. The number of sub-tables can be determined based on the scale of the business data; the larger the business data volume, the more sub-tables can be set. In practice, a system parameter GL_ACCTING_SUM can be configured in the system to determine the number of sub-tables. After the number of sub-tables is determined, when accounting transactions are generated in the DCN, the following algorithm can be used to determine which sub-table the current accounting transaction should be stored in.

[0101] Each accounting transaction's configuration information includes the transaction number of the corresponding main transaction. The grouping of the accounting transaction can be determined based on the main transaction transaction number. For example, performing a hash calculation on the main transaction transaction number yields the hash value of the main transaction transaction. Then, performing a modulo operation on the hash value of the main transaction transaction with the number of shards, and finally adding 1 to the modulo result, gives the corresponding shard number, i.e., $.

[0102] Since the master transaction number of a transaction request is the same, the resulting paired accounting transactions also carry the same master transaction number. The accounting transactions for the same transaction request within the same DCN will enter the same sub-table. However, in cross-DCN scenarios, accounting transactions in different DCNs may be distributed across sub-tables on different DCNs.

[0103] In this way, when the amount of data on a single node is large, the main table stored in the database is divided into multiple sub-tables, which effectively distributes the data for processing, improves the data insertion performance, and provides better conditions for subsequent data processing.

[0104] Based on the processing of recorded data grouping, the sub-node performs preliminary pairwise verification on the recorded data in the sub-node, including:

[0105] For any group, the sub-node performs preliminary pairwise verification on the recorded data in the group.

[0106] In practice, accounting transaction data on a single DCN can be distributed across multiple sub-tables, resulting in significantly less data volume per sub-table compared to a single table centralized in the accounting transaction register. For each sub-table, a scheduled task performs matching processing on the accounting transactions. This process involves finding the main transaction field in the "unmatched" (1-parsed) state, searching for accounting transaction records with the same main transaction field across the entire sub-table, and checking if two accounting transactions have the same transaction amount (with debits and credits reversed). If a match is found, these two accounting transactions are marked as matched.

[0107] Specifically, if a transaction request occurs within an account in a single DCN and there are no cross-DCN transactions, meaning the accounting flow under the same transaction request is processed in the same DCN, then after the generated accounting flow enters the sharded table, the data matching process for the sharded table does not actually need to match each main transaction flow. Instead, it only needs to check whether the debits and credits of the entire sharded table are balanced, which can reduce the computational load of the nodes.

[0108] The sub-nodes will perform a preliminary summary of the initial successful verification data, including:

[0109] For any given group, the sub-node performs a preliminary summary of the initial successful verification data in that group.

[0110] In the specific processing, the table structure of the partition table `gl_accting_detl_part_$` and the accounting transaction log `gl_accting_detl` is exactly the same. Key dimensions on the tables include: transaction institution, currency, transaction date, accounting entity type, balance attribute, transaction amount, debit / credit flags, on- and off-balance sheet flags, and supplementary accounting subjects. The synchronization of accounting transactions to the sub-ledger node is based on the data in the accounting transaction log `gl_accting_detl` table. That is, the data in the newly added partition table `gl_accting_detl_part_$` needs to be added to the accounting transaction log `gl_accting_detl` table before it will be synchronized to the sub-ledger node. Data in the partition table `gl_accting_detl_part_$` can be summarized and then added to the accounting transaction log `gl_accting_detl` table; this process can also be called data compression. Specifically, the matched accounting transactions in the sub-tables can be summarized according to dimensions such as transaction institution, currency, transaction date, on- or off-balance sheet status, account number, and debit / credit status. The preliminary summary results are then written into the accounting transaction register gl_accting_detl and synchronized to the general ledger.

[0111] Furthermore, the sub-tables can also adopt a different structure than the accounting transaction log table gl_accting_detl. For example, one sub-table can correspond to one key dimension. For any sub-table, all data related to that key dimension in the sub-nodes is stored in that sub-table. A summary result can be obtained by summarizing one sub-table. Then, each summary result is written to the corresponding key dimension in the accounting transaction log table gl_accting_detl.

[0112] Furthermore, before the sub-node performs preliminary pairwise verification on the record data corresponding to the requested data, it also includes:

[0113] The sub-node acquires the recorded data in the sub-node at a predetermined frequency.

[0114] In practice, a flexibly configurable scheduled task can be set up on each sub-table. This task summarizes the matched accounting transaction data from the sub-table according to different dimensions and then writes it to the accounting transaction register gl_accting_detl. The scheduling frequency of the scheduled task can be preset, for example, set to perform a summary operation on the sub-table once per hour, or to perform a summary operation on the sub-table at the end of the day. The scheduling frequency can be set to the same or different frequencies for different sub-tables; this embodiment of the invention does not impose any restrictions.

[0115] After aggregation, the accounting transactions synchronized to the small ledger node are in state 4 - Preliminary Matching and Aggregation. Some accounting transactions are initially verified as successful but have not undergone aggregation; their state is 2 - Matched. These transactions do not require further matching on the small ledger node and are directly aggregated with other transactions in state 2 - Matched, resulting in accounting transactions in state 3 - Aggregated. Additionally, the small ledger node may contain accounting transactions that failed to match, possibly due to cross-node issues or other reasons, preventing matching on the DCN. These are synchronized to the small ledger node in state 1 - Parsed. After all DCN accounting transactions are synchronized to the small ledger node, the small ledger node performs matching again on the unmatched accounting transactions. The specific implementation is similar to the above and will not be repeated.

[0116] To better understand the present invention, the above process is described in detail below with reference to specific embodiments. The specific steps are as follows:

[0117] Step S301: DCN processes transaction requests and generates accounting records.

[0118] Step S302: For each accounting transaction, DCN performs a parsing operation, determines the account number of the accounting transaction based on the obtained configuration parameters, and writes the account number into the accounting transaction register. After completion, the status of the accounting transaction is changed to 1-parsed.

[0119] Step S303: DCN determines the number of sub-tables and assigns each accounting transaction to the corresponding sub-table based on the main transaction number.

[0120] Step S304: DCN performs matching processing on the accounting transactions in each sub-table. For successfully matched accounting transactions, the status is changed to 2-Matched, while the status of unmatched accounting transactions remains 1-Resolved.

[0121] Step S305: DCN performs a summary operation on the accounting transactions in each sub-table with a status of 2-Matched according to the set frequency, writes the summary results into the register table of accounting transactions, and sets the status of the summary results in the register table of accounting transactions to 4-Preliminary Matching Summary.

[0122] Step S306: DCN also writes accounting transactions with status 1-parsed and 2-matched into the accounting transaction register.

[0123] Step S307: DCN synchronizes the accounting transaction log to the sub-ledger node.

[0124] Step S308: The minor ledger node performs a matching operation on accounting transactions with status 1 - parsed. If successful, it performs further aggregation with the accounting transactions with status 4 - preliminary matching and aggregation. It also performs an aggregation operation on accounting transactions with status 2 - matched, which means it performs further aggregation with the accounting transactions with status 4 - preliminary matching and aggregation. After aggregation, the status is changed to 3 - aggregated.

[0125] This invention also provides a distributed data processing device, such as... Figure 3 As shown, it includes:

[0126] The generation unit 31 is used to generate record data according to the processing flow of the request data by the sub-node; the sub-node is any one of the N sub-nodes in the distributed system.

[0127] The verification unit 32 is used to perform preliminary pairwise verification on the recorded data to obtain data that has been successfully verified and data that has not been successfully verified.

[0128] The summarization unit 33 is used to initially summarize the preliminary verification success data to obtain preliminary summary data;

[0129] The sending unit 34 is used to send the preliminary summary data and the preliminary verification failure data to the main node of the distributed system, so that the main node performs pairwise verification on the preliminary verification failure data of the N sub-nodes to obtain the verification success data again, and summarizes the verification success data again with the preliminary summary data.

[0130] Optionally, a parsing unit 35 is also included, used for:

[0131] For each record, the record data is parsed to determine the configuration information of the record data;

[0132] The data type of the recorded data is determined based on the relationship between the configuration information and the data type.

[0133] The sending unit is also used to send the data type of the recorded data to the master node.

[0134] Optionally, the verification unit 32 is specifically used for:

[0135] Determine whether there exists a second record data that is paired with the first record data among all the record data of the sub-node; the first record data is any record data among all the record data.

[0136] If it exists, the first record data and the second record data are determined as the preliminary verification success data;

[0137] If it does not exist, the first recorded data is determined to be the data from the initial verification that failed.

[0138] Optionally, the configuration information of the recorded data includes a request identifier corresponding to the recorded data;

[0139] The verification unit 32 is further configured to:

[0140] The record data is grouped according to the request identifier corresponding to the record data;

[0141] For any given group, perform a preliminary pairwise verification on the recorded data within that group;

[0142] The summarization unit 33 is also used to perform a preliminary summary of the preliminary verification success data in any group.

[0143] Optionally, the summarizing unit 33 is further configured to:

[0144] The preliminary verification success data will be initially summarized according to the predetermined frequency.

[0145] Based on the same principle, the present invention also provides an electronic device, such as... Figure 4 As shown, it includes:

[0146] It includes a processor 401, a memory 402, a transceiver 403, and a bus interface 404, wherein the processor 401, the memory 402, and the transceiver 403 are connected through the bus interface 404.

[0147] The processor 401 is used to read the program in the memory 402 and execute the following methods:

[0148] Based on the processing flow of the request data by the sub-node, record data is generated; the sub-node is any one of the N sub-nodes in the distributed system.

[0149] Perform preliminary pairwise verification on the recorded data to obtain data that has been successfully verified and data that has not been successfully verified.

[0150] The initial verification success data is initially summarized to obtain preliminary summary data;

[0151] The preliminary summary data and the preliminary unsuccessful verification data are sent to the main node of the distributed system, so that the main node performs pairwise verification on the preliminary unsuccessful verification data of the N sub-nodes to obtain the re-verified successful verification data, and then summarizes the re-verified successful verification data with the preliminary summary data again.

[0152] The processor 401 is also used for:

[0153] For each record, the record data is parsed to determine the configuration information of the record data;

[0154] The data type of the recorded data is determined based on the relationship between the configuration information and the data type.

[0155] Sending the preliminary summary data and the preliminary verification failure data to the master node also includes:

[0156] The preliminary summary data, the preliminary verification failure data, and the data type of the recorded data are sent to the master node.

[0157] The processor 401 is also used for:

[0158] After the branch node processes the request data and generates record data, before the branch node performs preliminary pairwise verification on the record data in the branch node, the process further includes:

[0159] The record data is grouped according to the request identifier corresponding to the record data;

[0160] For any given group, perform a preliminary pairwise verification on the recorded data within that group;

[0161] For any group, the initial successful verification data in the group is initially summarized; the configuration information of the recorded data includes the request identifier corresponding to the recorded data.

[0162] The processor 401 is also used for:

[0163] The preliminary verification success data will be initially summarized according to the predetermined frequency.

[0164] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0168] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A distributed data processing method, characterized in that, include: The sub-nodes generate record data according to the sub-node's processing flow for request data; The sub-node is any one of the N sub-nodes in the distributed system. A request for data generates a pair of record data. The pair of record data corresponds to the same main request identifier and is processed in opposite directions. ; The sub-node determines whether there is a second record data that is paired with the first record data in all the record data of the sub-node, where the first record data is any record data in all the record data; if it exists, the sub-node determines the first record data and the second record data as the preliminary verification success data; If it does not exist, the sub-node will determine the first record data as data that failed the initial verification; The sub-nodes perform preliminary aggregation of the initial verification success data to obtain preliminary aggregated data; The sub-nodes send the preliminary aggregated data and the preliminary unsuccessful verification data to the master node of the distributed system, so that the master node performs pairwise verification on the preliminary unsuccessful verification data of the N sub-nodes to obtain the re-verified successful data, and then aggregates the re-verified successful data with the preliminary aggregated data again.

2. The method as described in claim 1, characterized in that, After the sub-node generates record data based on its processing of the request data, and before the sub-node performs preliminary pairwise verification on the record data in the sub-node, the process further includes: For each record, the sub-node parses the record data to determine the configuration information of the record data; The sub-node determines the data type of the recorded data based on the relationship between configuration information and data type; The distributed data processing method further includes: The sub-node sends the data type of the recorded data to the main node.

3. The method as described in claim 2, characterized in that, The configuration information of the recorded data includes the request identifier corresponding to the recorded data; After the sub-node generates record data based on its processing of the request data, and before the sub-node performs preliminary pairwise verification on the record data in the sub-node, the process further includes: The sub-node groups the record data according to the request identifier corresponding to the record data; The sub-node performs preliminary pairwise verification on the recorded data in the sub-node, including: For any group, the sub-node performs preliminary pairwise verification on the record data in the group; The sub-nodes will perform a preliminary summary of the initial successful verification data, including: For any given group, the sub-node performs a preliminary summary of the initial successful verification data in that group.

4. The method as described in claim 1, characterized in that, The sub-node performs a preliminary summary of the initial verification success data to obtain preliminary summary data, which also includes: The sub-nodes perform preliminary aggregation of the initial verification success data at a predetermined frequency.

5. A distributed data processing system, characterized in that, include: The main node and N branch nodes, where N≥1; The sub-node is used to generate record data according to the sub-node's processing flow of request data; The sub-node is any one of the N sub-nodes. A request data generates a pair of record data. The main request identifiers corresponding to the pair of record data are the same, and the processing directions are opposite. The record data is initially paired and verified to obtain data that has been initially verified successfully and data that has not been initially verified successfully. The data that initially succeeded in the verification are initially summarized to obtain preliminary summary data; the preliminary summary data and the data that initially failed the verification are sent to the master node. The main node is used to receive the preliminary summary data and the preliminary verification failure data sent by N sub-nodes; The data that failed the initial verification of the N sub-nodes are re-verified in pairs to obtain the data that succeeded in the second verification, and the data that succeeded in the second verification is then summarized again with the initial summary data.

6. The system as described in claim 5, characterized in that, The sub-node is also used for: For each record, the record data is parsed to determine the configuration information of the record data; The data type of the recorded data is determined based on the relationship between the configuration information and the data type. The data type of the recorded data is sent to the master node.

7. The system as described in claim 6, characterized in that, The configuration information of the recorded data includes the request identifier corresponding to the recorded data; The sub-node is also used for: The record data is grouped according to the request identifier corresponding to the record data; For any given group, perform a preliminary pairwise verification on the recorded data within that group; For any given group, perform a preliminary summary of the initial successful verification data within that group.

8. The system as described in claim 5, characterized in that, The sub-node is also used for: The preliminary verification success data will be initially summarized according to the predetermined frequency.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-4.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-4.

Images