Data synchronization method and device based on distributed database and electronic equipment

Abstract

The invention provides a data synchronization method and device based on a distributed database and electronic equipment, which can be applied to the field of electric digital data processing. The method comprises the following steps: in response to a batch transaction processing request, reading a target batch transaction operation number corresponding to a current to-be-processed batch transaction from a batch transaction operation number queue; determining change data queues respectively corresponding to the L source data nodes; screening N target data updating records of which the transaction operation numbers are smaller than the transaction operation number of the target batch from the L change data queues, wherein the N target data updating records are associated with the M source data tables; based on a streaming merging algorithm, merging the target data updating records associated with the M source data tables in parallel by using M merging threads to obtain M data merging hash tables corresponding to the M source data tables; and synchronizing the merged data of the M source data tables into the target database in parallel by utilizing the M loading threads.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing, and in particular to a data synchronization method, device and electronic equipment based on a distributed database. Background Art

[0002] With the rise and development of distributed transaction database cluster technology, a cluster supports hundreds of nodes to provide external transaction services. In the process of cluster database data synchronization, a unified cluster transaction log is used to merge transactions connected to hundreds of nodes at the same time, which will lead to performance bottlenecks. If each node merges and synchronizes data changes separately, a transaction often involves multiple data nodes, which cannot guarantee the consistency of the transaction and cause data errors. In addition, in high-concurrency and high-throughput scenarios, there will be many repeated transaction operations, resulting in unnecessary synchronization operations, which seriously affects the performance of synchronization. Therefore, how to balance database performance and transaction logic consistency in cluster database data synchronization is an urgent problem to be solved. Summary of the invention

[0003] In view of the above problems, the present invention provides a data synchronization method, device and electronic device based on a distributed database.

[0004] According to a first aspect of the present invention, a data synchronization method based on a distributed database is provided, wherein the distributed database includes a source database and a target database for data synchronization, wherein the source database includes L source data nodes; the method includes: in response to a batch transaction processing request, reading a target batch transaction operation number corresponding to a current batch transaction to be processed from a batch transaction operation number queue; determining a change data queue corresponding to each of the L source data nodes, wherein the change data queue stores multiple groups of data update records generated by the source data node executing transaction operations during the data synchronization process, and the multiple groups of data update records correspond to multiple transaction operation numbers; screening N target data update records whose transaction operation numbers are less than the target batch transaction operation number from the L change data queues, wherein the N target data update records are associated with M source data tables, and the L source data nodes jointly complete the data storage of the M source data tables; based on a streaming merge algorithm, using M merge threads to merge the target data update records associated with each of the M source data tables in parallel, to obtain M data merge hash tables corresponding to the M source data tables, wherein the mth source data table is associated with The target data update record is stored in the data merge hash table corresponding to the mth source data table. Primary key associated merge results, the primary key is derived from the target data update record, each merge result contains at most one merged data, , ; Use M loading threads to synchronize the merged data of M source data tables to the target database in parallel.

[0005] The second aspect of the present invention provides a data synchronization device based on a distributed database, comprising: a reading module, which is used to respond to a batch transaction processing request and read the target batch transaction operation number corresponding to the current batch transaction to be processed from the batch transaction operation number queue; a determination module, which is used to determine the change data queues corresponding to L source data nodes, wherein the change data queues store multiple groups of data update records generated by the source data nodes through the execution of transaction operations during the data synchronization process, and the multiple groups of data update records correspond to multiple transaction operation numbers; a screening module, which is used to screen N target data update records whose transaction operation numbers are less than the target batch transaction operation number from the L change data queues, the N target data update records are associated with M source data tables, and the L source data nodes jointly complete the data storage of the M source data tables; a merging module, which is used to merge the target data update records associated with the M source data tables in parallel based on a streaming merge algorithm using M merging threads to obtain M data merge hash tables corresponding to the M source data tables, wherein the mth source data table is associated with The target data update record is stored in the data merge hash table corresponding to the mth source data table. Primary key associated merge results, the primary key is derived from the target data update record, each merge result contains at most one merged data, , ; A synchronization module is used to synchronize the merged data of M source data tables to the target database in parallel using M loading threads.

[0006] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data synchronization method based on a distributed database.

[0007] According to the data synchronization method, device and electronic device based on a distributed database provided by the present invention, a batch transaction operation number queue is established to periodically read the batch transaction operation number, and the transaction submission boundaries between multiple batch transaction operation numbers in the batch transaction operation number queue are clear. Then, based on the target batch transaction operation number, the batch target data update records associated with L source data nodes are screened out, and batch transaction processing is performed according to the transaction operation number in the target data update record, thereby ensuring the consistency of the intra-batch transaction logic and the inter-batch transaction boundary; a change data queue of each source data node is established to continuously collect data update records on the source data node, and when batch transaction processing is involved, the data is directly read from the change data queue without the need for source data node operation, thereby alleviating the pressure on the source data node and improving the database performance; the change data queues of each L source data node are collected in parallel, the M source data tables are processed in batches in parallel, and the M loading threads are loaded in parallel, thereby ensuring parallel synchronization processing and effectively utilizing the central processing unit resources; in addition, the batch data update records obtained are merged in batches and the M source data tables are synchronized in batches twice, thereby removing redundant transaction operations and synchronization operations, thereby greatly improving the synchronization performance of the distributed database. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0009] Figure 1 The application scenario of the data synchronization method and device based on a distributed database according to an embodiment of the present invention is shown.

[0010] Figure 2 A flow chart of a data synchronization method based on a distributed database according to an embodiment of the present invention is shown.

[0011] Figure 3 A schematic diagram of a distributed database according to an embodiment of the present invention is shown.

[0012] Figure 4 A structural block diagram of a data synchronization device based on a distributed database according to an embodiment of the present invention is shown.

[0013] Figure 5 A block diagram of an electronic device suitable for implementing a data synchronization method based on a distributed database according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0014] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0015] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0016] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0017] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0018] In the process of implementing the present invention, it is found that with the rise and development of distributed transaction database cluster technology, a cluster supports hundreds of nodes to provide transaction services to the outside world. In the process of cluster database data synchronization, a unified cluster transaction log is used to merge transactions connected to hundreds of nodes at the same time, which will result in a performance bottleneck. If each node merges and synchronizes data changes separately, a transaction often involves multiple data nodes, and the consistency of the transaction cannot be guaranteed, resulting in data errors. In addition, in high-concurrency and high-throughput scenarios, there will be many repeated transaction operations, resulting in unnecessary synchronization operations (such as: if a field value of the same record is updated multiple times, only the last synchronization is required), which seriously affects the synchronization performance. Therefore, how to balance database performance and transaction logic consistency in cluster database data synchronization is an urgent problem to be solved.

[0019] In view of this, an embodiment of the present invention provides a data synchronization method, device and electronic device based on a distributed database. The method includes: in response to a batch transaction processing request, reading the target batch transaction operation number corresponding to the current batch transaction to be processed from the batch transaction operation number queue; determining the change data queue corresponding to each of the L source data nodes; screening N target data update records whose transaction operation number is less than the target batch transaction operation number from the L change data queues, and the N target data update records are associated with M source data tables; based on a streaming merge algorithm, using M merge threads in parallel to merge the target data update records associated with each of the M source data tables, to obtain M data merge hash tables corresponding to the M source data tables; using M loading threads in parallel to synchronize the merged data of the M source data tables to the target database.

[0020] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0021] Figure 1 The application scenario of the data synchronization method and device based on a distributed database according to an embodiment of the present invention is shown.

[0022] like Figure 1 As shown, the business system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0024] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0025] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The cluster distributed database is deployed on the cluster server 105. The server 105 includes multiple server nodes that can analyze and process the received user request and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0026] It should be noted that the data synchronization method based on the distributed database provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data synchronization device based on the distributed database provided in the embodiment of the present invention can generally be set in the server 105. The data synchronization method based on the distributed database provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data synchronization device based on the distributed database provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0027] Alternatively, the data synchronization method based on a distributed database provided in the embodiment of the present invention may also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the data synchronization apparatus based on a distributed database provided in the embodiment of the present invention may also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0028] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0029] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0030] Figure 2 A flow chart of a data synchronization method based on a distributed database according to an embodiment of the present invention is shown.

[0031] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0032] In operation S210 , in response to a batch transaction processing request, a target batch transaction operation number corresponding to the current batch transaction to be processed is read from a batch transaction operation number queue.

[0033] Optionally, the distributed database includes a source database and a target database to be synchronized, the source database includes L source data nodes, the source database stores a source data table to be synchronized, and the target database stores a target table mapped to the source data table.

[0034] Optionally, create a batch transaction request. A batch transaction request is a batch data synchronization task. When creating a batch transaction request, configure parameters. For example, the configuration parameters include source database connection parameters (such as uniform resource locator (JavaDatabase Connectivity URL, JDBC URL), connection profile (properites), etc.), target database connection parameters, batch transaction operation number reading interval parameters, etc. After creation, the system saves the configuration parameters and starts the batch transaction request.

[0035] Optionally, a batch transaction operation number queue can be created using the Kafka component. The batch transaction operation number queue is used to store batch transaction operation numbers periodically read from the global transaction manager based on the batch transaction operation number reading interval parameter. In the global transaction manager, transaction operations with transaction operation numbers less than or equal to the batch transaction operation number have been submitted, and data synchronization operations can then be performed.

[0036] Optionally, a plurality of batch transaction operation numbers are stored in the batch transaction operation number queue, and a minimum batch transaction operation number is read from the batch transaction operation number queue based on the response time as a target batch transaction operation number corresponding to the current batch transaction to be processed.

[0037] In operation S220 , a changed data queue corresponding to each of the L source data nodes is determined.

[0038] Optionally, a Kafka component may be used to create a change data queue for each source data node, where L is a positive integer, and L change data queues corresponding to the L source data nodes are determined.

[0039] Optionally, a two-layer producer-consumer model is adopted to form an efficient pipeline operation; at the same time, the introduction of Kafka components effectively improves the reliability and high availability of the queue.

[0040] Optionally, the changed data queue stores multiple sets of data update records generated by the source data node executing transaction operations during the data synchronization process, and the multiple sets of data update records correspond to multiple transaction operation numbers.

[0041] Optionally, the transaction operation represents a transaction operation such as querying, deleting, or modifying a source data table, and the data update record is data change information generated by executing the transaction operation on the source data table. In response to a transaction operation, a globally unique transaction operation number is requested from the global transaction manager.

[0042] Optionally, the data update record generated by the source data node executing the transaction operation includes a transaction operation number corresponding to the transaction operation, and the data update record is stored in a change data queue corresponding to the source data node.

[0043] Optionally, when a transaction operation involves multiple source data nodes, the multiple sets of data update records generated correspond to the same transaction operation number; the multiple sets of data update records generated for multiple transaction operations correspond to multiple transaction operation numbers.

[0044] In operation S230, N target data update records whose transaction operation numbers are smaller than the target batch transaction operation number are screened from the L change data queues.

[0045] Optionally, the source database includes M source data tables, M is a positive integer, the N target data update records are batch data to be synchronized screened based on the target batch transaction operation number, N is a positive integer, and the N target data update records are associated with the M source data tables.

[0046] Optionally, L source data nodes jointly complete the data storage of M source data tables. Multiple source data nodes jointly complete the storage of a source data table. A transaction operation may involve operating on part of the data in the source data table. If part of the data is stored in multiple source data nodes, multiple source data nodes are required to jointly complete the execution of a transaction operation.

[0047] Optionally, target data update records whose transaction operation numbers are smaller than the target batch transaction operation number are read in sequence from each change data queue to obtain N target data update records.

[0048] In operation S240 , based on a streaming merge algorithm, M merge threads are used to merge target data update records associated with the M source data tables in parallel to obtain M data merge hash tables corresponding to the M source data tables.

[0049] Optionally, during the reading process, determining the source data table corresponding to the filtered target data update record may include: determining the associated source data table according to table name information in the target data update record.

[0050] Optionally, based on a streaming merge algorithm, a merge thread merges target data update records associated with a source data table to obtain a data merge hash table corresponding to the source data table.

[0051] Optionally, the data merge hash table is a hash table storage structure. target data update records, each of which contains a primary key. If the same primary key does not exist in the target data update record, ;exist If the same primary key exists in the target data update records, it means that the data record corresponding to this primary key has been operated on multiple times. The sum of multiple target data update records associated with each of the M source data tables is the total number of batch data to be synchronized, N. .

[0052] Optional, traverse in ascending order according to transaction operation number target data update records, merge the target data update records associated with each primary key, and obtain a merge result. Each merge result contains 1 or 0 merged data. The data merge hash table corresponding to the mth source data table stores Primary key associated Merge results.

[0053] Optionally, multiple target data update records associated with a source data table may come from the respective change data queues of multiple source data nodes. Therefore, the multiple target data update records can be sorted based on the order of transaction operation numbers, and then the multiple target data update records corresponding to each primary key can be merged and deduplicated to obtain the merged result for each primary key.

[0054] For example, the merge processing includes: converting the update operation into a delete-then-add operation; for repeated update operations and delete operations, only the last operation is retained; for consecutive operations that can be merged, they are merged. Example 1: a piece of data is inserted, then the field is updated, and finally the data is deleted. After the merge, these three operations will be deleted and regarded as if they have not occurred; Example 2: a field is updated to a value of 1 and then updated to a value of 2. After the merge, only the latter is retained.

[0055] In operation S250 , the merged data of the M source data tables are synchronized to the target database in parallel using M loading threads.

[0056] Optionally, create a dynamic data operation statement (Structured Query Language, SQL) to merge the data of the source data table, assign each merged data variable of the source data table to the dynamic data operation statement according to the transaction operation type, and store it in the operation queue.

[0057] Optionally, the configuration parameters include the single batch submission data volume parameter. Each source data table will start a loading thread to batch read the dynamic data operation statements executed from the operation queue and execute them in the target database to synchronize the merged data to the target table in the target database that is mapped to the source data table. After the operation queue is read, the batch transaction submission operation is executed.

[0058] Optionally, batch processing is used in multiple places to improve performance. When collecting target data update records, processing is performed based on the target batch transaction operation number. In-batch processing is used when analyzing, merging, and deduplicating target data update records. Batch processing is also used when synchronizing merged data to the target database.

[0059] For example, three source data nodes generate a total of multiple data update records. In response to a batch transaction request, the target batch transaction operation number 1000 is obtained, and multiple target data update records with transaction operation numbers less than 1000 are screened from the change data queues of the three source data nodes, and a total of 10,000 target data update records are obtained. Similarly, in response to a batch transaction request, the target batch transaction number 2001 is obtained, and a total of 30,000 target data update records are obtained from one of the three source data nodes. In response to a batch transaction request, the target batch transaction number 3001 is obtained, and a total of 40,000 target data update records are obtained from the three source data nodes. Thus, multiple data update records from multiple source data nodes are synchronized through multiple batches. For each batch, the target batch transaction operation number is used to realize the intra-batch merging of multiple data update records under a controllable number across source data nodes, ensuring that the transactions of multiple source data nodes in a single batch are consistent and consuming fewer resources to realize the intra-batch merging of data update records.

[0060] Optionally, by establishing a batch transaction operation number queue to periodically read the batch transaction operation number, the transaction submission boundaries between multiple batch transaction operation numbers in the batch transaction operation number queue are clear, and then the batch target data update records associated with L source data nodes are screened out based on the target batch transaction operation number, and batch transaction processing is performed according to the transaction operation number in the target data update record, thereby ensuring the consistency of intra-batch transaction logic and inter-batch transaction boundaries; establishing respective change data queues for source data nodes to continuously collect data update records on source data nodes, and directly reading from the change data queue when batch transaction processing is involved, without the need for source data node operation, thereby alleviating the pressure on source data nodes and improving database performance; parallel collection of change data queues for L source data nodes, parallel batch merging of M source data tables, and parallel loading of M loading threads ensure parallel synchronous processing and effective use of central processing unit resources; in addition, two batch processes of intra-batch merging of acquired batch data update records and batch synchronization of M source data tables remove redundant transaction operations and synchronization operations, thereby greatly improving the synchronization performance of distributed databases.

[0061] Optionally, the data synchronization method based on a distributed database also includes: in response to a transaction operation request for each source data table, determining a transaction operation number and i target source data table sub-tables to be operated, wherein the transaction operation number is allocated from a global transaction manager, the source data table includes I source data table sub-tables, i target source data table sub-tables are derived from the I source data table sub-tables, and source data nodes correspond to source data table sub-tables one-to-one, i≤I; performing data change operations on the i target source data table sub-tables to obtain i data update records corresponding to the i target source data table sub-tables, wherein each data update record includes a transaction operation number, a transaction operation type, a target source data table sub-table name, a primary key, and a changed column; synchronously storing the i data update records in i logical replication slots corresponding to the target source data nodes where the i target source data table sub-tables are located; and using i read threads to store the data update records in the i logical replication slots in parallel in i change data queues corresponding to the i target source data nodes.

[0062] Optionally, in response to a transaction operation request for a source data table, the server applies for a transaction operation number from the global transaction manager, and determines i target source data table sub-tables to be operated from the i source data table sub-tables, and then determines the target source data nodes where the i target source data table sub-tables are located.

[0063] Optionally, source data nodes correspond to logical replication slots one by one. A distributed database will have multiple source data nodes for storing and processing transactions. Each source data node will provide a logical replication slot to provide external data change capture services. Change data is read from these logical replication slots in parallel and stored in the Kafka change data queue. Each source data node corresponds to a change data queue.

[0064] Optionally, the target source data node performs a data change operation on the target source data table sub-table to obtain a data update record. The logical replication slot records the data update record on the target source data node. The reading thread may be a logical replication slot reading thread. The logical replication slot reading thread then reads the data update record into a change data queue corresponding to the target source data node.

[0065] For example, for each source data node in the distributed database, a logical replication slot data reading thread is started (such as data thread 1, reading data changes on source data node node1), and after reading the data update record, it is synchronously added to the change data queue node1_db1 of the source data node node1.

[0066] Optionally, the transaction operation type may be an insert type, a delete type, an update type, etc. The change column is a change field in a subtable of the target source data table.

[0067] Optionally, i data update records are synchronously stored in i logical replication slots corresponding to the target source data nodes where the i target source data table sub-tables are located; and i logical replication slot reading threads are used to store the data update records in the i logical replication slots in parallel into i change data queues corresponding to the i target source data nodes.

[0068] Optionally, the logical replication slot reading thread captures data changes of each source data node in the distributed database in parallel, and grows linearly with the increase of source data nodes. Therefore, the performance is not affected by the scale of the distributed database and can match the throughput of the distributed database.

[0069] Optionally, before the merging process, the process further includes: for any m-th source data table, based on the transaction operation number, The target data update records are sorted to obtain the data sorting hash table corresponding to the m-th source data table, wherein the data sorting hash table corresponding to the m-th source data table stores the data corresponding to the m-th source data table. Primary key associated The sorting results corresponding to the hth primary key contain ordered Target data update records, , ; Wherein, for any m-th source data table, the merge processing includes: for the sorting result corresponding to the h-th primary key in the sorted hash table, based on the streaming merge algorithm, using the merge thread to merge the ordered target data update records are merged to obtain the merge result corresponding to the hth primary key; primary keys and the merge results corresponding to each primary key to obtain a data merge hash table corresponding to the mth source data table.

[0070] Optionally, in the process of respectively reading the target data update records from the change data queues corresponding to the source data nodes, the source data table corresponding to the target data update records is determined. A sorting thread is started for each source data table, and based on the transaction operation number, M sorting threads are used to perform sorting processing on multiple target data update records associated with the M source data tables in parallel to obtain data sorting hash tables corresponding to the M source data tables.

[0071] For example, read the target data update records from the change data queues node1_db1, node2_db1, and node3_db1 corresponding to the source data nodes node1, node2, and node3, respectively. The target data update records a1 (transaction operation number 001, xxx) and a2 (transaction operation number 005, xxx) are read from node1_db1; the target data update records b1 (transaction operation number 004, xxx) and b2 (transaction operation number 006, xxx) are read from node2_db1; the target data update records c1 (transaction operation number 003, xxx) and c2 (transaction operation number 008, xxx) are read from node3_db1. Source data table 1 is associated with a1, b2, and c1, and source data table 2 is associated with a2, b1, and c2. Based on the ascending rule of transaction operation number, two sorting threads are used to sort the target data update records associated with source data table 1 and source data table 2, so that the data sorting hash table corresponding to source data table 1 includes ordered a1, c1, and b2, and the data sorting hash table corresponding to source data table 2 includes ordered b1, a2, and c2.

[0072] Optionally, the data sorting hash table corresponding to the mth source data table includes the data associated with the mth source data table. Target data update records The sorting result corresponding to each primary key contains multiple ordered target data update records, and the sorting results of different primary keys are also ordered.

[0073] For example, the data sorting hash table corresponding to source data table 3 includes d1 (transaction operation number 010, xxx), d2 (transaction operation number 011, xxx), d3 (transaction operation number 012, xxx), and d3 (transaction operation number 014, xxx). The sorting result corresponding to primary key d1 includes one target data update record d1 (transaction operation number 010, xxx); the sorting result corresponding to primary key d2 includes one target data update record d2 (transaction operation number 011, xxx); the sorting result corresponding to primary key d3 includes two ordered target data update records d3 (transaction operation number 012, xxx) and d3 (transaction operation number 014, xxx).

[0074] Optional, h, , , All are positive integers. The sum of the ordered target data update records corresponding to each primary key is the total number of target data update records associated with the mth source data table. , .

[0075] Optionally, the configuration parameters include a merge parameter. When the merge parameter is 0, all target data update records are recorded without performing a merge operation; when the merge parameter is 1, a merge operation is performed.

[0076] Optionally, the merging and deduplication of the target data update records in a batch are performed based on the source data table. A merging thread is started for each source data table. While sorting each source data table, the data merging hash table is handed over to the merging thread for merging. For the sorting result corresponding to each primary key in the sorted hash table, based on the streaming merging algorithm, the merging thread is used to merge at least one ordered target data update record corresponding to each primary key to obtain the merged result corresponding to each primary key. The merged result corresponding to each primary key contains 1 or 0 merged data.

[0077] Optional, according to primary keys and the merge results corresponding to each primary key to obtain a data merge hash table corresponding to the mth source data table.

[0078] Optionally, target data update records are read from the change data queue corresponding to each source data node based on the target batch transaction operation number. Since the target data update records read out from a change data queue are sorted according to the transaction operation number, when sorting target data update records from different change data queues in a source data table, the merge sort algorithm can be directly used to sort only the target data update records between the change data queues. The target data update records in the same change data queue do not need to be sorted again. The sorting time complexity is O(N), which greatly improves the database performance.

[0079] Optionally, for the sorting result corresponding to the hth primary key, based on the streaming merge algorithm, use the merge thread to sort the ordered The target data update records are merged and processed, and the merge results corresponding to the hth primary key are obtained, including: traversing the ordered target data update record; for each target data update record, according to the hth primary key and transaction operation type, update the data change operation in the linked list corresponding to the hth primary key; according to the data change operation in the linked list corresponding to the hth primary key, obtain the merge result corresponding to the hth primary key.

[0080] Optionally, the data merge hash table includes a primary key and a linked list corresponding to the primary key, and the linked list stores data change operations of the primary key in order of transaction operation numbers.

[0081] Optionally, for the sort result corresponding to the hth primary key, traverse the ordered target data update records, and for each target data update record, according to the hth primary key and the transaction operation type in the target data update record, update the data change operation in the linked list corresponding to the hth primary key.

[0082] Optional, in order After the traversal of the target data update records is completed, the merge result corresponding to the hth primary key is obtained according to the data change operation in the linked list corresponding to the hth primary key.

[0083] Optionally, for each target data update record, according to the hth primary key and the transaction operation type, updating the data change operation in the linked list corresponding to the hth primary key includes: for each target data update record, if the hth primary key does not exist in multiple candidate target data update records in the data sorting hash table, adding a data change operation in the linked list corresponding to the hth primary key, wherein the transaction operation number in the candidate target data update record is less than the transaction operation number in the target data update record; for each target data update record, if the hth primary key exists in multiple candidate target data update records and the transaction operation type is an insert type, clearing the data change operation in the linked list corresponding to the hth primary key. The linked list corresponding to the primary key adds the current data insertion operation; for each target data update record, if there is an h-th primary key in multiple candidate target data update records and the transaction operation type is an update type, traverse the linked list corresponding to the h-th primary key, and add the current data update operation if there is no historical data update operation for the changed column in the linked list; for each target data update record, if there is an h-th primary key in multiple candidate target data update records and the transaction operation type is a delete type, clear the linked list corresponding to the h-th primary key, and add the current data deletion operation if there is no historical data insertion operation for the h-th primary key in the linked list.

[0084] Optionally, for a certain target data update record, a target data update record whose transaction operation number is less than the transaction operation number of the target data update record is determined as a candidate target data update record in the data sorting hash table. The target data update record includes the hth primary key, and the absence of the hth primary key in multiple candidate target data update records indicates that the target data update record is generated based on the first data modification operation on the hth primary key, such as the first data insertion operation on the hth primary key.

[0085] Optionally, when the hth primary key does not exist in multiple candidate target data update records, a data change operation corresponding to this target data update record is added to the linked list corresponding to the hth primary key. The data change operation is an operation to change the data in the subtable of the target source data table based on a transaction operation request for the source data table.

[0086] Optionally, the existence of the hth primary key in the multiple candidate target data update records indicates that the target data update record is generated based on a non-first data modification operation on the hth primary key.

[0087] Optionally, when there is an h-th primary key in multiple candidate target data update records and the transaction operation type is an insert type, the linked list corresponding to the h-th primary key is cleared and the current data insertion operation is added. This is equivalent to clearing all operations before the h-th primary key and performing a merge.

[0088] Optionally, if the hth primary key exists in multiple candidate target data update records and the transaction operation type is update type, traverse the linked list corresponding to the hth primary key, add the current data update operation if there is no historical data update operation for the changed column in the linked list, and merge the historical data update operations for other columns. This is equivalent to merging the results of all updates, which will not affect the final data result.

[0089] Optionally, if there is a historical data update operation for the changed column in the linked list, the historical data update operation is removed and the current data update operation is added.

[0090] Optionally, when the hth primary key exists in multiple candidate target data update records and the transaction operation type is a deletion type, the linked list corresponding to the hth primary key is cleared, and the current data deletion operation is added when there is no historical data insertion operation for the hth primary key in the linked list. This is equivalent to ignoring all previous operations on the hth primary key and only retaining the data deletion operation, which does not affect the final data result.

[0091] Optionally, if there is a historical data insertion operation for the hth primary key in the linked list, the current data deletion operation is not added. This is to avoid that there is only a data deletion operation, but the historical data insertion operation is deleted, resulting in an error when deleting.

[0092] Optionally, multiple ordered target data update records corresponding to the primary key are merged and deduplicated. According to the characteristics of the changed data involved in the synchronization process, repeated data update operations and data deletion operations are implemented in the primary key linked list, and only the last operation is retained. The operations that can be merged between the previous and subsequent consecutive operations are merged, and then the redundant data change operations with low importance are removed without affecting the final result, thereby effectively improving the synchronization performance.

[0093] Optionally, the data synchronization method based on a distributed database also includes: determining the execution status of the transaction operation request corresponding to the transaction operation number; based on the batch transaction operation number reading interval parameter, reading multiple transaction operation numbers corresponding to the execution status representing the uncommitted status from the global transaction manager; selecting the minimum transaction operation number from the multiple transaction operation numbers to determine as the target transaction operation number; determining the previous transaction operation number of the target transaction operation number as the batch transaction operation number, wherein the execution status of the batch transaction operation number and the transaction operation number before the batch transaction operation number both represent the committed status; and storing the batch transaction operation number in the batch transaction operation number queue.

[0094] Optionally, the execution status of the transaction operation request represents a committed or uncommitted state of the transaction.

[0095] Optionally, based on a batch transaction operation number reading interval parameter, multiple transaction operation numbers corresponding to the execution state representing the uncommitted state are read from the global transaction manager, and a minimum transaction operation number is selected from the multiple transaction operation numbers to be determined as the target transaction operation number.

[0096] For example, the batch transaction operation number reading interval parameter is 10 minutes, and the target transaction operation number is read from the global transaction manager every 10 minutes. During one reading, the execution status corresponding to the transaction operation request number in the global transaction manager is (transaction operation number 0000: committed status, xxx, transaction operation number 1000: committed status, transaction operation number 1001: uncommitted status, transaction operation number 1002: committed status, transaction operation number 1003: uncommitted status, xxx). The smallest transaction operation number among the multiple transaction operation numbers corresponding to the uncommitted execution status read is 1001, so the target transaction operation number is determined to be transaction operation number 1001, and thus transaction operation number 1000 is determined to be the batch transaction operation number.

[0097] Optionally, the execution status of the batch transaction operation number and the transaction operation number before the batch transaction operation number both represent the submission status. Based on the batch transaction operation number reading interval parameter, a batch transaction operation number is periodically read each time and stored in the batch transaction operation number queue. When the synchronization of the previous batch of transactions is completed, the target batch transaction number is read from the batch transaction operation number queue to complete the synchronization of the next batch of transactions.

[0098] Optionally, the batch transaction operation number is the maximum global safe transaction boundary, and all transaction operation requests less than this batch transaction operation number have been submitted. There will be many submitted transactions between this batch transaction operation number and the last batch transaction operation number read, thereby ensuring that there is no repeated submission or omission of unsubmitted data between the target data update records of the batch processed by the target batch transaction operation number each time. Therefore, data synchronization based on the target batch transaction operation number in the transaction operation number queue ensures the eventual consistency of the transaction boundary and will not cause data logic errors.

[0099] Optionally, using M loading threads to synchronize the merged data of M source data tables to the target database in parallel includes: for the merged data in each source data table, performing variable assignment processing on the changing column according to the primary key and transaction operation type to obtain dynamic data operation statements associated with each source data table; using M loading threads to synchronize the dynamic data operation statements associated with the M source data tables to the target database in parallel.

[0100] Optionally, after the merge processing of each source data table is completed, traverse the linked list of each primary key in the data merge hash table corresponding to the source data table, and assign the changed column to the variable of the dynamic data operation statement according to the transaction operation type corresponding to the data change operation in the linked list to obtain the dynamic data operation statement associated with each source data table.

[0101] Optionally, a loading thread is started for each source data table, and M loading threads are used to synchronize dynamic data operation statements associated with the M source data tables in batches to target tables mapped to the source data tables in the target database in parallel.

[0102] Optional, parallel synchronous loading of multiple source data tables ensures parallel synchronous processing and effective use of CPU resources.

[0103] Optionally, the dynamic data operation statements include dynamic data insert operation statements, dynamic data update operation statements, and dynamic data delete operation statements; wherein, for each merged data in each source data table, variable assignment processing is performed on the change column according to the primary key and transaction operation type, and the dynamic data operation statements associated with each source data table include: for each merged data, when the transaction operation type is an insert type, the primary key and the change column are updated to the dynamic data insert operation statement; for each merged data, when the transaction operation type is an update type, based on the merge clause processing threshold, the primary key and the change column are updated to the dynamic data update operation statement; for each merged data, when the transaction operation type is a delete type, based on the merge clause processing threshold, the primary key is updated to the dynamic data delete operation statement.

[0104] Optionally, the linked list of each primary key in the data merge hash table corresponding to the source data table is traversed to generate dynamic data insertion operation statements, dynamic data update operation statements, and dynamic data deletion operation statements.

[0105] Optionally, for each merged data, according to the primary key in the merged data and the transaction operation type of the data change operation in the linked list corresponding to the primary key, the merged data is updated to a dynamic data operation statement corresponding to the transaction operation type.

[0106] Optionally, when the transaction operation type is insert type, the primary key and the changing column are assigned to the primary key variable and the column variable in the dynamic data insert operation statement.

[0107] Optionally, when the transaction operation type is an update type, and when the same column variable value does not exist in the dynamic data update operation statement, the primary key and the changing column are assigned to the primary key variable and the column variable in the dynamic data update operation statement.

[0108] Optionally, the configuration parameters include merge clause parameters and merge clause processing threshold. The merge clause processing threshold represents the upper limit of the merge processing clause. If the merge clause processing threshold is not reached when the same column variable value exists in the dynamic data update operation statement, the primary key is assigned to the primary key variable in the dynamic data update operation statement based on the merge clause. The merge clause can be an IN clause.

[0109] For example, if the merge clause processing threshold is 1000, it means that a maximum of 1000 data update operations can be merged in one dynamic data update operation statement. If there is a column variable value of 80 corresponding to the column variable "score" in the dynamic data update operation statement, if the merge clause processing threshold is not reached, the primary key is assigned to the primary key variable in the dynamic data update operation statement based on the merge clause. For example, the dynamic data update operation statement: UPDATE source data table 1 SET score =80 WHERE department_id IN (10, 15, 20), where the primary key variable is department_id and the column variable score is "score". It can be seen that one IN clause merges three statements with primary key variables department_id as 10, 15, and 20.

[0110] Optionally, when the transaction operation type is a deletion type, if the merge clause processing threshold is not reached, the primary key is assigned to the primary key variable in the dynamic data deletion operation statement based on the merge clause; if the merge clause processing threshold is reached, the primary key is directly assigned to the primary key variable in the dynamic data deletion operation statement.

[0111] Optionally, if the Use Merge Clause parameter is set, when assigning values ​​to variables, data changes to different primary keys in the same transaction are merged into the merge clause.

[0112] For example, the dynamic data deletion operation statement is: DELETE FROM source data table 1 WHERE customer_id IN (21, 22, 23), where the primary key variable is customer_id. It can be seen that one IN clause combines three statements where the primary key variable customer_id is 21, 22, and 23.

[0113] Optionally, dynamic variable assignment is to traverse each data change operation and assign the value of the data change operation. When the set single batch submission data volume parameter is reached, the dynamic data operation statement is executed.

[0114] Optionally, update the data change operation to the dynamic data operation statement, put the dynamic data operation statement into the dynamic operation queue, and read and execute the dynamic data operation statement from the dynamic operation queue when the set single batch submission data volume parameter is reached. The memory pool used for data buffering in the database can allocate the execution plan of the dynamic data operation statement based on the single batch submission data volume parameter, effectively utilizing the fragmented execution plan cache of the database for batch loading synchronization. In addition, by using the merge clause to optimize the dynamic data operation statement, you can batch merge multiple statements into one statement, effectively improving the synchronization performance and the performance of the target database.

[0115] Figure 3 A schematic diagram of a distributed database according to an embodiment of the present invention is shown.

[0116] like Figure 3 As shown, the distributed database includes source data nodes Node1, Node2, Node3 and GTM. GTM can be understood as the global transaction manager (Globale Transaction Manager, GTM). The logical replication slot records the data update records on the source data nodes. The logical replication slot reading threads: data thread 1, data thread 2, data thread 3 are used to logically copy the data update records on each source data node to the KAFKA change data queues Node1_db1, Node2_db1, and Node3_db1 corresponding to the source data nodes Node1, Node2, and Node3, respectively. The batch transaction operation number reading thread (GTM thread) is used to periodically read a batch transaction operation number from the global transaction manager (GlobaleTransaction Manager, GTM) based on the batch transaction operation number reading interval parameter and store it in the batch transaction operation number queue Batch_GTM.

[0117] Optionally, read the target batch transaction operation number from the batch transaction operation number queue, and then screen N target data update records whose transaction operation numbers are less than the target batch transaction operation number from multiple change data queues. Based on the streaming merge algorithm, use three merge threads to perform data merge processing on the target data update records associated with source data table 1 (or db1_table1), source data table 2 (or db1_table2), and source data table 3 (or db1_table3) in parallel, and then generate dynamic data operation statements (or dynamic SQL), put the variable assignments (db1_table1 dynamic SQL+value, db1_table2 dynamic SQL+value, db1_table3 dynamic SQL+value) in the dynamic data operation statements corresponding to each source data table into the KAFKA dynamic operation queue corresponding to each source data table, use three loading threads (db1_table1 loading thread 1, db1_table2 loading thread 2, db1_table3 loading thread 3) to read and execute the dynamic data operation statements from the KAFKA dynamic operation queue, and then synchronize the merged data of each source data table to the target database.

[0118] Based on the above-mentioned data synchronization method based on a distributed database, the present invention also provides a data synchronization device based on a distributed database. Figure 4 The device is described in detail.

[0119] Figure 4 A structural block diagram of a data synchronization device based on a distributed database according to an embodiment of the present invention is shown.

[0120] like Figure 4 As shown, the data synchronization device 400 based on a distributed database of this embodiment includes a reading module 410 , a determining module 420 , a screening module 430 , a merging module 440 and a synchronization module 450 .

[0121] The reading module 410 is used to respond to the batch transaction processing request and read the target batch transaction operation number corresponding to the current batch transaction to be processed from the batch transaction operation number queue. In one embodiment, the reading module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0122] The determination module 420 is used to determine the change data queues corresponding to the L source data nodes, wherein the change data queues store multiple sets of data update records generated by the source data nodes by executing transaction operations during the data synchronization process, and the multiple sets of data update records correspond to multiple transaction operation numbers. In one embodiment, the determination module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0123] The screening module 430 is used to screen N target data update records whose transaction operation numbers are less than the target batch transaction operation number from the L change data queues, the N target data update records are associated with the M source data tables, and the L source data nodes jointly complete the data storage of the M source data tables. In one embodiment, the screening module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0124] The merging module 440 is used to merge the target data update records associated with each of the M source data tables in parallel using M merging threads based on the stream merging algorithm to obtain M data merging hash tables corresponding to the M source data tables, wherein the mth source data table is associated with The target data update record is stored in the data merge hash table corresponding to the mth source data table. Primary key associated merge results, the primary key is derived from the target data update record, each merge result contains at most one merged data, , In one embodiment, the merging module 440 may be used to perform the operation S240 described above, which will not be described in detail herein.

[0125] The synchronization module 450 is used to synchronize the merged data of the M source data tables to the target database in parallel using the M loading threads. In one embodiment, the synchronization module 450 can be used to perform the operation S250 described above, which will not be described in detail here.

[0126] Optionally, the data synchronization device 400 based on a distributed database further includes a response module, a modification module, a synchronization module and a storage module.

[0127] A response module is used to respond to a transaction operation request for each source data table, determine a transaction operation number and i target source data table sub-tables to be operated, wherein the transaction operation number is allocated from a global transaction manager, the source data table includes I source data table sub-tables, the i target source data table sub-tables are derived from the I source data table sub-tables, the source data nodes correspond to the source data table sub-tables one by one, and i≤I.

[0128] The change module is used to perform data change operations on i target source data table sub-tables to obtain i data update records corresponding to the i target source data table sub-tables, wherein each data update record includes a transaction operation number, a transaction operation type, a target source data table sub-table name, a primary key, and a changed column.

[0129] The synchronization module is used to synchronously store i data update records in i logical replication slots corresponding to the target source data nodes where the i target source data table sub-tables are located.

[0130] The storage module is used to store the data update records in i logical replication slots in parallel into i change data queues corresponding to i target source data nodes by using i reading threads.

[0131] Optionally, the data synchronization device 400 based on a distributed database further includes a sorting module, a processing module and an obtaining module.

[0132] The sorting module is used to sort the data associated with the mth source data table based on the transaction operation number for any mth source data table. The target data update records are sorted to obtain the data sorting hash table corresponding to the m-th source data table, wherein the data sorting hash table corresponding to the m-th source data table stores the data corresponding to the m-th source data table. Primary key associated The sorting results corresponding to the hth primary key contain ordered Target data update records, , .

[0133] The processing module is used to sort the sort results corresponding to the hth primary key in the sorted hash table, based on the stream merge algorithm, using the merge thread to sort the ordered The target data update records are merged to obtain the merge result corresponding to the hth primary key.

[0134] Get the module for primary keys and the merge results corresponding to each primary key to obtain a data merge hash table corresponding to the mth source data table.

[0135] Optionally, the processing module includes a first processing submodule, a second processing submodule and a third processing submodule.

[0136] The first processing submodule is used to traverse the ordered Target data update record.

[0137] The second processing submodule is used to update the record for each target data, and update the data change operation in the linked list corresponding to the hth primary key according to the hth primary key and the transaction operation type.

[0138] The third processing submodule is used to obtain a merging result corresponding to the hth primary key according to the data modification operation in the linked list corresponding to the hth primary key.

[0139] Optionally, any multiple modules among the reading module 410, the determination module 420, the screening module 430, the merging module 440 and the synchronization module 450 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the reading module 410, the determination module 420, the screening module 430, the merging module 440 and the synchronization module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the reading module 410 , the determining module 420 , the screening module 430 , the merging module 440 , and the synchronization module 450 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0140] Figure 5 A block diagram of an electronic device suitable for implementing a data synchronization method based on a distributed database according to an embodiment of the present invention is shown.

[0141] Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0142] like Figure 5 As shown, the computer electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an on-board memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0143] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0144] Optionally, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0145] Optionally, the method flow according to the embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. Optionally, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0146] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the data synchronization method based on the distributed database according to the embodiment of the present invention is implemented.

[0147] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0148] For example, optionally, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0149] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data synchronization method based on a distributed database provided by the embodiment of the present invention.

[0150] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are performed. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0151] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0152] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0154] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data synchronization method based on a distributed database, characterized in that: The distributed database includes a source database and a target database to be synchronized with each other, the source database includes L source data nodes; the method includes: In response to the batch transaction processing request, read the target batch transaction operation number corresponding to the current batch transaction to be processed from the batch transaction operation number queue; Determine a change data queue corresponding to each of the L source data nodes, wherein the change data queue stores multiple groups of data update records generated by the source data node executing transaction operations during the data synchronization process, and the multiple groups of data update records correspond to multiple transaction operation numbers; Filtering N target data update records whose transaction operation numbers are smaller than the target batch transaction operation numbers from the L change data queues, wherein the N target data update records are associated with M source data tables, and the L source data nodes jointly complete the data storage of the M source data tables; Based on the stream merging algorithm, M merging threads are used to merge the target data update records associated with the M source data tables in parallel to obtain M data merging hash tables corresponding to the M source data tables, wherein the mth source data table is associated with The target data update record is stored in the data merge hash table corresponding to the mth source data table. Primary key associated merge results, the primary key is derived from the target data update record, each of the merge results contains at most one merged data, , ; The merged data of the M source data tables are synchronized to the target database in parallel using M loading threads.

2. The method according to claim 1, characterized in that The method further comprises: In response to a transaction operation request for each of the source data tables, determining a transaction operation number and i target source data table sub-tables to be operated, wherein the transaction operation number is allocated from a global transaction manager, the source data table includes I source data table sub-tables, the i target source data table sub-tables are derived from the I source data table sub-tables, the source data nodes correspond to the source data table sub-tables one by one, and i≤I; Performing data modification operations on i sub-tables of the target source data table to obtain i data update records corresponding to the i sub-tables of the target source data table, wherein each of the data update records includes the transaction operation number, the transaction operation type, the name of the sub-table of the target source data table, the primary key, and the changed column; Synchronously storing the i data update records in the i logical replication slots corresponding to the target source data nodes where the i target source data table sub-tables are located; The data update records in the i logical replication slots are stored in parallel by using i reading threads into the i change data queues corresponding to the i target source data nodes.

3. The method according to claim 1, characterized in that Before the merging process, the method further includes: For any m-th source data table, based on the transaction operation number, The target data update records are sorted to obtain a data sorting hash table corresponding to the m-th source data table, wherein the data sorting hash table corresponding to the m-th source data table stores Primary key associated The sorting result corresponding to the hth primary key contains the ordered the target data update record, , ; Wherein, for any of the m-th source data tables, the merging process includes: For the sorting result corresponding to the hth primary key in the sorted hash table, based on the stream merge algorithm, the merge thread is used to merge the ordered merging the target data update records to obtain a merging result corresponding to the hth primary key; according to The primary keys and the merging results corresponding to each of the primary keys are used to obtain the data merging hash table corresponding to the mth source data table.

4. The method according to claim 3, characterized in that For the sorting result corresponding to the hth primary key, based on the stream merging algorithm, the merging thread is used to merge the ordered The target data update records are merged to obtain the merge result corresponding to the hth primary key, including: Traverse the ordered update record of the target data; For each target data update record, according to the hth primary key and the transaction operation type, update the data change operation in the linked list corresponding to the hth primary key; According to the data modification operation in the linked list corresponding to the hth primary key, the merging result corresponding to the hth primary key is obtained.

5. The method according to claim 4, characterized in that For each target data update record, according to the hth primary key and the transaction operation type, the data modification operation in the linked list corresponding to the hth primary key is updated, including: For each of the target data update records, if the h-th primary key does not exist in the multiple candidate target data update records in the data sorting hash table, add a data change operation in the linked list corresponding to the h-th primary key, wherein the transaction operation number in the candidate target data update record is smaller than the transaction operation number in the target data update record; For each of the target data update records, if the hth primary key exists in the multiple candidate target data update records and the transaction operation type is an insert type, clear the linked list corresponding to the hth primary key and add a current data insert operation; For each target data update record, if the hth primary key exists in the multiple candidate target data update records and the transaction operation type is an update type, traverse the linked list corresponding to the hth primary key, and add a current data update operation if there is no historical data update operation for the changed column in the linked list; For each of the target data update records, if the hth primary key exists in the multiple candidate target data update records and the transaction operation type is a deletion type, clear the linked list corresponding to the hth primary key, and add a current data deletion operation if there is no historical data insertion operation for the hth primary key in the linked list.

6. The method according to claim 1, characterized in that The method further comprises: Determine the execution status of the transaction operation request corresponding to the transaction operation number; Based on a batch transaction operation number reading interval parameter, read from a global transaction manager a plurality of transaction operation numbers corresponding to an execution state representing an uncommitted state; Selecting the smallest transaction operation number from the plurality of transaction operation numbers as the target transaction operation number; Determine a transaction operation number preceding the target transaction operation number as a batch transaction operation number, wherein the execution status of the batch transaction operation number and the transaction operation number preceding the batch transaction operation number both represent a commit status; The batch transaction operation number is stored in the batch transaction operation number queue.

7. The method according to claim 1, characterized in that Using M loading threads to synchronize the merged data of the M source data tables to the target database in parallel includes: For the merged data in each of the source data tables, variable assignment processing is performed on the change column according to the primary key and the transaction operation type to obtain a dynamic data operation statement associated with each of the source data tables; The dynamic data operation statements associated with the M source data tables are synchronized to the target database in parallel using M loading threads.

8. The method according to claim 7, characterized in that The dynamic data operation statements include dynamic data insert operation statements, dynamic data update operation statements, and dynamic data delete operation statements; Among them, for the merged data in each of the source data tables, performing variable assignment processing on the change column according to the primary key and the transaction operation type, and obtaining the dynamic data operation statement associated with each of the source data tables includes: For each piece of the merged data, when the transaction operation type is an insert type, updating the primary key and the change column into the dynamic data insert operation statement; For each of the merged data, when the transaction operation type is an update type, based on a merge clause processing threshold, updating the primary key and the change column into the dynamic data update operation statement; For each piece of the merged data, when the transaction operation type is a deletion type, based on the merge clause processing threshold, the primary key is updated into the dynamic data deletion operation statement.

9. A data synchronization device based on a distributed database, characterized in that: The device comprises: A reading module, used for responding to a batch transaction processing request and reading a target batch transaction operation number corresponding to a current batch transaction to be processed from a batch transaction operation number queue; A determination module, configured to determine a change data queue corresponding to each of the L source data nodes, wherein the change data queue stores a plurality of groups of data update records generated by the source data node executing a transaction operation during the data synchronization process, and the plurality of groups of data update records correspond to a plurality of transaction operation numbers; A screening module is used to screen N target data update records whose transaction operation numbers are smaller than the target batch transaction operation numbers from the L change data queues, the N target data update records are associated with M source data tables, and the L source data nodes jointly complete the data storage of the M source data tables; A merging module is used to merge the target data update records associated with the M source data tables in parallel using M merging threads based on a stream merging algorithm to obtain M data merging hash tables corresponding to the M source data tables, wherein the mth source data table is associated with The target data update record is stored in the data merge hash table corresponding to the mth source data table. Primary key associated merge results, the primary key is derived from the target data update record, each of the merge results contains at most one merged data, , ; A synchronization module is used to synchronize the merged data of the M source data tables to the target database in parallel using M loading threads.

10. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 8.

Images