Data synchronization method and device

Abstract

The invention relates to a data synchronization method and device, and belongs to the technical field of database synchronization, and the method comprises the steps: dividing tables in a source library into a plurality of groups, and carrying out the initial loading of the tables in each group; and when the target group completing initial loading exists, performing incremental synchronization on the table in the target group. Initial loading and increment synchronization are performed in parallel, so that the waiting time of increment synchronization can be saved, and the data synchronization performance is optimized.

Description

Technical Field

[0001] This invention relates to the field of database synchronization technology, and in particular to a data synchronization method and apparatus. Background Technology

[0002] Data synchronization is divided into two phases: initial loading and incremental synchronization. The initial loading phase processes existing data in the source database tables, while the incremental phase processes changed data. During the initial loading phase, the source database tables are sliced ​​based on primary keys or ROWID. The sliced ​​result sets are read in parallel on a slice-by-slice basis, and the data in the result sets is transformed and written to the target database. The incremental phase mines the source database archive logs, parsing out incremental logs for inserts, deletes, and updates, and executes and writes these transactions into the database on a transaction-by-transaction basis.

[0003] Traditional data synchronization methods require waiting for all tables to be loaded during the initial loading phase before entering the incremental phase to capture incremental logs. During the initial loading process, even if some tables have already loaded their existing data, these tables cannot enter the incremental synchronization phase, and archived logs cannot be read, parsed, or executed for storage. Traditional synchronization methods have strict phase divisions; during the initial loading phase, the mining, parsing, and execution of incremental logs are all in a waiting state. The longer the initial loading process takes, the longer the incremental synchronization wait time, resulting in relatively low synchronization performance. Summary of the Invention

[0004] In view of this, it is necessary to provide a data synchronization method and apparatus to solve the problem of low data synchronization performance in existing databases.

[0005] To address the aforementioned problems, in a first aspect, the present invention provides a data synchronization method, comprising: The tables in the source database are divided into multiple groups, and the tables in each group are initially loaded. When a target group has completed its initial loading, the tables in that target group are incrementally synchronized.

[0006] In one possible implementation, the initial loading of the tables in each group includes: Set priorities for each group, and perform initial loading of the tables in each group according to the priorities.

[0007] In one possible implementation, the method further includes: The status of each group is updated in real time; wherein, the status includes: initial loading completed; When a target group has completed its initial loading, the incremental synchronization of tables in the target group includes: When the status of the group is "initial loading complete", the group is used as the target group, and the tables in the target group are incrementally synchronized.

[0008] In one possible implementation, the method further includes: The incremental synchronization main thread parses the incremental logs of the source database to obtain transactions, and then splits the transactions according to the grouping rules to obtain group transactions corresponding to each group. The incremental synchronization of tables in the target group includes: The target group execution thread corresponding to the target group sends the target group transaction to the execution end so that the execution end executes the target group transaction in the target database.

[0009] In one possible implementation, the incremental synchronization main thread is further used to split the transaction into master group transactions corresponding to the master group; the step of sending the target group transaction corresponding to the target group to the execution end through the group execution thread corresponding to the target group includes: If there is a first transaction to be committed in the target group transaction, and the difference between the commit SCN of the first transaction to be committed and the SCN of the log of the incremental synchronization main thread is less than a preset threshold, or if there is no first transaction to be committed in the target group transaction, then wait until the committed transactions in the target group transaction are completed by the execution terminal, then merge the target group into the main group, merge the target group transaction into the main group transaction, and update the grouping rules; The incremental synchronization main thread sends the pending transactions in the main group of transactions to the execution end, so that the execution end executes the pending transactions in the target database.

[0010] In one possible implementation, the method further includes: When the first transaction to be committed exists in the target group transaction, and the difference between the commit SCN of the first transaction to be committed and the SCN of the log of the incremental synchronization main thread is less than the preset threshold, or when the first transaction to be committed does not exist in the target group transaction, the incremental synchronization main thread is paused. When the grouping rules are updated, the incremental synchronization main thread is woken up.

[0011] In one possible implementation, the step of sending the transactions to be executed in the main group of transactions to the execution end through the incremental synchronization main thread includes: If the target group transaction is merged into the main group transaction, and the target group transaction includes the first transaction to be committed, then when the target group transaction is merged into the main group transaction, and there is a second transaction to be committed in the main group transaction that shares the same log sequence number as the first transaction to be committed, the first transaction to be committed will be executed first through the execution terminal.

[0012] In one possible implementation, the method further includes: When the target group execution thread starts, the status of the target group is updated to incremental synchronization. When the target group is merged into the main group, the status of the target group is updated to "merged".

[0013] In one possible implementation, the table in the source database is divided into multiple groups, including: Group the primary and secondary tables with primary and foreign key relationships into the same group; The incremental synchronization of tables in the target group includes: Based on foreign key constraints, incremental synchronization is performed on the primary and secondary tables associated with primary and foreign keys in the target group.

[0014] In a second aspect, the present invention also provides a data synchronization device, comprising: The initial loading module is used to divide the tables in the source database into multiple groups and perform initial loading on the tables in each group. The incremental synchronization module is used to perform incremental synchronization of tables in a target group when an initial load of the target group has been completed.

[0015] The beneficial effects of this invention are: Compared to the traditional method that requires waiting for all tables in the source database to complete initial loading before incremental synchronization, this invention divides the tables in the source database into multiple groups and performs initial loading on the tables in each group separately. When a target group has completed initial loading, incremental synchronization is performed on the tables in the target group. During the data synchronization process of the source database, initial loading and incremental synchronization are performed in parallel, thereby saving the waiting time for incremental synchronization and optimizing data synchronization performance. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1A flowchart illustrating an embodiment of the data synchronization method provided by the present invention; Figure 2 A flowchart illustrating another embodiment of the data synchronization method provided by the present invention; Figure 3 A schematic diagram of an incremental synchronization process provided by the present invention; Figure 4 This invention provides a schematic diagram of a grouped asynchronous initial loading process; Figure 5 This is a schematic diagram illustrating the execution order of merged grouped transactions provided by the present invention; Figure 6 This invention provides a schematic diagram of transaction conflict detection for each group. Figure 7 This is a schematic diagram of an embodiment of the data synchronization device provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0019] In the description of the embodiments of this invention, unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," etc., used in the embodiments of this invention are used to distinguish similar objects, and are not used to describe a specific order or sequence, nor to indicate or imply their relative importance or implicitly specify the number of indicated technical features. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, and the number of objects is not limited; for example, a first object can be one or more.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Before demonstrating the embodiments, the following terms will be explained.

[0022] Database transaction: A database transaction is a sequence of database operations that access and potentially manipulate various data items. These operations are either all executed or none are executed; it is an indivisible unit of work. A transaction consists of all database operations performed between the start and end of the transaction.

[0023] Transaction splitting: Divide the operation sequence of a transaction into multiple sub-transactions (grouped transactions).

[0024] System Change Number (SCN): A sequentially increasing number in the database used to precisely distinguish the order of operations.

[0025] A thread is the smallest unit of computation that an operating system can schedule. It is contained within a process and is the actual unit of operation within a process.

[0026] Queue: A queue is a linear data structure that allows insertion at one end and deletion at the other. Direct access to any data outside of these two ends is prohibited. Furthermore, a queue is a first-in, first-out (FIFO) data structure.

[0027] Inbound: Execute a transaction operation on the target database to achieve data synchronization.

[0028] Pending transactions: These are transactions that have been committed in the source database but not yet executed in the target database.

[0029] Pending transactions: refers to transactions that have already been committed in the source database.

[0030] Log Sequence Number (LSN): A unique number used in a database system to identify the physical location of transaction logs (such as Redo Log and Binlog).

[0031] This invention provides a data synchronization method and apparatus, which will be described below.

[0032] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of the data synchronization method provided by the present invention, the method comprising: S101, divide the tables in the source database into multiple groups, and perform initial loading of the tables in each group.

[0033] The source database can be a database such as DM database or Oracle database.

[0034] The tables in the source database can be grouped according to preset grouping rules to obtain multiple groups. Each group can include at least one table, and each group can be initially loaded in a certain order or in parallel.

[0035] S102, when there is a target group that has completed the initial loading, perform incremental synchronization on the tables in the target group.

[0036] Incremental synchronization is performed on groups that have completed initial loading first, without waiting for all tables to complete initial loading.

[0037] The data synchronization method provided in this embodiment can be applied to a data synchronization system, which can be a software system running on a terminal device. The terminal device can be a tablet computer, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), mobile phone, etc. This embodiment does not impose any restrictions on the specific type of terminal device.

[0038] In summary, this embodiment synchronizes data from the source database to the target database by performing initial loading and incremental synchronization in parallel during the synchronization process, thereby saving the waiting time for incremental synchronization and optimizing data synchronization performance.

[0039] In some embodiments of the present invention, the step of initially loading the tables in each group may include: setting a priority for each group, and initially loading the tables in each group according to the priority.

[0040] In this embodiment, priority can be set for groups, and the tables corresponding to core businesses can be grouped into high-priority groups, so that the high-priority groups, that is, the tables corresponding to core businesses, can complete the initial loading and data synchronization faster.

[0041] For tables that are not core business operations, such as log tables or tables with large fields, the amount of data may be very large and the initial loading speed may be relatively slow. Therefore, they can be assigned to lower priority groups.

[0042] Referring to Table 1, a grouping table provided by the present invention is shown. Tables A and B are assigned to group 1, and the priority of group 1 is set to 2. Tables C and D are assigned to group 2, and the priority of group 2 is set to 1.

[0043] Table 1 Grouping Table

[0044] In some embodiments of the present invention, the method further includes: generating grouping information; wherein the grouping information includes: group number, status, priority, expression, etc. The status includes: initial loading not started, initial loading in progress, initial loading completed, incremental synchronization in progress, or merged.

[0045] "Not started initial load" means that all tables in the group have not started initial load.

[0046] The initial load indicates that there are tables in the group undergoing initial loading.

[0047] Initial load complete means that all tables in the group have been initially loaded.

[0048] Incremental synchronization means that all tables in the group have been initially loaded and the group execution thread has been started, and the group transaction (the incremental synchronization transaction of the group) has been issued.

[0049] "Merged" indicates that the group transaction has been completed and the group has been merged into the main group.

[0050] The group status can be represented by characters. For example, 0, 1, 2, 3, and 4 can represent not starting initial loading, initial loading in progress, initial loading completed, incremental synchronization in progress, or merged, respectively.

[0051] In some embodiments of the present invention, the step of initially loading the tables in each group may include: First, the tables in the group are sorted according to priority and placed into the queue of tables to be sharded in order. If there are multiple groups with the same priority, each group is polled and placed into the queue of tables to be sharded.

[0052] Then, the sharding thread retrieves the tables to be sharded from the queue of tables to be sharded, shards the tables based on the primary key or row identifier (ROWID), and generates sharding information. This sharding information includes the left boundary value, right boundary value, and high-water mark (SCN). For tables in databases such as DM and Oracle, ROWID can be used as the boundary value for sharding; for tables in other databases, the primary key can be used. The purpose of sharding is to split the table into smaller ranges of query result sets based on boundary values, thus enabling concurrent reading of table shards.

[0053] Finally, a fixed number of initial load threads for each shard are started to perform the initial load on the sharded tables. During the initial load, the group status is updated to "Initial Loading". After each table is initially loaded, the group information is updated. If all tables in a group have been initially loaded, the group status is updated to "Initial Loading Completed".

[0054] In some embodiments of the present invention, the method further includes: Update the status of each group in real time; Therefore, S102 includes: When a group is in the state of initial load complete, the group is used as the target group, and the tables in the target group are incrementally synchronized.

[0055] The status of a group can indicate the progress of data synchronization for that group, allowing for corresponding data synchronization operations to be performed based on the group's status.

[0056] Reference Figure 2 The diagram illustrates a flowchart of another embodiment of the data synchronization method provided by the present invention, the method comprising: S201, divide the tables in the source database into multiple groups, and perform initial loading of the tables in each group.

[0057] S202, the incremental synchronization main thread parses the incremental log of the source database to obtain transactions, and splits the transactions according to the grouping rules to obtain the grouped transactions corresponding to each group.

[0058] The incremental synchronization main thread parses the incremental logs of the source database in real time to obtain transactions, splits the transactions according to the grouping rules to obtain the grouped transactions corresponding to each group, and stores the grouped transactions in the cache.

[0059] S203, when there is a target group that has completed the initial loading, the target group transaction corresponding to the target group is sent to the execution end through the target group execution thread, so that the execution end executes the target group transaction in the target library.

[0060] After the initial loading of a group is completed, the shard loading thread notifies the incremental synchronization main thread. The incremental synchronization main thread then starts the target group execution thread corresponding to the target group that has completed the initial loading, with one group corresponding to one execution thread. The target group execution thread then pulls the target group transaction from the cache and sends the target group transaction to the execution end so that the execution end can execute the target group transaction in the target database.

[0061] This embodiment splits transactions into groups, which reduces the granularity of transaction execution. Transactions in different groups are executed independently, reducing transaction dependencies and waiting, and improving transaction execution efficiency.

[0062] Reference Figure 3This diagram illustrates an incremental synchronization process provided by the present invention. The HEAD group, also known as group 0, is the master group. The master group is a separately reserved group and does not include tables requiring initial loading. It is used to merge groups; all groups will eventually be merged into the master group. After group 1 completes its initial loading, group execution thread 1 retrieves the group transaction from the cache and sends it to the execution end, enabling the execution end to perform database insertion processing based on the group transaction. Similarly, after group 2 completes its initial loading, group execution thread 2 retrieves the group transaction from the cache and sends it to the execution end, enabling the execution end to perform database insertion processing based on the group transaction.

[0063] The thread corresponding to the HEAD group is the incremental synchronization main thread, which is responsible for business logic such as source database log reading, source database log parsing, group transaction caching, and transaction deployment. In the source database log parsing business logic, the incremental synchronization main thread splits transactions based on the group number and the master group number, storing group transactions and master group transactions with different group numbers in a cache. The key value is distinguished by the group number prefix for easy traversal. In the transaction deployment business logic, the incremental synchronization main thread only deploys the master group transactions corresponding to the HEAD group.

[0064] In some embodiments of the present invention, S203 includes: If there is a first transaction to be committed in the target group transaction, and the difference between the commit SCN of the first transaction to be committed and the SCN of the log of the incremental synchronization main thread is less than a preset threshold, or if there is no first transaction to be committed in the target group transaction, then wait until the committed transactions in the target group transaction are completed by the execution end, then merge the target group into the main group, merge the target group transaction into the main group transaction, and update the grouping rules. The main thread incrementally synchronizes the pending transactions in the main group of transactions to the execution end, so that the execution end can execute the pending transactions in the target database.

[0065] In this embodiment, the preset threshold can be 100. An example of merging the target group into the main group is: group 1 is merged into the HEAD group, and tables A and B no longer belong to group 1, but to the HEAD group.

[0066] Incremental synchronization typically employs multi-threaded parallel processing of transactions to accelerate synchronization. However, excessive parallelism can lead to problems such as resource contention between threads and increased costs associated with maintaining transaction order. Therefore, in this embodiment, when the SCN of the first transaction to be committed in the target group is close to the SCN of the incremental synchronization main thread log, it indicates that the group transactions of the target group and the main group transactions are consecutive or overlap in time. Merging the target groups at this time can reduce the number of threads and lower scheduling overhead.

[0067] Similarly, a merge can also be performed when there is no first transaction to be committed in the target group of transactions.

[0068] In some embodiments of the present invention, the method further includes: When there is a first transaction to be committed in the target group transaction, and the difference between the commit SCN of the first transaction to be committed and the SCN of the log of the incremental synchronization main thread is less than a preset threshold, or when there is no first transaction to be committed in the target group transaction, the incremental synchronization main thread is paused. When the grouping rules are updated, the incremental synchronization main thread is awakened.

[0069] When the incremental synchronization main thread parses the logs in the source database to obtain transactions and groups the transactions, if the target group needs to be merged, the incremental synchronization main thread needs to pause and stop parsing the logs. This is because after the target group is merged, both the main group and the target group change. At this time, the logs need to be parsed according to the merged group and the main group. Therefore, the incremental synchronization main thread needs to be woken up after the group merging is completed and the grouping rules are updated.

[0070] Reference Figure 4 This diagram illustrates a grouped asynchronous initial loading process provided by the present invention. After the tables in the source database are grouped, the tables in the group are sharded according to the group's priority, and the shards are initially loaded. Once all shards in a group have completed their initial loading, incremental synchronization is performed on the group. During incremental synchronization, if there are no pending transactions in the group's transactions, or if the commit SCN of the group's execution thread is close to the SCN of the incremental synchronization main thread log, the group is merged into the main group; otherwise, no merging is performed.

[0071] In some embodiments of the present invention, the method further includes: When the target group execution thread starts, the status of the target group is updated to incremental synchronization. When the target group is merged into the main group, the status of the target group is updated to "merged".

[0072] In some embodiments of the present invention, the step of sending the transactions to be executed in the main group of transactions to the execution end through incremental synchronization of the main thread includes: If the target group transaction is merged into the main group transaction and the target group transaction includes a first transaction to be committed, then when the target group transaction is merged into the main group transaction, if there is a second transaction to be committed in the main group transaction that shares the same log sequence number as the first transaction to be committed, the first transaction to be committed will be executed first through the execution end.

[0073] Reference Figure 5This diagram illustrates a merging and grouping transaction execution order provided by the present invention. Group 2 includes tables C and D. After merging the groups, Group 2 may contain a pending transaction 1001. After the merge, transaction 1001 is committed, requiring further processing. Figure 5 As shown, before merging the groups, there is a pending transaction 1001-2. At this time, group 2 is merged into group 0, and subsequent transaction operations will be assigned to 1001-0. When transaction 1001 commits, sub-transactions 1001-2 and 1001-0 are sent out together, and sub-transaction 1001-2 must be executed before 1001-0. If sub-transactions 1001-2 and 1001-0 contain the same table operations, to ensure data integrity, sub-transaction 1001-2 must be executed first.

[0074] In some embodiments of the present invention, the execution end concurrently performs transaction conflict detection for each group. For example... Figure 6 As shown, TRX represents a transaction, and OP represents the operations within a transaction. Transactions from different groups are sent to the execution end and executed concurrently for database insertion. Since they are processed in the same execution queue, transaction conflicts need to be considered. In the transaction execution queue, the commit SCN for transactions within the same group is incremented. When executing transaction operations, the execution end only needs to perform conflict detection with transactions in the same group that have a smaller commit SCN. If no conflict exists, they can be executed in parallel. For transactions from different groups, no transaction conflict detection is required, and they can be executed and inserted into the database in parallel.

[0075] Within the same group, the transaction conflict detection method includes: taking out other transactions with a smaller SCN than the current transaction from the transaction execution queue and recording them as the set of transactions to be detected; traversing the transactions in the set of transactions to be detected in turn and detecting whether there is a conflict between the execution operation of the current transaction and the transaction to be detected.

[0076] For example, if the current transaction is T1, the transaction to be detected is T2, and the table that the current transaction is operating on is TableA, then the transaction conflict detection method is as follows: 1. If the transaction to be tested does not have any operations on TableA, then there is no transaction conflict.

[0077] 2. If the current transaction operation is an INSERT operation on TableA, and transaction T2 only contains INSERT operations on TableA, then there is no transaction conflict.

[0078] 3. If the table has a primary key, and transaction T2 only contains delete operations on that table, then it is determined that there is no conflict.

[0079] 4. If conflict detection is performed based on ROWID, and transaction T2 detects an operation with the same ROWID, the current transaction operation can be executed simply by waiting for transaction T2 to commit.

[0080] In some embodiments of the present invention, since incremental synchronization is performed in real time during the entire data synchronization process, and the sharding information of the tables in the group needs to wait for the initial loading of the tables in the group to be completed before it is generated, the incremental synchronization phase cannot be connected and filtered with the sharding information when reading and parsing the logs. Therefore, the logs can be connected and filtered at the execution end according to the submitted SCN.

[0081] When the execution end processes log operations, it locates the shard information to which the log belongs based on the log's location column value (ROWID for DM and ORALE databases, and the primary key for other databases). The commit SCN of the transaction to which the log belongs is compared with the high-water mark SCN recorded in the shard. If the commit SCN is larger than the shard's high-water mark SCN, the log is valid; otherwise, it should be filtered. Shard information includes left boundary values, right boundary values, and the high-water mark SCN. The purpose of sharding is to split the table into smaller range query result sets based on boundary values, enabling concurrent reading of table shards. The high-water mark SCN is the current SCN of the database recorded when reading the shard, used for subsequent filtering and concatenation of incremental logs from the table corresponding to this shard.

[0082] In some embodiments of the present invention, the step of dividing the tables in the source database into multiple groups includes: Group the primary and secondary tables with primary and foreign key relationships into the same group; The steps for incrementally synchronizing tables in the target group include: Based on foreign key constraints, perform incremental synchronization of the primary and secondary tables in the target group that are related by primary and foreign keys.

[0083] In databases, when primary and foreign key relationships exist between tables, the order of insert, delete, and update operations must follow certain rules to ensure data integrity and consistency. Therefore, when using asynchronous loading for data synchronization: During the initial load phase, the primary and secondary tables with a primary-foreign key relationship should be placed in the same group. The data to be synchronized during the initial load is all INSERT data. Considering the primary-foreign key relationship, data should be inserted into the primary table first, followed by data into the secondary table. The secondary table's foreign key needs to reference the primary key value of the primary table. If the secondary table is inserted first, the foreign key constraint will be violated because there is no corresponding primary key in the primary table. Therefore, in asynchronous initial loads, the primary and secondary tables should be in the same group, and the initial load priority of the primary table is higher than that of the secondary table.

[0084] During incremental synchronization, change logs exist for inserts, deletions, and modifications. For INSERT operations, data should be inserted into the primary table first, then into the secondary table. For UPDATE operations, foreign key values ​​in the secondary table should be updated first, then primary key values ​​in the primary table. If the primary key in the primary table is modified first, the foreign key in the secondary table will lose its reference, leading to data inconsistency. For DELETE operations, related data in the secondary table should be deleted first, then data in the primary table. If data in the primary table is deleted first, records referencing the primary key may still exist in the secondary table, violating foreign key constraints.

[0085] Reference Figure 7 The diagram illustrates a structural schematic of an embodiment of the data synchronization device provided by the present invention. The device 700 includes: The initial loading module 701 is used to divide the tables in the source database into multiple groups and perform initial loading on the tables in each group respectively; The incremental synchronization module 702 is used to perform incremental synchronization of tables in the target group when there is a target group that has completed initial loading.

[0086] It should be noted that the implementation principles or processes of the above modules can be referred to the aforementioned implementation examples of the data synchronization method, and will not be elaborated here.

[0087] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0088] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized by, The method comprises the following steps: dividing tables in a source database into multiple groups, and performing initial loading on tables in each group respectively; when there is a target group that has completed initial loading, performing incremental synchronization on tables in the target group.

2. The data synchronization method of claim 1, wherein, The step of performing initial loading on tables in each group respectively comprises the following steps: setting priorities for each group, and performing initial loading on tables in each group according to the priorities.

3. The data synchronization method of claim 1, wherein, The method further comprises the following steps: updating the state of each group in real time; wherein the state comprises: initial loading completed; The step of, when there is a target group that has completed initial loading, performing incremental synchronization on tables in the target group comprises the following steps: when the state of the group is initial loading completed, taking the group as a target group, and performing incremental synchronization on tables in the target group.

4. The data synchronization method of claim 1, wherein, The method further comprises the following steps: obtaining transactions by parsing the incremental log of the source database through an incremental synchronization main thread, and splitting the transactions according to a group rule to obtain group transactions corresponding to each group; The step of performing incremental synchronization on tables in the target group comprises the following steps: downloading target group transactions corresponding to the target group to an execution end through a target group execution thread corresponding to the target group, so that the execution end executes the target group transactions in a target database.

5. The data synchronization method of claim 4, wherein, The incremental synchronization main thread is further used for splitting the transactions to obtain master group transactions corresponding to a master group; the step of downloading the target group transactions corresponding to the target group to the execution end through the target group execution thread corresponding to the target group comprises the following steps: if there is a first to-be-committed transaction in the target group transactions, and the difference between the commit SCN of the first to-be-committed transaction and the SCN of the log of the incremental synchronization main thread is less than a preset threshold, or there is no first to-be-committed transaction in the target group transactions, then after the committed transactions in the target group transactions are executed by the execution end, the target group is merged into the master group, the target group transactions are merged into the master group transactions, and the group rule is updated; downloading to-be-executed transactions in the master group transactions to the execution end through the incremental synchronization main thread, so that the execution end executes the to-be-executed transactions in the target database.

6. The data synchronization method of claim 5, wherein, The method further comprises the following steps: when there is the first to-be-committed transaction in the target group transactions, and the difference between the commit SCN of the first to-be-committed transaction and the SCN of the log of the incremental synchronization main thread is less than the preset threshold, or there is no first to-be-committed transaction in the target group transactions, the incremental synchronization main thread is paused; after the group rule is updated, the incremental synchronization main thread is woken up.

7. The data synchronization method of claim 5, wherein, The step of downloading to-be-executed transactions in the master group transactions to the execution end through the incremental synchronization main thread comprises the following steps: if the target group transactions include the first to-be-committed transaction when the target group transactions are merged into the master group transactions, and there is a second to-be-committed transaction that shares the same log sequence number with the first to-be-committed transaction in the master group transactions after the target group transactions are merged into the master group transactions, then the first to-be-committed transaction is preferentially executed by the execution end.

8. The data synchronization method of claim 5, wherein, The method further comprises the following steps: when the target group execution thread is started, updating a state of the target group to incremental synchronization in progress; when the target group is merged into the master group, updating the state of the target group to merged.

9. The data synchronization method of claim 1, wherein, the step of dividing the tables in the source database into groups comprises: the master table and the slave table associated with the master foreign key are divided into the same group; the step of performing incremental synchronization on the tables in the target group comprises: the master table and the slave table associated with the master foreign key in the target group are subjected to incremental synchronization according to the foreign key constraint.

10. A data synchronization apparatus, characterized by comprising: the method comprises: an initial loading module, configured to divide the tables in the source database into groups and perform initial loading on the tables in each group respectively; an incremental synchronization module, configured to perform incremental synchronization on the tables in a target group when the target group has completed the initial loading.

Images