A method and apparatus for packet-based data synchronization

Abstract

The application relates to the technical field of database synchronization, and provides a data synchronization method and device with grouping execution. In the application, multi-version grouping management is introduced, the grouping rules can be adjusted at any time in the data synchronization process on the basis of guaranteeing the consistency of the synchronized data and the execution granularity of the split transaction, the convenience and fault tolerance of the synchronization program are improved, and the data synchronization efficiency is improved. By supporting the setting of the execution priority of each group in the grouping rules, the adaptability of the data synchronization to the business demand is improved, and the manual designation of the synchronization sequence is supported in the case of avoiding the running error of the synchronization service.

Description

Technical Field

[0001] This invention relates to the field of database synchronization technology, and in particular to a data synchronization method and apparatus for group execution. Background Technology

[0002] Real-time data synchronization is performed and executed on a transaction-by-transaction basis, with the smallest concurrent granularity being the transaction. Data synchronization is the process of copying changed data from the source database to the destination database in the shortest possible time. A transaction is a sequence of database operations that access and potentially manipulate various data items; these operations are either all executed or none are executed, forming an indivisible unit of work. A transaction consists of all database operations executed between its start and end. When parsing logs, transaction information and operation records are cached on a transaction-by-transaction basis; a transaction operation refers to the database operations contained within a transaction. When the logs detect a transaction commit operation, the corresponding information for the transaction to be committed is sent to the execution end. The execution end then iterates through the cache to retrieve all transaction operations related to that transaction and executes them before writing them into the database. Writing into the database involves executing a specific transaction operation on the destination database to achieve data synchronization.

[0003] Current data synchronization methods group transaction operations within a transaction by table, but do not consider group adjustments or execution priorities. Grouping rules cannot be adjusted; once data synchronization begins, the grouping rules cannot be modified. Furthermore, complex dependencies between transactions result in low convenience and fault tolerance, easily leading to synchronization service errors. The execution order of dependent transactions strictly follows the order of their SCN (System Change Number), increasing waiting time and resulting in low synchronization efficiency. Specifically, if table A in the database is updated by a transaction in transaction 1, and another transaction in transaction 2 continues to update table A based on the updated table A, meaning the corresponding transaction operation in transaction 2 needs to wait for the corresponding transaction operation in transaction 1 to execute successfully, then there is a dependency between these two transactions. Since a transaction is an indivisible unit of work, transaction operations within a transaction cannot execute independently, so transaction 2 needs to wait for transaction 1 to execute successfully, meaning there is a dependency between transaction 2 and transaction 1. Dependencies may exist between different transactions, affecting their execution order. SCN is a sequentially increasing number in the database system used to precisely distinguish the order of operations.

[0004] Since transactions are generally determined based on business needs, a single transaction often involves updating multiple tables, leading to complex dependencies between transactions. When it's necessary to avoid errors in the synchronization service, the only option is to avoid splitting the execution granularity of transactions. This means disallowing independent execution of table updates within a single transaction, and preventing the prioritization of table updates within a specific transaction, further reducing synchronization efficiency. For example, in transaction 1, the operation updating table A is independent, meaning it doesn't depend on other transactions. However, since updating table A is part of transaction 1, it cannot execute independently and is subject to the pre-defined execution order within transaction 1. This causes transactions in transaction 1 that don't depend on each other to wait for each other, wasting execution time. Furthermore, because transaction 1 containing the operation updating table A might also contain operations updating table B, the operation updating table A cannot be prioritized. During execution, transaction 1 and other transactions need to wait for each other, resulting in low synchronization efficiency.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a data synchronization method and apparatus for group execution. Its purpose is to realize the dynamic adjustment of grouping rules, support the setting of execution priorities, execute sub-transactions in each group in parallel and put them into the database, reduce the waiting time for execution transactions, and solve the problem of low data synchronization efficiency under the condition that the synchronization service is running normally.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a data synchronization method for group execution, comprising:

[0009] Obtain grouping rules; wherein each current group corresponds to at least one table, and set the priority of each current group; when the grouping rules change, the changed grouping rules are used as the current version of the grouping rules, and each version of the grouping rules corresponds to a default group;

[0010] Receive logs, parse the logs, and divide the transaction operations into the corresponding current group or default group according to the grouping rules of the current version, generating at least one current sub-transaction; wherein, the transaction operations of the main transaction are grouped according to the tables they depend on;

[0011] When the grouping rules of the current version change before the main transaction is committed, the current group is assigned to the historical group, and the current sub-transaction is assigned to the historical sub-transaction.

[0012] When the commit operation of the main transaction is parsed, at least one sub-transaction message and at least one group queue are constructed according to all historical sub-transactions and all current sub-transactions of the main transaction. The sub-transaction messages of the historical sub-transactions are sent to the group queue of the historical group, and the sub-transaction messages of the current sub-transactions are sent to the group queue of the current group.

[0013] According to the priority, the sub-transaction messages in the corresponding group queues of each version are retrieved in sequence, and the sub-transactions to be entered into the database are obtained and executed according to the sub-transaction messages.

[0014] In a second aspect, the present invention also provides a data synchronization apparatus for group execution, used to implement the data synchronization method for group execution described in the first aspect, the data synchronization apparatus for group execution comprising:

[0015] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the grouped data synchronization method described in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the data synchronization method for group execution described in the first aspect.

[0017] Unlike existing technologies, the present invention has at least the following beneficial effects:

[0018] This invention introduces multi-version group management, which, while ensuring data consistency and splitting transaction execution granularity, allows for adjustments to grouping rules at any time during data synchronization. This improves the convenience and fault tolerance of the synchronization program, as well as data synchronization efficiency. Furthermore, by supporting the setting of execution priorities for each group within the grouping rules, the adaptability of data synchronization to business needs is enhanced. It also supports manually specifying the synchronization order while avoiding errors in the synchronization service.

[0019] Furthermore, by designing transaction conflict detection logic, unnecessary mutual waiting during transaction execution is greatly reduced, improving the efficiency of parallel transaction execution, thereby further improving data synchronization efficiency and broadening the application scenarios of table grouping execution. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1 This is a schematic diagram of the overall process of a data synchronization method for group execution provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the specific process of step 20 in an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the specific process of step 40 in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of a specific process of step 50 in an embodiment of the present invention;

[0025] Figure 5 This is another specific flowchart of step 50 in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the execution order of a multi-version transaction provided by an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram illustrating the specific process of transaction conflict detection according to an embodiment of the present invention;

[0028] Figure 8 This is a schematic diagram of the specific process of fault or anomaly handling in step 50 of an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram of the architecture of a data synchronization device for group execution provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0032] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0033] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] Example 1:

[0035] In the log parsing-based group synchronization method, the destination database receives logs containing multiple source master transactions. Each master transaction contains at least one transaction operation. The master transactions are split into units based on the tables they depend on and classified into corresponding groups. Multi-threaded parallel execution of transactions in each group is used to improve synchronization efficiency.

[0036] There are two existing methods for data synchronization:

[0037] Data synchronization method one: Traditional data synchronization. When parsing transaction operation logs, transaction IDs and corresponding transaction operations are cached as key-value pairs. When a commit operation is encountered, the transaction ID is sent to the transaction commit queue. The execution end concurrently retrieves transactions to be committed from the transaction commit queue, and executes them by traversing the cached transaction operation objects according to the transaction ID. When the transaction is executed and written to the database, if there is another transaction with a commit SCN lower than the commit SCN of the currently executing transaction, it is necessary to determine whether there is a transaction dependency. If a transaction dependency exists, the transaction with the smaller commit SCN must be executed and written to the database first. The queue is a linear data structure that only allows insertion at one end and deletion at the other. Direct access to data outside the two ends is prohibited, and the queue is a first-in, first-out (FIFO) data structure.

[0038] Data synchronization method two: Table grouping execution, but without considering group adjustments and execution priorities. When parsing transaction operation logs, the corresponding group number is looked up based on the table name in the log. Sub-transactions are constructed based on the transaction ID and group number, and the transaction relationship between the main transaction and sub-transactions is maintained. When a transaction commits, all sub-transaction IDs are retrieved from the transaction relationship based on the main transaction ID, and each sub-transaction ID is sent to the corresponding group queue. The group execution thread on the execution end iterates through the cached transaction operations based on the sub-transaction IDs and executes the data insertion. When executing data insertion within a group, transaction dependencies must also be considered. For sub-transactions with dependencies, the sub-transaction with the smaller commit SCN must be executed first. Transactions outside the group do not require consideration of concurrency control between transactions and do not have dependencies.

[0039] In Method 1, the execution granularity of transactions cannot be broken down, making it impossible to execute transactions of individual tables independently or to prioritize the execution of specific tables. Combining operations on multiple tables into a single transaction is inefficient because, in real-world applications, many independent transactions exist within the same transaction, and these independent transactions cannot be executed concurrently. Method 2 groups transactions by table, but the grouping rules cannot be adjusted. Once data synchronization begins, the grouping rules cannot be modified, resulting in low convenience and fault tolerance. Furthermore, it cannot prioritize the execution of certain tables, failing to meet the needs of real-world production environments. In both Methods 1 and 2, to avoid errors in the synchronization service, the execution order of transactions with dependencies strictly follows the order of their commit SCNs at the source. However, in real-world production environments, a database often contains dozens to thousands of tables, and sometimes multiple databases are synchronized, including many transactions with dependencies. If a transaction that could execute in parallel waits, the execution time of multiple dependent transactions is also delayed, resulting in excessive time wasted on mutual waiting and low synchronization efficiency.

[0040] To address the aforementioned problems, Embodiment 1 of the present invention provides a data synchronization method for group execution, such as... Figure 1 As shown, it includes:

[0041] Step 10: Obtain grouping rules; wherein, each current group corresponds to at least one table, and the priority of each current group is set; when the grouping rules change, the changed grouping rules are used as the current version of the grouping rules, and each version of the grouping rules corresponds to a default group.

[0042] Before data synchronization, users need to configure grouping rules. In the grouping rules of the data synchronization method of this embodiment, each group corresponds to at least one table. The mapping relationship between groups and tables is maintained. In an optional embodiment, the group ID of the group and the full name of the table are cached in the form of key-value pairs so that data synchronization can be performed according to the mapping relationship between groups and tables (i.e., grouping rules) during data synchronization.

[0043] Multi-version management of grouping rules is implemented. At the data synchronization destination, the version number of the initial version of the grouping rule is set to 1. Only the initial version of the grouping rule exists at this time, and the current version of the grouping rule is the initial version of the grouping rule. Each time a user modifies a grouping rule, the version number of the current version of the grouping rule before the modification is incremented by 1, becoming the version number of the current version of the grouping rule after the modification. Each time a user configures a grouping rule, a group number is set for each version of the group, with the number starting from 0 and incrementing by 1 each time. Each time a user configures a grouping rule, the priority of each group's execution order is set. In an optional embodiment, the number representing the priority is set to start from 1 and increment by 1 each time, with smaller numbers indicating higher priority. Depending on the actual needs of the data synchronization usage scenario, two or more groups are allowed to have the same priority. The group ID of each version of the group is set as "version number-group number-priority". Within each version, a default group is set, with the default group's group number always being 0. For example, in the initial version of the grouping rule, the default group with priority 3 has a group ID of "1-0-3".

[0044] Step 20: Receive logs, parse the logs, and divide the transaction operations into the corresponding current group or default group according to the grouping rules of the current version, generating at least one current sub-transaction; wherein, the transaction operations of the main transaction are grouped according to the tables they depend on.

[0045] Deploy a data synchronization service on the destination end, which includes a log parsing service. During data synchronization, the destination end receives the logs from the source end and parses them, restoring the binary log data into at least one transaction operation record. The destination end categorizes transactions belonging to a specific main transaction on the source end into the corresponding group based on the current version's grouping rules and the tables the transaction depends on, and then assigns them to the sub-transactions corresponding to that main transaction. If a sub-transaction for that table belonging to that main transaction does not exist, a corresponding sub-transaction is created.

[0046] Step 30: When the grouping rule of the current version changes before the main transaction is committed, the current group is assigned to the historical group and the current sub-transaction is assigned to the historical sub-transaction.

[0047] After a user modifies the grouping rules, logs continue to be received and corresponding operations are performed. After a historical sub-transaction is executed and written to the database, the sub-transaction IDs in the cache are traversed to determine if there exists a historical sub-transaction whose version number matches the version number in the sub-transaction ID. If not, the grouping rules for that historical version are cleared from the cache, releasing the memory space they occupy. Here, "historical version grouping rules" refers to the grouping rules for all versions except the current version.

[0048] Step 40: When the commit operation of the main transaction is parsed, at least one sub-transaction message and at least one group queue are constructed according to all historical sub-transactions and all current sub-transactions of the main transaction. The sub-transaction messages of the historical sub-transactions are sent to the group queue of the historical group, and the sub-transaction messages of the current sub-transactions are sent to the group queue of the current group.

[0049] When the commit operation of the main transaction is parsed, it indicates that the main transaction has been committed on the source side. Because a transaction is an indivisible unit of work, meaning that all transaction operations in the main transaction on the source side are either executed or none are executed, data synchronization for the main transaction can begin at this point. Each group in each version corresponds to a group queue.

[0050] Step 50: According to the priority, retrieve the sub-transaction messages from the corresponding group queues of each version in sequence, obtain the sub-transaction to be entered into the database according to the sub-transaction message, and execute the sub-transaction to be entered into the database.

[0051] By using a queue data structure to store each version of the sub-transaction in the corresponding group queue, it is convenient to add the corresponding sub-transaction to the execution queue and execute and store it in the database according to the commit order of the main transaction's transaction operations at the source.

[0052] To better illustrate the data synchronization method for group execution of the present invention, step 20 of the data synchronization method for group execution of the present invention will be further refined below. Specifically, as follows: Figure 2 As shown, step 20 includes:

[0053] Step 201: Based on the grouping rules of the current version, obtain the first mapping relationship between the full name of the table and the group number of the current group. The full name of the table is a unique identifier for the table. For example, in an Oracle database, different schemas or databases exist, and tables with the same name can exist under different schemas or databases. In an optional embodiment, the first mapping relationship is cached in the form of key-value pairs, and a HashMap (an implementation of the Map interface based on a hash table) is used to store the key-value pair mapping.

[0054] Step 202: Receive the log, parse the log to obtain at least one transaction operation, obtain the full name of the table on which each transaction operation depends, and determine whether there is a corresponding group number in the first mapping relationship based on the full name of the table; if there is, assign the transaction operation to the corresponding current group; if there is no, assign the transaction operation to the default group of the current version.

[0055] If a table exists that cannot be associated with any group in the latest grouping rules, then that table is classified into the default group. In real-world data synchronization scenarios, the number of tables requiring data synchronization from the source end is often large. However, when users set grouping rules based on specific business needs, it is not necessary to specify groups for all tables from the source end. Grouping rules often only involve a few tables, and the transaction operations corresponding to tables not involved in the grouping rules are assigned to the default group. By setting a default group for each version, it is ensured that any group is globally unique on the destination end.

[0056] Step 203: Based on the main transaction of the transaction operation, the version number of the grouping rule of the current version, and the group number, construct the sub-transaction ID of the current sub-transaction; check if there is a current sub-transaction corresponding to the sub-transaction ID; if not, generate the current sub-transaction and add the transaction operation to the current sub-transaction; if it exists, add the transaction operation to the current sub-transaction.

[0057] Each time a log entry is parsed, the group ID (i.e., "version number-group number-priority") is searched in the first mapping relationship of the latest grouping rule based on the full name of the table on which the transaction operation depends in the log. The corresponding group number is then obtained, and the transaction operation is assigned to the group corresponding to that group number. It is then determined whether a sub-transaction belonging to that group exists for the main transaction. If not, a sub-transaction is created and the transaction operation is assigned to that sub-transaction. The main transaction number (i.e., main transaction ID) of the source main transaction corresponding to the transaction operation, the version number of the latest grouping rule, and the group number are used as the sub-transaction ID (i.e., "main transaction number-version number-group number"). If a sub-transaction exists, the transaction operation is assigned to the corresponding sub-transaction.

[0058] Within each of the aforementioned versions' groups, there are sub-transactions of the same version belonging to different main transactions, and the tables that these sub-transactions depend on are all tables specified in the grouping rules of the corresponding version. Each of these sub-transactions contains only transaction operations belonging to the same main transaction.

[0059] After completing step 20, it is also necessary to obtain the operation number of the corresponding transaction operation based on the number of transaction operations contained in the current sub-transaction; construct a mapping relationship between sub-transaction ID and quantity, and construct a mapping relationship between operation number and corresponding transaction operation log, so that the log of the corresponding transaction operation can be obtained according to the sub-transaction ID when performing subsequent data entry.

[0060] In an optional embodiment, the mapping relationship between sub-transaction IDs and quantities is cached in key-value pairs, with the sub-transaction ID as the key and the transaction header information containing the corresponding quantity as the value. Each time a sub-transaction is generated, a corresponding key-value pair is created. Each time a transaction operation is grouped, the corresponding key-value pair is updated, and the quantity of transaction operations within it is incremented by 1. The transaction header information is used to store relevant information about the corresponding transaction, so that information about the transaction and its operations can be obtained based on the transaction header information. Those skilled in the art can determine the content that the transaction header information should include according to different practical application scenarios.

[0061] In an optional embodiment, the mapping relationship between sub-transactions and transaction operation logs is cached in key-value pairs to facilitate finding the specific executable information of the corresponding transaction operation based on the sub-transaction ID. The operation number of the transaction operation is obtained based on its position in the corresponding group queue, using "sub-transaction ID - operation number" as the key and the corresponding log entry as the value. A corresponding key-value pair is created each time a sub-transaction is generated; the corresponding key-value pair is updated each time a transaction operation is grouped. Those skilled in the art can also choose the specific method for recording sub-transaction IDs and transaction operation logs according to different practical application scenarios.

[0062] Step 204: Construct a second mapping relationship between the main transaction ID of the main transaction and the sub-transaction ID; update the second mapping relationship each time a current sub-transaction is generated.

[0063] During log parsing, it is necessary to maintain the mapping relationship between the main transaction ID and the sub-transaction ID. Each time a sub-transaction is generated, a corresponding second mapping relationship is constructed. In an optional embodiment, a key-value pair is added to construct the second mapping relationship. Those skilled in the art can choose the specific method for maintaining the mapping relationship between the main transaction ID and the sub-transaction ID according to different practical use cases, as long as it enables the association of the mapping relationship (including storage, retrieval, and clearing). In an optional embodiment, the mapping relationship can be cached, or a database table can be used to record the mapping relationship.

[0064] Since the logs received by the destination are all the logs generated by the source when performing transaction operations, they include logs of transaction operations that are being executed but not yet committed. In order to maintain database consistency, data synchronization is based on the commits made at the source. That is, the destination only executes and writes transaction operations that have been committed at the source but have not yet been written to the database at the destination, thus completing its data synchronization.

[0065] In an optional embodiment, a log parsing service is deployed at the source end to independently number the parsed transaction operations on a transaction-by-transaction basis, ensuring that the operation numbers of the transaction operations are sequentially incremented within the transaction. The logs corresponding to the transaction operations are then populated with the full names of the tables the transaction operations depend on. The destination-end data synchronization service receives the logs sent from the source end, categorizes the corresponding transaction operations into active transactions by transaction ID, extracts the corresponding full names of the tables, and categorizes the transaction operations into the corresponding groups based on these full names. Under the active transaction, sub-transactions are processed according to the operations in step 20, adding the transaction operation to the corresponding sub-transaction and recording the number of the last transaction operation of each sub-transaction, using this number as the commit number of the sub-transaction. If the corresponding sub-transaction does not exist, it is generated. Specifically, when a single main transaction parsed from the source end depends on more than one table, during destination-end grouping, if the tables depend on more than one table are divided into different groups, an overall active transaction is created to store the sub-transactions corresponding to the main transaction that have not received a commit operation. This collection of sub-transactions manages the sub-transactions divided from the main transaction and categorizes the transaction operations of the main transaction.

[0066] It is worth noting that step 30 includes two scenarios: In the extreme case, when the grouping rules change, either all transaction operations of the main transaction have been completed and entered into the database, or no logs of any transaction operations of the main transaction have been received. The same main transaction does not involve a situation where it was divided according to the grouping rules of a historical version and then divided again according to the current version of the grouping rules. When the grouping rules change, only the transaction operations in the continuously received logs are divided according to the new version of the grouping rules for the main transaction.

[0067] If the main transaction has not yet been committed at the source, and the grouping rules change, the transaction operations already divided within the main transaction remain unchanged, and their versions can be determined based on their sub-transaction IDs. However, for transaction operations received in the logs, the main transaction is further divided according to the current version's grouping rules. That is, if the user modifies the grouping rules before the main transaction is committed at the source, the transaction operations within the main transaction are split into at least two versions. In optional embodiments, the main transaction splitting method in this case includes dynamic modification and static modification methods.

[0068] First, in this embodiment of the invention, the change in grouping rules manifests as a modification of the mapping relationship between groups and tables. For example, in the historical version of the grouping rules, transaction operations dependent on tables T1 and T2 were assigned to group 1, and transaction operations dependent on table T3 were assigned to group 2. In the current version of the grouping rules, transaction operations dependent on table T1 are assigned to group 1, and transaction operations dependent on tables T2 and T3 are assigned to group 2. When the grouping rules change, the destination end, based on the historical and current version of the grouping rules, obtains the tables with group changes as the synchronization tables to be modified. For example, in the aforementioned example, the synchronization table to be modified is T2. Furthermore, to prevent active transactions from being modified by transaction operations in the continuously received logs during transaction splitting, the destination end's reception of logs from the source end needs to be paused before the active transaction can be split.

[0069] Secondly, when the grouping rules change at the destination end, the dynamic modification method performs transaction splitting on the active transactions involving the synchronization table to be modified, constructs virtual transactions to save the modification operations of the grouping rules, and constrains the execution order of the groups before and after the synchronization table to be modified, as follows:

[0070] In the current active transaction, identify all active transactions that depend on the synchronization table to be modified. For each active transaction, split and construct virtual transactions within its current group to constrain the execution order of the groups before and after the synchronization table to be modified. Find all historical groups of the synchronization table to be modified within the current active transaction. If a sub-transaction contains transaction operations that depend on the synchronization table to be modified, classify the active transaction as requiring transaction splitting. Then, find the current group of the synchronization table to be modified within the active transaction. If a corresponding sub-transaction exists, the active transaction is considered to require splitting; other active transactions not involved do not need processing. If no corresponding sub-transaction exists, no processing is required. When an active transaction contains sub-transactions that depend on historical groups of the synchronization table to be modified, mark the sub-transaction as closed. When an active transaction contains sub-transactions that depend on the current version group of the synchronization table to be modified, an end marker is added to the sub-transaction. Based on the last operation number of the active transaction, a virtual transaction is constructed for the current group containing the synchronization table to be modified. The virtual transaction's operation is as follows: wait for the sub-transactions of the historical group to execute up to the commit SCN and commit number of this transaction, set the commit number of the virtual transaction to the last operation number of the active transaction, and add the group rule modification operation for the synchronization table to be modified to the virtual transaction. All sub-transactions of the active transaction involved are divided according to the above steps, completing the modification of all related active transactions. A checkpoint is then performed on the transaction state formed after the modification in the current system, saving the relevant information of the above transaction operations to the corresponding checkpoint file to facilitate restoration to the system's operating state before the failure in the event of a synchronization service failure. The group configuration of the synchronization table to be modified is modified and saved.

[0071] The specific differences between static and dynamic modification methods are as follows:

[0072] Perform a checkpoint action, saving the group information of each sub-transaction in all current active transactions and the table information involved in the sub-transactions, then exit the synchronization service. Modify the group configuration of the synchronization table to be modified. Start the synchronization service, recover the active transactions from the last checkpoint, including the active transactions and their corresponding sub-transactions. The sub-transactions include the grouping information when the synchronization service was last stopped and the full names of the tables the sub-transactions depend on. Iterate through each active transaction for sub-transactions without an end marker, and based on the full names of the tables the sub-transactions depend on, assign the sub-transaction to the corresponding group in the modified current version of the grouping rules. If the group assigned to the sub-transaction is the same as the group saved in the checkpoint, no change is needed; if the group assigned to the sub-transaction is different from the group saved in the checkpoint, mark the sub-transaction with an end marker, obtain the last operation number of the active transaction, and construct a virtual transaction for the current group containing the synchronization table to be modified. This virtual transaction is the same as the one in the dynamic modification method. Complete the above steps to cut all the sub-transactions of all involved active transactions, thus completing the modification of all related active transactions. Next, perform a checkpoint on the transaction state formed after the modification in the current system, and save the relevant information of the above transaction operations to the corresponding checkpoint file so as to restore the system running state before the failure after the synchronization service fails.

[0073] Finally, both static and dynamic modification methods perform the following operations:

[0074] After completing the startup of the destination synchronization service, continue receiving new logs. After receiving the commit message from the main transaction, the segmented active transaction iterates through the sub-transactions of the main transaction, retrieves the virtual transaction within the sub-transactions, and sets the commit SCN of the main transaction's commit operation to the operation executed by the sub-transaction in the current group before modification. This ensures that when the virtual transaction executes, it must wait for the synchronization of the sub-transaction in the historical group whose commit SCN is less than or equal to the current commit number before it can begin execution.

[0075] When the commit operation of the main transaction is parsed, data synchronization for that main transaction begins. Specifically, as follows: Figure 3 As shown, step 40 includes:

[0076] Step 401: When the commit operation is parsed, obtain all the sub-transaction IDs corresponding to the main transaction based on the main transaction ID and the second mapping relationship.

[0077] Step 402: Construct the corresponding sub-transaction message based on all the sub-transaction IDs.

[0078] Step 403: Based on the version number and group number of all the sub-transaction IDs, generate the corresponding historical group queue and the corresponding current group queue. The sub-transaction IDs are sent to the corresponding group queues in the form of lightweight messages. By generating the corresponding group queues only after parsing the commit operation of the main transaction, memory space occupied by data synchronization is saved.

[0079] Step 404: Send the sub-transaction message of the historical sub-transaction to the group queue of the historical group, and send the sub-transaction message of the current sub-transaction to the group queue of the current group.

[0080] Retrieve all sub-transaction IDs from the second mapping relationship based on the main transaction ID, and construct a sub-transaction message object. Send each version of the sub-transaction message to the corresponding group queue. The group queues are categorized by the version number in the sub-transaction ID. Sub-transaction messages corresponding to historical sub-transactions enter the historical group's group queue, and sub-transaction messages corresponding to the current sub-transaction enter the current group's group queue.

[0081] After a sub-transaction message is sent to the group queue, the sub-transactions to be imported are retrieved from the group queue. To avoid errors in the synchronization service, these sub-transactions are executed concurrently according to the version of the grouping rules and the corresponding group priority. Within each group, the transaction operations corresponding to the sub-transactions to be imported are executed sequentially according to the order of commits from the source end. Specifically, for example... Figure 4 As shown, step 50 includes:

[0082] Step 501a: Traverse the group queues of each version in ascending order of version number.

[0083] Step 502a: When there are sub-transactions to be added to the database in the current group's group queue, traverse the current version's group queue in descending order of priority.

[0084] Step 503a: When there are sub-transactions to be added to the database in the current priority group queue, pull sub-transaction messages sequentially from the corresponding group queue, obtain the corresponding transaction operation of the sub-transaction to be added to the database according to the corresponding sub-transaction message, and execute the transaction operation.

[0085] Step 504a: When there are group queues with the same priority, the commit SCNs of the sub-transactions to be added to the database in the group queues are sorted, and the group queues are processed accordingly in ascending order of the commit SCNs.

[0086] After the sub-transaction message parsed from the main transaction's commit operation is sent to the group queue, the execution thread of the data synchronization method for group execution in this embodiment of the invention sequentially pulls the transactions to be committed from the queue according to the version number order and priority of the group queue. It then iterates through the transaction operations in the cache according to the sub-transaction ID, executes the transaction operations, and commits them. After executing the commit operation of the sub-transaction to be added to the database, the transaction cleanup thread is notified to clean up the cache related to the sub-transaction to be added to the database. Specifically, if the group queue of a historical group is empty, the group queue of the current group is pulled. Within the same version, the group queue with the higher priority is pulled first; if the higher priority group queue is empty, the data from the queue with the lower priority is pulled. If there are group queues with the same priority, the commit SCN of the sub-transactions in the two queues is compared, and the sub-transaction with the smaller commit SCN is processed first.

[0087] When executing a sub-transaction to be inserted into the database, it is necessary to determine whether the sub-transaction depends on the results of other sub-transactions. Specifically, such as... Figure 5 As shown, step 50 further includes:

[0088] Step 501b: Add the sub-transaction to be entered into the database to the execution queue.

[0089] Step 502b: Before executing the sub-transaction to be entered into the database, if there is another sub-transaction to be entered into the database in the execution queue with a commit SCN lower than that of the sub-transaction to be entered into the database, except when the other sub-transactions to be entered into the database have the same version and different groupings as the sub-transaction to be entered into the database, a transaction conflict detection is performed on the other sub-transactions to be entered into the database and the sub-transaction to be entered into the database to determine whether there is a transaction dependency.

[0090] Step 503b: If transaction dependencies exist, execute the sub-transactions to be added to the database sequentially in ascending order of version.

[0091] Step 504b: If there is no transaction dependency, then execute the other pending sub-transactions and the pending sub-transactions in parallel.

[0092] When a sub-transaction to be inserted into the database is executed, sub-transactions of the same version but different groups do not need to consider transaction dependencies and waiting because there are no transaction dependencies between different groups. For sub-transactions of different versions belonging to the same main transaction, the current version of the sub-transaction to be inserted into the database needs to consider whether there is a transaction dependency with the historical version of the sub-transaction to be inserted into the database. If there is a transaction dependency, it is necessary to wait for the historical version of the sub-transaction to be inserted into the database to be executed first.

[0093] For example, such as Figure 6As shown, during data synchronization, the grouping rules of main transaction 1001 are adjusted, changing from version 1 to version 2. The transaction operations of transaction 1001 are divided into sub-transactions of version 1 and version 2. According to the version 1 grouping rules, the main transaction's operations are divided into sub-transactions 1001-1-1 and 1001-1-2; according to the version 2 grouping rules, they are divided into sub-transactions 1001-2-1 and 1001-2-2. Sub-transactions 1001-1-1 and 1001-1-2 belong to the same version but different groups, so no transaction dependency is considered between them; sub-transactions 1001-2-1 and 1001-2-2 belong to the same version but different groups, and no transaction dependency is considered between them either. Since the version numbers of subtransactions 1001-1-1 and 1001-1-2 are less than the version numbers of subtransactions 1001-2-1 and 1001-2-2, subtransactions 1001-2-1 and 1001-2-2 need to wait for subtransactions 1001-1-1 and 1001-1-2 to complete before they can be executed.

[0094] Because when sub-transactions of the same version and group are executed, or when sub-transactions of different versions are executed, the sub-transaction with a larger commit SCN depends on the sub-transaction with a smaller commit SCN for execution. If the sub-transaction with a smaller commit SCN is executed completely before the sub-transaction with a larger commit SCN is executed sequentially, it will affect the execution efficiency. Therefore, the data synchronization method for grouped execution in this embodiment of the invention performs transaction conflict detection on the sub-transactions to be imported in the execution queue. If there is no transaction conflict between two transactions, they can be executed in parallel.

[0095] In each version's default group, sub-transactions awaiting data entry that have passed transaction conflict detection are executed in parallel. When synchronizing data across a large number of tables, due to business needs, only a small number of tables are typically grouped each time, with the majority being assigned to the default group. Therefore, this embodiment of the invention significantly improves the overall data synchronization efficiency by enhancing the synchronization efficiency of the default group.

[0096] To better illustrate the data synchronization method for group execution of the present invention, the transaction conflict detection method of the data synchronization method for group execution of the present invention will be further refined below. Specifically, as follows: Figure 7 As shown, the transaction conflict detection includes:

[0097] Step 5021: Iterate through all the other sub-transactions to be added to the database in the execution queue, treat the other sub-transactions to be added to the database as transactions to be detected, and treat the sub-transactions to be added to the database as the current transaction.

[0098] Step 5022: When performing transaction conflict detection based on rowid, if the log of the transaction to be detected contains the same rowid as the log of the current transaction, the transaction operation of the current transaction is executed after the transaction operation of the transaction to be detected is completed.

[0099] Step 5023: When performing transaction conflict detection based on transaction dependencies, if the table on which the transaction to be detected depends is different from the table on which the current transaction depends, there is no transaction dependency; if the table on which the transaction to be detected depends is the same as the table on which the current transaction depends, the existence or non-existence of transaction dependency is selectively determined according to the type of transaction operation therein.

[0100] There is no transaction dependency when both the transaction to be detected and the current transaction only perform insert operations on the table; there is no transaction dependency when the table has a primary key and the transaction to be detected only performs delete operations on the table.

[0101] Transaction conflict detection includes checks for inserts, reads, and updates on the same table, as well as rowid conflict detection. A rowid is a pseudo-column used to uniquely identify rows in a table. It's the internal address of the row's data in the physical table, containing two addresses: one pointing to the data file containing the block of the table, and the other directly locating the row within the data block. In Oracle, each record has a unique rowid, which identifies each record within the entire database and determines which data file, block, and row it belongs to. When performing insert, read, and update checks on the same table, if the table lacks a primary key, a delete operation may result in the deletion of multiple records from the table.

[0102] The data synchronization method for group execution in this embodiment of the invention provides an example of transaction conflict detection, as follows:

[0103] Take other transactions from the execution queue whose commit SCN is smaller than that of the current transaction and denote them as the set of transactions to be detected. Iterate through the transactions to be detected in the set and check whether there is a transaction conflict between the execution operations of the current transaction and the execution operations of the transactions to be detected. Assume that the current transaction is T1, the transaction to be detected is T2, and the table that the current transaction is operating on is TableA.

[0104] When performing transaction conflict detection based on insert, delete, and update operations on the same table, if transaction T2 does not contain any transaction operations targeting TableA, then there is no transaction conflict. If transaction T1 involves an INSERT operation on TableA, and transaction T2 only contains INSERT operations on TableA, then there is no transaction conflict. If TableA has a primary key, and transaction T2 only contains delete operations on TableA, then there is no transaction conflict.

[0105] When performing transaction conflict detection based on rowid, if transaction T2 detects an operation with the same rowid, transaction T1 can execute simply by waiting for transaction T2 to commit.

[0106] To better illustrate the data synchronization method for group execution of the present invention, step 50 of the data synchronization method for group execution of the present invention will be further refined below. Specifically, when the data synchronization service restarts after a failure, as follows: Figure 8 As shown, step 50 further includes:

[0107] Step 501c: Create a checkpoint file. After each execution of a sub-transaction to be inserted into the database, write the sub-transaction ID of the sub-transaction to be inserted into the checkpoint file.

[0108] Step 502c: Each time a new version of the grouping rules is obtained, the sub-transaction IDs of the sub-transactions to be entered into the database that have been executed in the current system are written into the checkpoint file.

[0109] During fault recovery, based on the sub-transaction ID in the checkpoint file, the executed sub-transaction to be imported is loaded; when executing the sub-transaction to be imported, it is determined whether the commit SCN of the sub-transaction to be imported is less than or equal to the commit SCN of the executed sub-transaction to be imported; if so, the sub-transaction to be imported is discarded directly; if not, the sub-transaction to be imported is executed.

[0110] During data synchronization, when a runtime exception (i.e., an exception inherited from RuntimeException) is thrown from the source end, the transaction will automatically roll back to control and maintain the consistency and integrity of each transaction operation within the transaction. When a rollback operation of the main transaction is parsed, the logs corresponding to all historical sub-transactions and all current sub-transactions in the cache are cleared; when a partial rollback operation of the main transaction is parsed, the logs in the cache of all historical sub-transactions and all current sub-transactions whose commit SCN is greater than or equal to the commit SCN of the partial rollback operation are cleared.

[0111] In all the above operations, those skilled in the art can specify and use either the submitted LSN (Log sequence number) or the submitted SCN as a unified standard for identifying the timing of data synchronization in the entire database, depending on the specific production environment.

[0112] It is worth noting that, taking data synchronization as an example, this invention proposes an improvement to the execution of multi-version table group transactions in the data synchronization process. In actual production, the data synchronization method of group execution in this invention can be applied not only to data synchronization scenarios, but also to data migration and some large transaction processing application scenarios.

[0113] The following is a brief description of the application of the grouped execution data synchronization method of the present invention in different applicable scenarios: For business scenarios such as data synchronization, data migration, and large transaction processing, the grouped execution approach can be used for optimization. When the performance of transaction processing reaches a bottleneck, grouped execution can be considered. Transaction operations are split into groups based on the table, and sub-transactions in different groups are executed concurrently to improve transaction execution efficiency. If there is a requirement in the business scenario that some tables need to be executed first, a group priority setting method can be used to place these tables in higher-priority groups. The execution thread prioritizes pulling transactions from the higher-priority group queues to ensure these transactions are executed first. If the table grouping rules in the business scenario change midway, a multi-version grouping management mechanism is introduced to support the adjustment of grouping rules. Specific methods for data synchronization examples are described in the embodiments of the present invention. During data migration, the problem of a large number of tables and large amounts of data in a single table is often encountered. Grouping tables, separating large tables and partitioned tables into separate groups, and configuring higher priorities can fully utilize system resources and improve data migration efficiency. When a transaction is executed and written to the database, if the transaction contains too many transaction operations and the processing time is too long, the transaction operations in the transaction are grouped. Transaction operations of related tables are grouped into one group, and transaction operations of unrelated tables are grouped into several groups. Large transactions are broken down into small transactions and executed concurrently to reduce the processing time of large transactions and improve efficiency.

[0114] Example 2:

[0115] Based on Embodiment 1 above, this embodiment of the invention provides a specific example of dividing groups and executing transactions in a data synchronization method for group execution, in order to better understand the synchronization process. This embodiment of the invention will use the log sequence shown in the table below as an example for illustration:

[0116]

[0117] For ease of description, the table names in the above table are used as the full table names in this embodiment of the invention; transaction operations with SCNs of 8 to 94 and 100 and above are omitted, and none of the corresponding transaction operations belong to the main transaction with a main transaction ID of 1; and when describing the division of transaction operations, transaction operations with SCNs of 8 to 94 and 100 and above and main transaction IDs of 2 and 3 are omitted.

[0118] The destination obtains the latest grouping rules, i.e., grouping rule version 1: Tables A, B, and C correspond to group 1, with priority 1; table D corresponds to group 2, with priority 2; and table E corresponds to group 3, with priority 3. The group ID of group 1 is 1-1-1, the group ID of group 2 is 1-2-2, and the group ID of group 3 is 1-3-3, where smaller numbers indicate higher priority.

[0119] Upon receiving the above log sequence, the destination end, based on the latest grouping rules, divides the transaction operations into the corresponding current group and generates at least one current sub-transaction.

[0120] Transactions involving tables A, B, and C are grouped into group 1-1-1; transactions involving table D are grouped into group 1-2-2; and transactions involving table E are grouped into group 1-3-3.

[0121] For a transaction operation with SCN 1, the log contains table A, finds group number 1, and generates a subtransaction with subtransaction ID 1-1-1;

[0122] For the transaction operation with SCN 2, the log contains table B, finds group number 1, and generates a subtransaction with subtransaction ID 2-1-1;

[0123] For a transaction operation with SCN 3, the log contains table C, finds group number 1, and generates a subtransaction with subtransaction ID 3-1-1;

[0124] For the transaction operation with SCN 4, the log contains table B, finds group number 1, and assigns it to the subtransaction with subtransaction ID 1-1-1;

[0125] For the transaction operation with SCN 5, the log contains table D, finds group number 2, and generates a subtransaction with subtransaction ID 3-1-2;

[0126] For the transaction operation with SCN 6, the log contains table A, finds group number 1, and assigns it to the subtransaction with subtransaction ID 1-1-1;

[0127] For the transaction operation with SCN 7, the log contains table B, finds group number 2, and generates a subtransaction with subtransaction ID 2-1-2.

[0128] For the transaction operation with SCN 95, the log contains table A, which finds group number 1 and assigns it to the subtransaction with subtransaction ID 1-1-1.

[0129] The cache contains multiple key-value pairs, as shown in the table below, containing subtransaction IDs and the number of transaction operations within each subtransaction:

[0130] 1-1-1 4 2-1-1 1 3-1-1 1 3-1-2 1 2-1-2 1

[0131] The following table shows the cached key-value pairs of sub-transaction IDs, transaction operation numbers, and transaction operation logs:

[0132] 1-1-1-1 Logs with SCN of 1 2-1-1-1 Logs with SCN of 2 3-1-1-1 Logs with SCN of 3 1-1-1-2 Logs with SCN of 4 3-1-2-1 Logs with SCN of 5 1-1-1-3 Logs with SCN of 6 2-1-2-1 Logs with SCN of 7 1-1-1-4 Logs with an SCN of 95

[0133] The following table shows the cached key-value pairs of main transaction IDs and sub-transaction IDs:

[0134] 1 1-1-1 2 2-1-1 3 3-1-1 3 3-1-2 2 2-1-2

[0135] Before parsing the log with commit SCN 95, obtain the new grouping rules, i.e., grouping rule version 2: Tables A and B correspond to group 1, with priority 1; table C corresponds to group 2, with priority 2; and tables D and E correspond to group 3, with priority 3. The group ID for group 1 is 2-1-3, the group ID for group 2 is 2-2-2, and the group ID for group 3 is 2-3-1. Compared to grouping rule 1, grouping rule 2 modifies table C to group 2, modifies table D to group 3, changes the priority of group 1 to 3, and changes the priority of group 3 to 1.

[0136] For the transaction operation with SCN 96, the log contains table E, finds group number 3, and generates a subtransaction with subtransaction ID 1-2-3;

[0137] For the transaction operation with SCN 97, the log contains table E, finds group number 3, and assigns it to the subtransaction with subtransaction ID 1-2-3;

[0138] For the transaction operation with SCN 98, the log contains table A, finds group number 1, and assigns it to the subtransaction with subtransaction ID 1-2-1;

[0139] For the transaction operation with SCN 99, the log contains table E, which finds group number 3 and assigns it to the subtransaction with subtransaction ID 1-2-3.

[0140] The cache contains multiple key-value pairs, as shown in the table below, containing subtransaction IDs and the number of transaction operations within each subtransaction:

[0141] 1-2-1 1 1-2-3 3

[0142] The following table shows the cached key-value pairs of sub-transaction IDs, transaction operation numbers, and transaction operation logs:

[0143] 1-2-3-1 Logs with SCN of 96 1-2-3-2 Logs with SCN of 97 1-2-1-1 Logs with SCN of 98 1-2-3-3 Logs with SCN of 99

[0144] The following table shows the cached key-value pairs of main transaction IDs and sub-transaction IDs:

[0145] 1 1-2-1 1 1-2-3

[0146] The commit SCN of a subtransaction is the SCN of the last transaction operation contained within the subtransaction.

[0147] After dividing the above transaction operations, upon receiving the commit operation of the main transaction with main transaction ID 1 from the source end, retrieve all sub-transaction IDs with main transaction ID 1 from the cache, namely 1-1-1, 1-2-1, and 1-2-3. Construct group queue 1 for version 1, and group queues 1 and 3 for version 2. Generate sub-transaction messages for group queue 1 of version 1, containing sub-transaction ID 1-1-1; generate sub-transaction messages for group queue 1 of version 2, containing sub-transaction ID 1-2-1; generate sub-transaction messages for group queue 3 of version 2, containing sub-transaction ID 1-2-3. Distribute the corresponding sub-transaction messages to the respective group queues.

[0148] The execution thread executes the sub-transaction with the main transaction ID 1. First, it pulls sub-transaction 1-1-1 from group queue 1 of version 1 and adds it to the execution queue. Since group 1 has a priority of 3 and group 3 has a priority of 1 in version 2, sub-transaction 1-2-3 is added to the execution queue first, and then sub-transaction 1-2-1 is added to the execution queue.

[0149] The execution thread pulls sub-transaction 1-1-1 from the execution queue. Since the commit SCN of sub-transaction 1-1-1 is 95, the commit SCN of sub-transaction 1-2-3 is 99, and the commit SCN of sub-transaction 1-2-1 is 98, there are no sub-transactions with a commit SCN less than 95 in the execution queue, so sub-transaction 1-1-1 is executed.

[0150] The execution thread pulls sub-transaction 1-2-3 from the execution queue. The commit SCN of sub-transaction 1-2-1 in the execution queue is less than the commit SCN of sub-transaction 1-2-3, but sub-transaction 1-2-3 and sub-transaction 1-2-1 have the same version but different groupings, so no transaction conflict detection is required. The commit SCN of the currently executing sub-transaction 1-1-1 is less than the commit SCN of sub-transaction 1-2-3, so transaction conflict detection is performed.

[0151] Transaction conflict detection is performed based on transaction dependencies. The transaction operations in sub-transaction 1-1-1 depend on tables A and B, and the transaction operations in sub-transaction 1-2-3 depend on table E. There is no transaction dependency between sub-transaction 1-1-1 and sub-transaction 1-2-3. Sub-transaction 1-1-1 and sub-transaction 1-2-3 are executed in parallel.

[0152] The execution thread pulls sub-transaction 1-2-1 from the execution queue. Since the commit SCN of the currently executing sub-transaction 1-1-1 is less than that of sub-transaction 1-2-1, a transaction conflict detection is performed. Transaction conflict detection is then performed based on transaction dependencies. Sub-transaction 1-1-1's transaction operations depend on tables A and B, while sub-transaction 1-2-1's transaction operations depend on table A. Since the dependent tables are the same, further judgment is made based on the type of transaction operations. Because sub-transaction 1-2-1 is the current transaction and sub-transaction 1-1-1 is the transaction to be detected in this transaction conflict detection, there is a transaction dependency between sub-transaction 1-2-1 and sub-transaction 1-1-1. Sub-transaction 1-1-1 is executed first, and then sub-transaction 1-2-1 begins execution.

[0153] Example 3:

[0154] like Figure 9 The diagram shown is an architectural schematic of a data synchronization device for group execution according to an embodiment of the present invention. The data synchronization device for group execution in this embodiment includes one or more processors 31 and a memory 32. Figure 9 Take a processor 31 as an example.

[0155] Processor 31 and memory 32 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0156] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the grouped data synchronization method in Embodiment 1. The processor 31 executes the grouped data synchronization method by running the non-volatile software program and instructions stored in the memory 32.

[0157] Memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 32 may optionally include memory remotely located relative to processor 31, which can be connected to processor 31 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0158] The program instructions / modules are stored in the memory 32. When executed by one or more processors 31, they perform the grouped execution data synchronization method described in Embodiment 1 above, for example, the method described above. Figures 1-5 and Figures 7-8 The steps shown.

[0159] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0160] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data synchronization method for group execution, characterized in that, include: Obtain grouping rules; wherein each current group corresponds to at least one table, and set the priority of each current group; when the grouping rules change, the changed grouping rules are used as the current version of the grouping rules, and each version of the grouping rules corresponds to a default group; According to the grouping rules of the current version, obtain the first mapping relationship between the full name of the table and the group number of the current group; receive logs, parse the logs to obtain at least one transaction operation, obtain the full name of the table that each transaction operation depends on, and determine whether there is a corresponding group number in the first mapping relationship based on the full name of the table; if there is, assign the transaction operation to the corresponding current group; if there is no, assign the transaction operation to the default group of the current version; construct the sub-transaction ID of the current sub-transaction based on the main transaction of the transaction operation, the version number of the grouping rules of the current version, and the group number; check whether there is a current sub-transaction corresponding to the sub-transaction ID; if there is no, generate the current sub-transaction and add the transaction operation to the current sub-transaction; if there is, add the transaction operation to the current sub-transaction; construct the second mapping relationship between the main transaction ID of the main transaction and the sub-transaction ID; update the second mapping relationship each time a current sub-transaction is generated; wherein, the transaction operations of the main transaction are grouped according to the tables they depend on. When the grouping rules of the current version change before the main transaction is committed, the current group is assigned to the historical group, and the current sub-transaction is assigned to the historical sub-transaction. When the commit operation of the main transaction is parsed, at least one sub-transaction message and at least one group queue are constructed according to all historical sub-transactions and all current sub-transactions of the main transaction. The sub-transaction messages of the historical sub-transactions are sent to the group queue of the historical group, and the sub-transaction messages of the current sub-transactions are sent to the group queue of the current group. According to the priority, the sub-transaction messages in the corresponding group queues of each version are retrieved in sequence, and the sub-transactions to be entered into the database are obtained and executed according to the sub-transaction messages.

2. The data synchronization method for group execution according to claim 1, characterized in that, When the commit operation of the main transaction is parsed, at least one sub-transaction message and at least one group queue are constructed based on all historical sub-transactions and all current sub-transactions of the main transaction. The sub-transaction messages of the historical sub-transactions are sent to the group queues of the historical groups, and the sub-transaction messages of the current sub-transactions are sent to the group queues of the current groups, including: When the commit operation is parsed, all sub-transaction IDs corresponding to the main transaction are obtained according to the main transaction ID and the second mapping relationship; Based on all the sub-transaction IDs, construct the corresponding sub-transaction message; Based on the version number and group number of all the sub-transaction IDs, generate the corresponding historical group group queue and the corresponding current group group queue; The sub-transaction messages of the historical sub-transactions are sent to the group queue of the historical group, and the sub-transaction messages of the current sub-transactions are sent to the group queue of the current group.

3. The data synchronization method for group execution according to claim 2, characterized in that, The step of sequentially retrieving the sub-transaction messages from the corresponding group queues of each version according to the priority, obtaining the sub-transaction to be inserted into the database according to the sub-transaction messages, and executing the sub-transaction to be inserted into the database includes: The group queues of each version are traversed sequentially in ascending order of version number; When there are sub-transactions to be added to the database in the current group's queue, the current version's queue is traversed in descending order of priority. When there are sub-transactions to be added to the database in the current priority group queue, sub-transaction messages are pulled sequentially from the corresponding group queue, and the transaction operation of the corresponding sub-transaction to be added to the database is obtained according to the corresponding sub-transaction message and the transaction operation is executed. When there are group queues with the same priority, the commit SCNs of the sub-transactions to be added to the database in the group queues are sorted, and the group queues are processed accordingly in ascending order of the commit SCNs.

4. The data synchronization method for group execution according to claim 3, characterized in that, The step of sequentially retrieving the sub-transaction messages from the corresponding group queues of each version according to the priority, obtaining the sub-transaction to be imported according to the sub-transaction messages, and executing the sub-transaction to be imported further includes: Create a checkpoint file. After each sub-transaction to be inserted is executed, write the sub-transaction ID of the sub-transaction to be inserted into the checkpoint file. Each time a new version of the grouping rules is obtained, the sub-transaction IDs of the sub-transactions to be added to the database that have been executed in the current system are written into the checkpoint file.

5. The data synchronization method for group execution according to claim 1, characterized in that, The step of sequentially retrieving the sub-transaction messages from the corresponding group queues of each version according to the priority, obtaining the sub-transaction to be imported according to the sub-transaction messages, and executing the sub-transaction to be imported further includes: Add the sub-transactions to be entered into the database to the execution queue; Before executing the sub-transaction to be inserted, if there is another sub-transaction to be inserted in the execution queue with a commit SCN that is less than the commit SCN of the sub-transaction to be inserted, except when the other sub-transactions to be inserted have the same version and different groupings as the sub-transaction to be inserted, a transaction conflict detection is performed on the other sub-transactions to be inserted and the sub-transaction to be inserted to determine whether there is a transaction dependency. If transaction dependencies exist, the sub-transactions to be inserted into the database will be executed sequentially in ascending order of version. If there are no transaction dependencies, the other pending sub-transactions and the pending sub-transactions are executed in parallel.

6. The data synchronization method for group execution according to claim 5, characterized in that, The transaction conflict detection includes: The process involves sequentially traversing all other sub-transactions awaiting entry into the database in the execution queue, treating each of these sub-transactions as a transaction to be detected, and treating the current transaction as the one awaiting entry into the database. When performing transaction conflict detection based on rowid, if the log of the transaction to be detected contains the same rowid as the log of the current transaction, the transaction operation of the current transaction will be executed after the transaction operation of the transaction to be detected is completed. When performing transaction conflict detection based on transaction dependencies, if the table on which the transaction to be detected depends is different from the table on which the current transaction depends, there is no transaction dependency; if the table on which the transaction to be detected depends is the same as the table on which the current transaction depends, the existence or non-existence of transaction dependency is selectively determined according to the type of transaction operation therein.

7. The data synchronization method for group execution according to claim 6, characterized in that, When the table on which the transaction to be detected depends is the same as the table on which the current transaction depends, selectively determining whether a transaction dependency exists or not, based on the type of transaction operation therein, includes: There is no transaction dependency when both the transaction to be detected and the current transaction only have insert operations on the table; When the table has a primary key, and the transaction to be tested only involves deletion operations on the table, there is no transaction dependency.

8. The data synchronization method for group execution according to any one of claims 1-7, characterized in that, The step of assigning the current group to a historical group and the current sub-transaction to a historical sub-transaction when the grouping rule of the current version changes before the main transaction is committed includes: During fault recovery, based on the sub-transaction ID in the checkpoint file, the executed sub-transaction to be inserted is loaded; when executing the sub-transaction to be inserted, it is determined whether the commit SCN of the sub-transaction to be inserted is less than or equal to the commit SCN of the executed sub-transaction to be inserted; if so, the sub-transaction to be inserted is discarded directly; if not, the sub-transaction to be inserted is executed. When a rollback operation of the main transaction is parsed, clear the logs of all corresponding historical sub-transactions and all corresponding current sub-transactions in the cache; when a partial rollback operation of the main transaction is parsed, clear the logs of all corresponding historical sub-transactions and all corresponding current sub-transactions in the cache whose commit SCN is greater than or equal to the commit SCN of the partial rollback operation.

9. A data synchronization device for group execution, characterized in that, It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to perform the data synchronization method of group execution as described in any one of claims 1-8.

Images