A method and apparatus for modifying data synchronization packets statically

Abstract

The application relates to the technical field of database synchronization, and provides a method and device for modifying data synchronization grouping statically. According to the table information related to all to-be-cut sub-transactions and a new grouping configuration, the target grouping corresponding to the to-be-cut sub-transactions is obtained, the to-be-modified table of the current modification grouping configuration is determined according to the table corresponding to the different table information of the target grouping and the original grouping, and the to-be-cut active transaction of the corresponding source end master transaction is determined. According to the original grouping of the to-be-modified table and the target grouping of the to-be-modified table, a virtual transaction of the to-be-modified table is constructed, the virtual transaction is taken as a sub-transaction of the corresponding to-be-cut active transaction, and all the sub-transactions of the to-be-cut active transaction are stored. By setting the active transaction and the virtual transaction, corresponding operations are stored according to the order of operations in the master transaction at the target end, the modification of the grouping configuration of the to-be-modified table between groupings is completed without affecting data synchronization.

Description

Technical Field

[0001] This invention relates to the field of database synchronization technology, and in particular to a method and apparatus for statically modifying data synchronization groups. Background Technology

[0002] Currently, in real-time database data synchronization based on log parsing, to ensure database synchronization performance and improve parallelism, operations are often grouped according to the tables they depend on, and then the operations in different groups are synchronized separately. The tables updated by an operation are the tables it depends on. In actual production environments, grouping configurations and data synchronization strategies are often customized based on specific business needs to meet business requirements or improve the data synchronization performance of individual groups. Over time, business requirements often change, and the grouping configuration of the tables to be synchronized also needs to be changed accordingly; that is, during data synchronization, the groups to which operations dependent on the tables to be synchronized are assigned are modified.

[0003] During the operation of the data synchronization service, the latency of data synchronization among different groups may vary. Some groups may continuously synchronize data, while others may experience significant latency due to differences in synchronization efficiency. In this case, the group configuration of the table to be modified involves transaction dependencies between groups. The table to be modified is the table whose group configuration needs to be modified for synchronization, determined by comparing the group configurations before and after modification. If the group configuration of a table to be modified is adjusted multiple times within a short period, and this adjustment occurs between groups with the same group number, the single main transaction involving the table to be modified will be split into multiple parts. These sub-transactions will be scattered across multiple groups, or multiple sub-transactions may exist within a single group. These sub-transactions belong to the same main transaction, but current technology cannot execute them according to the preset order of all operations within that main transaction. This makes it impossible to guarantee the update order of the table to be synchronized, which can easily lead to inconsistencies between the data of the table to be modified in the target database and the data of the table to be modified in the source database, disrupting data consistency in the database system and causing errors in the data synchronization service.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the prior art, the present invention provides a method and apparatus for statically modifying data synchronization groups. Its purpose is to strictly adhere to the execution order of operations in the source main transaction during data synchronization, and modify the group configuration of the table to be modified without affecting data synchronization.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for statically modifying data synchronization groups, comprising:

[0008] Based on the table information and new grouping configuration of all sub-transactions to be split, obtain the target group corresponding to the sub-transaction to be split;

[0009] The table corresponding to the table information that is different between the target group and the original group is taken as the table to be modified;

[0010] Based on the original grouping and the target grouping of the table to be modified, a virtual transaction of the table to be modified is constructed. The virtual transaction is used as a sub-transaction of the corresponding active transaction to be split. All sub-transactions of the active transaction to be split are stored in the database. Among them, all sub-transactions of the active transaction to be split include the corresponding sub-transaction to be split and the corresponding virtual transaction.

[0011] Further, before obtaining the target group corresponding to the sub-transaction to be split based on the table information involved in all current sub-transactions to be split and the new group configuration, the following is included:

[0012] Stop the synchronization service, save the original grouping of each original sub-transaction in all current active transactions and the table information involved in the original sub-transaction to the checkpoint file, and then exit the synchronization service.

[0013] Obtain the new group configuration, modify the current group configuration of the system according to the new group configuration, and save it so that the system can divide the groups according to the current group configuration after restarting the synchronization service.

[0014] Restart the synchronization service and obtain the original grouping and table information for each original sub-transaction in all active transactions based on the latest checkpoint file;

[0015] The original sub-transactions that have not been marked with an end tag in all current active transactions are taken as sub-transactions to be cut, and the table information involved in the sub-transactions to be cut is obtained.

[0016] Further, before stopping the synchronization service, the process includes saving the original groupings of each original sub-transaction in all currently active transactions and the table information involved in the original sub-transactions to a checkpoint file, and before exiting the synchronization service after saving:

[0017] Receive logs and parse the logs to obtain at least one operation, the corresponding table information, the main transaction ID, and the operation number;

[0018] Determine whether a corresponding active transaction exists based on the main transaction ID. If no such transaction exists, create the active transaction corresponding to the main transaction ID.

[0019] Based on the table information, determine whether there is an original sub-transaction corresponding to the original group under the active transaction. If not, create the original sub-transaction; assign the operation to the original sub-transaction.

[0020] In this context, the operation number of the last operation in each original sub-transaction is used as the commit number of the corresponding original sub-transaction;

[0021] Upon receiving the commit message, the active transaction adds all original sub-transactions to the committed list of the corresponding original group in ascending order of the combination of the commit number and the commit LSN of all original sub-transactions in the active transaction, and then executes and inserts all original sub-transactions into the database according to the committed list.

[0022] Further, the step of constructing a virtual transaction for the table to be modified based on the original grouping and the target grouping of the table to be modified, using the virtual transaction as a sub-transaction of the corresponding active transaction to be split, and storing all sub-transactions of the active transaction to be split into the database includes:

[0023] The active transactions corresponding to the sub-transactions to be cut involving the table to be modified are taken as active transactions to be cut, and an end marker is added to the sub-transactions to be cut.

[0024] Construct a sub-transaction belonging to the target group of the activity transaction to be cut as a virtual transaction, and add an end marker to the virtual transaction;

[0025] Based on the original group and the target group, construct a modification operation to assign the table to be modified to the target group, and add the modification operation to the virtual transaction;

[0026] Continue receiving logs and commit messages, and based on the commit messages, store all sub-transactions of the activity to be split into the database.

[0027] Furthermore, the step of continuing to receive logs, receiving commit messages, and storing all sub-transactions of the activity to be split into the database according to the commit messages includes:

[0028] Receive a commit message, take the sub-transaction of the active transaction corresponding to the commit message as the sub-transaction to be entered into the database, and determine whether the active transaction is an active transaction to be split.

[0029] When a transaction is to be split, iterate through all the sub-transactions to be inserted into the database from the transaction to be split and find the corresponding virtual transaction.

[0030] The commit LSN of the commit message is used as the commit LSN and wait LSN of the virtual transaction; wherein, the commit number of the virtual transaction is the operation number of the last operation in the active transaction to be cut.

[0031] Add all the sub-transactions to be added to the committed list of the corresponding target group in ascending order of the combination of the commit number and the commit LSN of all the sub-transactions to be added to the database.

[0032] Based on the committed linked list and the waiting LSN, all the sub-transactions to be entered into the database.

[0033] Further, the step of inserting all pending sub-transactions into the database based on the committed linked list and the waiting LSN includes:

[0034] Sequentially determine whether the commit LSN of the sub-transaction to be added to the database in the committed list is less than or equal to the commit LSN of the corresponding target group, and determine whether the commit number of the sub-transaction to be added to the database is less than or equal to the commit number of the target group.

[0035] If yes, discard the sub-transaction to be entered into the database; if no, determine whether the sub-transaction to be entered into the database is a virtual transaction.

[0036] When it is a virtual transaction, based on the commit LSN and commit number corresponding to the original group on which the virtual transaction depends, all sub-transactions to be entered into the database in the target group to which the virtual transaction belongs may be selectively entered into the database or not entered into the database.

[0037] Specifically, after each sub-transaction to be inserted into the database is executed, the commit LSN of the sub-transaction to be inserted into the database is used as the commit LSN of the corresponding target group, and the commit number of the sub-transaction to be inserted into the database is used as the commit number of the target group.

[0038] Furthermore, when it is a virtual transaction, selectively including or not including all sub-transactions to be included in the database within the target group to which the virtual transaction belongs, based on the commit LSN and commit number corresponding to the original group on which the virtual transaction depends, includes:

[0039] Sequentially determine whether the commit LSN corresponding to the original group on which the virtual transaction depends is greater than or equal to the waiting LSN of the virtual transaction, and determine whether the commit number corresponding to the original group is greater than or equal to the commit number of the virtual transaction;

[0040] If not, then all pending sub-transactions in the target group to which the virtual transaction belongs in the committed list will not be entered into the database.

[0041] If so, then according to the committed linked list, all sub-transactions to be entered into the database in the target group to which the virtual transaction belongs.

[0042] Furthermore, the step of continuing to receive logs, receiving commit messages, and storing all sub-transactions of the activity to be split into the database according to the commit messages includes:

[0043] Restart the synchronization service, continue receiving logs, and determine the type of operation corresponding to the logs;

[0044] When the operation is a DML operation, the DML operation is divided into corresponding sub-transactions according to the main transaction ID, table information and grouping configuration of the DML operation; wherein, when the sub-transaction has added an end mark, the sub-transaction is a sub-transaction of the active transaction to be split, and a corresponding target sub-transaction is created according to the grouping configuration, and the operation number of the DML operation is used as the commit number of the target sub-transaction.

[0045] When the operation is a commit operation, the corresponding active transaction to be split is found according to the log corresponding to the commit operation. A commit message is added to all sub-transactions to be inserted into the database for the active transaction to be split. The commit LSN of the commit operation is used as the commit LSN of all sub-transactions to be inserted into the database. All sub-transactions to be inserted into the database are added to the committed list of the target group. All sub-transactions to be inserted into the database are inserted according to the committed list.

[0046] Furthermore, during the synchronization service failure restart and recovery, the following applies:

[0047] Based on the latest checkpoint file, restore the synchronized commit LSN, commit number, and virtual transactions in each target group;

[0048] When performing the data entry after recovery, it is determined whether the commit LSN and commit number of the executed sub-transaction are less than or equal to the commit LSN and corresponding commit number of the last synchronized sub-transaction.

[0049] If yes, then the execution and storage of the sub-transaction will not be performed; otherwise, the sub-transaction will be executed, and the current group configuration will be modified and saved according to the modification operation in the virtual transaction.

[0050] In a second aspect, the present invention also provides an apparatus for statically modifying data synchronization groups, used to implement the method for statically modifying data synchronization groups described in the first aspect, the apparatus comprising:

[0051] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method for statically modifying data synchronization groups as described in the first aspect.

[0052] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method for statically modifying data synchronization groups as described in the first aspect.

[0053] Unlike existing technologies, the present invention has at least the following beneficial effects:

[0054] This invention obtains the target group corresponding to the sub-transaction to be split based on the table information and new group configuration of all sub-transactions to be split. Tables whose information differs from the original group in the target group are designated as tables to be modified, thus determining the tables to be modified in this group configuration change and identifying the active transactions to be split in the corresponding source-side main transaction. Based on the original group and target group of the table to be modified, a virtual transaction is constructed for the table to be modified. This virtual transaction is used as a sub-transaction of the corresponding active transaction to be split, and all sub-transactions of the active transaction to be split are entered into the database. By setting active and virtual transactions, the corresponding operations are entered into the database on the target side in the order of operations in the main transaction, completing the modification of the group configuration of the table to be modified between groups without affecting data synchronization. This solves the problem that when modifying group configuration, the data of the table to be modified in the target database cannot be guaranteed to be updated in the order of the main transaction, leading to inconsistencies between the data of the table to be modified in the target database and the data of the table to be modified in the source database, thus disrupting data consistency in the database system and causing synchronization service errors. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0056] Figure 1 This is a flowchart illustrating a method for statically modifying data synchronization groups provided in an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the specific process of step 30 in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of a specific process for step 304 in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the specific process of step 3045a in an embodiment of the present invention;

[0060] Figure 5 This is another specific flowchart of step 304 in an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of the architecture of a device for statically modifying data synchronization groups provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0063] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0064] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0065] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0066] Example 1:

[0067] In the log-parsing-based group synchronization method, the target database receives logs sent by the source database, parses the logs to obtain at least one main transaction from the source database, and each main transaction contains at least one operation. The main transaction is split and classified by table to obtain sub-transactions belonging to different groups. That is, according to the tables on which the operations in the main transaction depend, the operations are classified into the corresponding groups, and the sub-transactions in each group are executed in parallel by using multi-threading to improve data synchronization efficiency.

[0068] During the operation of the synchronization service, there is a need to modify the grouping configuration of related tables. The following example illustrates the modification of the grouping configuration: Before the modification, operations dependent on table A and table B were assigned to the first group, and operations dependent on table C were assigned to the second group; after the modification, operations dependent on table A are assigned to the first group, and operations dependent on table B and table C are assigned to the second group; the table to be modified is table B.

[0069] When the existing technology modifies the grouping configuration of the table to be modified between groups, it only implements the operation that depends on the table to be modified after the modification, and divides it into a sub-transaction of the same main transaction in a different group than before the modification, but does not re-divide the operations that have been divided into the corresponding group before the modification but have not yet been executed. If a table to be modified is grouped multiple times in a short period of time (for example, operations on table A are first assigned to the first group, then to the second group, and finally to the third group in a short period of time), and there are adjustments between groups with the same group number (for example, operations on table A are first assigned to the first group, then to the second group, and finally to the first group), it will cause a single main transaction involving the table to be modified to be split into multiple sub-transactions in different groups in the target database, and the split sub-transactions will be scattered in multiple groups (for example, main transaction 1 is split into sub-transaction 1 in the first group and sub-transaction 2 in the second group), or there will be multiple split sub-transactions in a group (for example, since operations on table A are sequentially assigned to the first group, the second group, and the first group, there are sub-transactions 1 and 3 of main transaction 1 in the first group, and sub-transaction 2 of main transaction 1 in the second group, where the operations in main transaction 1 are sequentially divided into sub-transactions 1, 2, and 3).

[0070] When operations in different groups are executed in parallel and inserted into the database, the synchronization efficiency of sub-transactions within different groups varies. Operations belonging to different sub-transactions before and after group configuration modifications cannot be executed in the predetermined order of all operations within the main transaction. This results in tables in the target database that are to be synchronized, which were originally required to be updated in the order of the main transaction (i.e., after operation 1 updates the table to be synchronized, operation 2 continues to update based on the updated table by operation 1), no longer guarantee sequential updates after the group configuration of these tables is modified (i.e., there might be a situation where operation 2 updates the table first, and then operation 1 continues to update based on the updated table by operation 2). This can easily lead to inconsistencies between the data in the target database and the data in the source database, disrupting data consistency in the database system and causing errors in the data synchronization service.

[0071] Static modifications often involve directly altering the grouping configuration of the synchronization service in the configuration file. This involves large-scale modifications to grouping configurations, especially when the tables requiring these modifications are unknown to the service. Static modifications require stopping the entire synchronization service for an extended period before making these large-scale changes. Because the entire synchronization service must be stopped, including the database insertion thread, there is a high probability of many operations that were assigned to groups before modification but have not yet been inserted. When operations are then executed in parallel according to groups after modifying the grouping configuration, a significant number of these operations cannot be executed in the order specified in the main transaction at the source end, and a large number of sub-transactions within the same main transaction cannot strictly adhere to the execution order of the main transaction.

[0072] To address the aforementioned problems, embodiments of the present invention provide a method for statically modifying data synchronization groups, such as... Figure 1 As shown, it includes:

[0073] Step 10: Based on the table information and new group configuration of all sub-transactions to be split, obtain the target group corresponding to the sub-transaction to be split.

[0074] The table information refers to the table information that the corresponding operation depends on, contained in the logs sent from the source end to the target end. The new group configuration is the group configuration that the synchronization service obtains from the configuration file after the user modifies the configuration file. The target group is the group created after the group configuration is modified. One main transaction on the source end corresponds to one active transaction on the target end. The main transaction from the source end is divided into at least one sub-transaction. An active transaction is a set of at least one sub-transaction, and all active transactions are sub-transactions that have not received a corresponding commit operation. Sub-transactions to be divided are sub-transactions that have not been marked with an end tag.

[0075] Step 20: The tables corresponding to the table information that differs between the target group and the original group are used as the tables to be modified.

[0076] Here, the original group refers to the group created before the group configuration was modified; the table to be modified is the table whose grouping was modified when the group configuration was changed. The process involves traversing the sub-transactions to be split and matching the grouping information of each transaction in the new group configuration based on the table information involved in those sub-transactions.

[0077] Step 30: Based on the original grouping of the table to be modified and the target grouping of the table to be modified, construct a virtual transaction for the table to be modified, use the virtual transaction as a sub-transaction of the corresponding active transaction to be split, and store all sub-transactions of the active transaction to be split into the database; wherein, all sub-transactions of the active transaction to be split include the corresponding sub-transaction to be split and the corresponding virtual transaction.

[0078] In this context, "data insertion" involves executing an operation on the target database to achieve data synchronization. "Active transactions to be split" refers to those transactions involving modifications to tables during the current group configuration modification phase; these types of active transactions need to be split. The relationship between active transactions to be split and sub-transactions to be split is explained below. A specific example of an active transaction is as follows:

[0079] A1

[0080] {

[0081] Group G1:

[0082] G1_TRX1[Table:T1, End Marker:0, Commit LSN:0, Commit Number:1]

[0083] }

[0084] Because subtransaction G1_TRX1 did not have an end marker, it is a subtransaction to be split. Based on the table information (table T1) involved in subtransaction G1_TRX1 and the new grouping configuration (table T1 is in group G2), the target group corresponding to subtransaction G1_TRX1 is determined to be group G2. The original group of subtransaction G1_TRX1, i.e., group G1, is obtained. Since the target group (group G2) and the original group (group G1) are different, table T1 is determined to be modified in this modification. Active transaction A1 is found and identified as the active transaction to be split in this grouping configuration modification. At this time, active transaction A1 contains subtransaction G1_TRX1, and all subtransactions of active transaction A1 include only subtransaction G1_TRX1 (the virtual transaction has not yet been built).

[0085] The static modification data synchronization grouping method of this invention can solve at least the following problems: First, it is necessary to solve the data synchronization problem involving the table to be modified before and after the grouping configuration is modified; second, it is necessary to modify the current system's grouping configuration so that after receiving the target's logs, the operations contained in the logs can be divided according to the modified grouping configuration. Since the table to be modified is unknown, it is necessary to use the table information involved in all the sub-transactions to be split. By locating the corresponding grouping information in the new grouping configuration, comparing the grouping information with the grouping information before the grouping configuration was modified, the table whose grouping configuration has changed is obtained and identified as the table to be modified. In the current active transactions, all active transactions involving the table to be modified are found, and the sub-transactions involving their groups are split and virtual transactions are constructed to constrain the execution order of the groups before and after the table to be modified.

[0086] The static modification data synchronization grouping method of this invention obtains the target group corresponding to the sub-transaction to be split based on the table information involved in all sub-transactions to be split and the new grouping configuration. It then identifies the tables whose information differs from the original group in the target group as the tables to be modified, thus determining the tables to be modified in this grouping configuration modification and identifying the active transactions to be split in the corresponding source-side main transaction. Based on the original group and target group of the table to be modified, a virtual transaction for the table to be modified is constructed. This virtual transaction is used as a sub-transaction of the corresponding active transaction to be split, and all sub-transactions of the active transaction to be split are entered into the database. By setting active and virtual transactions, the corresponding operations are entered into the database on the target side according to the order of operations in the main transaction. This completes the modification of the grouping configuration of the table to be modified between groups without affecting data synchronization. This solves the problem that when modifying grouping configuration, the data of the table to be modified in the target database cannot be guaranteed to be updated in the order of the main transaction, leading to inconsistencies between the data of the table to be modified in the target database and the data of the table to be modified in the source database, thus disrupting data consistency in the database system and causing synchronization service errors.

[0087] This invention, through the deployment of a log parsing service at the source end, independently numbers the parsed operations based on the main transaction at the source end, ensuring the sequential incrementality of operation numbers within the main transaction. The log corresponding to each operation is then populated with the table information involved. Those skilled in the art can specify the specific implementation of operation numbers and table information according to the specific data synchronization scenario. The target-end synchronization service receives operations from the logs sent by the source end, categorizes them into the corresponding active transactions by main transaction ID, extracts the table information involved in the operations, locates the group to which the operation belongs using the table information, and constructs the sub-transaction corresponding to the main transaction under the active transaction based on the group information.

[0088] To better illustrate the static modification data synchronization grouping method of the present invention, the following further explains the preparatory work required before modifying the grouping configuration in the embodiments of the present invention. Specifically, logs are received, and the logs are parsed to obtain at least one operation, corresponding table information, a main transaction ID, and an operation number. Based on the main transaction ID, it is determined whether a corresponding active transaction exists. If not, an active transaction corresponding to the main transaction ID is created. Based on the table information, it is determined whether an original sub-transaction corresponding to the original group exists under the active transaction. If not, the original sub-transaction is created; the operation is assigned to the original sub-transaction. The operation number of the last operation in each original sub-transaction is used as the commit number of the corresponding original sub-transaction.

[0089] Upon receiving the commit message, the active transaction adds all original sub-transactions to the committed list of the corresponding original group in ascending order of the combination of the commit number of all original sub-transactions and the commit LSN (Log sequence number) of all original sub-transactions. Based on the committed list, all original sub-transactions are then executed and stored in the database.

[0090] In the static data synchronization grouping method of this invention, when parsing logs at the source end, the operations in each source-end main transaction are numbered sequentially. Since a transaction is a collection of operations, operations within the same transaction are either committed together or not committed together. Therefore, the commit LSNs of target-end sub-transactions corresponding to the same source-end main transaction are all the same. Within the group, the operation order needs to be sorted according to the operation number to strictly distinguish the order of operations in the source-end main transaction. At the target end, when the sub-transaction is executed, the operation number of the last operation of the transaction is used to ensure data consistency after fault recovery. After receiving the logs, the target end groups at least one operation in each source-end main transaction according to the tables it depends on, dividing it into multiple sub-transactions. A single main transaction obtained by parsing the logs at the source end involves operations on multiple tables. When grouping at the target end, these multiple tables are divided into different groups, and the target end creates an overall transaction (i.e., an active transaction) to categorize the operations of the main transaction. Multiple sub-transactions are created under the active transaction, each corresponding to a group. Operations belonging to the same group are grouped under that sub-transaction, and the operation number of the last operation in each sub-transaction is recorded as its commit number. By setting up active transactions, the main transaction on the source side is managed as a set of active transactions on the target side. This facilitates locating the sub-transactions in each group even when the main transaction is divided according to its dependent tables and the grouping configuration is modified and split into multiple sub-transactions, and then inserting them into the database in the order of operations in the main transaction. The tables updated by the main transaction are the tables that the main transaction depends on.

[0091] After modifying the grouping configuration of the table to be modified, the sub-transactions involved in the relevant groups are split, with the pre-modification and post-modification sub-transactions forming independent sub-transactions. Since the target end receives all logs generated by the source end during operation, it includes logs of ongoing but uncommitted operations. To maintain database consistency, synchronization is based on the commit of the source end's main transaction. That is, when the target end receives the corresponding logs, it only assigns uncommitted operations from the source end to active transactions and their corresponding groups. Data synchronization for those operations only occurs when the target end receives a committed operation.

[0092] When operations on a table to be modified are switched from one group to another, sub-transactions belonging to these two groups will form a dependency relationship. Specifically, if table A in the database is updated by operations in transaction 1, and subsequent operations in transaction 2 can only update table A based on the updated table A, then transaction 2 is said to depend on transaction 1, or transaction 1 and transaction 2 have a transaction dependency relationship. Because operations within the same sub-transaction must be executed strictly in the order of their operation numbers within the main transaction, operations switched to the target group must wait for the sub-transaction to complete its data entry in the original group before they can begin. To solve this problem, when switching the active transaction (corresponding to the source main transaction), a virtual transaction is added to the target group corresponding to the active transaction to check the completion status of the original group's operations. The target group checks the completion status of the original group's operations on the involved tables by executing the virtual transaction; only when the conditions are met can the sub-transactions belonging to the target group begin execution.

[0093] To obtain table information and new grouping configurations for all currently pending sub-transactions, the following steps are included before step 10:

[0094] Stop the synchronization service, save the original groupings of each original sub-transaction in all current active transactions and the table information involved in the original sub-transactions to the checkpoint file, and then exit the synchronization service. Because static modification involves large-scale grouping changes during the splitting of active transactions, to prevent new operations from being assigned to active transactions and causing changes to them, and to determine the tables to be modified by comparing the grouping information with the grouping information in the previous checkpoint, the synchronization service must be stopped before splitting the active transactions.

[0095] Since the table to be modified is unknown, a checkpoint action needs to be performed after stopping the synchronization service. The active transaction is then recovered from the checkpoint, and the grouping information of the table involved in the sub-transactions within the active transaction is located in the new grouping configuration using the table information. This grouping information is compared with the grouping information from the previous checkpoint to obtain the table whose grouping configuration has changed and needs modification. If the grouping obtained from the sub-transaction match is the same as the grouping saved in the checkpoint, no change is needed. If the grouping obtained from the sub-transaction match is different from the grouping saved in the checkpoint, then as in step 20, the table corresponding to the table information where the target group differs from the original group is taken as the table to be modified. All active transactions involving the table to be modified are found in the current active transaction. The sub-transactions involving these groups are segmented and virtual transactions are constructed to constrain the execution order of the groups before and after the table modification.

[0096] An example of an active transaction provided by an embodiment of the present invention is as follows:

[0097] A1

[0098] {

[0099] Group G1:

[0100] G1_TRX1[Table:T1, End Marker:0, Commit LSN:0, Commit Number:1]

[0101] }

[0102] When an operation with LSN 21 is received, an active transaction A1 with ID 1 is created using the primary transaction ID. At this moment, T1 belongs to the G1 group. A sub-transaction G1_TRX1 belonging to the G1 group is created on the active transaction, and the operation is added to the sub-transaction. The operation number is used as the commit number of the transaction.

[0103] Prepare to modify the grouping of T1, changing it from the original group G1 to the target group G2. (The synchronization service includes log reception functionality.) Stop receiving logs from the source end, traverse active transactions, and save the original grouping of each original sub-transaction in all current active transactions and the table information involved in the original sub-transaction to the checkpoint file. The active transaction saved in the checkpoint is A1, which has a sub-transaction G1_TRX1, belonging to group G1, and the table information involved in the sub-transaction is T1. Exit the synchronization service.

[0104] Obtain the new group configuration, modify the current group configuration of the system according to the new group configuration, and save it so that the system can divide the groups according to the current group configuration after restarting the synchronization service. For example, modify the synchronization group configuration of T1 from group G1 (original group) to group G2 (target group).

[0105] Restart the synchronization service and obtain the original grouping and table information for each original sub-transaction in all active transactions based on the latest checkpoint file.

[0106] For example, the synchronization service is started, the active transactions saved in the last checkpoint file are loaded, and the grouping of subtransactions to be split within the active transactions is reassigned. At this point, subtransaction G1_TRX1 in active transaction A1 is assigned to group G2.

[0107] Compare the information of the original group and the target group of the subtransaction in the active transaction to see if they are consistent. The target group of the subtransaction G1_TRX1 is group G2, while the group saved in the checkpoint is group G1. They are inconsistent, and the group of the table T1 to be modified needs to be switched.

[0108] The original sub-transactions without an end marker among all current active transactions are taken as sub-transactions to be split, thus obtaining the table information involved in the sub-transactions to be split. The end marker is used to mark sub-transactions that have already been split. Since it is necessary to split the current active transactions involving the table to be modified, construct virtual transactions to save the modification operations of the grouping configuration, and constrain the execution order of the groups before and after the table to be modified, it is necessary to first find the sub-transactions involving the table to be modified (i.e., find the active transactions involving the table to be modified).

[0109] When statically modifying data, after the synchronization service restarts, it is necessary to restore the active transaction from the checkpoint, and then locate the grouping information of the table in the modified grouping configuration based on the table information involved in the sub-transactions in the active transaction.

[0110] When operations on the table to be modified are switched from one group to another, a transaction dependency arises between the two groups. This is because operations within the same transaction must be executed strictly according to their operational numbers within the transaction. Therefore, operations switched to the target group cannot begin their own insertion process until the original group has completed its insertion process. To address this issue, a virtual transaction is added to the target group corresponding to the active transaction to check the completion status of the original group's operations. The target group checks the completion status of the original group's operations by executing this virtual transaction. Only when the conditions are met can the target group's transaction begin its own execution.

[0111] To better illustrate the static modification data synchronization grouping method of the present invention, step 30 of the static modification data synchronization grouping method of the present invention will be further refined below. Specifically, as follows: Figure 2 As shown, step 30 includes:

[0112] Step 301: Take the active transaction corresponding to the sub-transaction to be cut involving the table to be modified as the active transaction to be cut, and add an end marker to the sub-transaction to be cut.

[0113] For example, take the active transaction A1 corresponding to the sub-transaction G1_TRX1 to be split as the active transaction to be split, and add an end marker to the sub-transactions to be split in the original group involved in the T1 table on the active transaction to be split, as follows:

[0114] A1

[0115] {

[0116] Group G1:

[0117] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0118] }

[0119] In this context, an end marker of 0 indicates that no end marker has been added, while an end marker of 1 indicates that an end marker has been added.

[0120] Step 302: Construct a sub-transaction belonging to the target group of the activity transaction to be cut as a virtual transaction, and add an end marker to the virtual transaction.

[0121] Step 303: Based on the original group and the target group, construct a modification operation to assign the table to be modified to the target group, and add the modification operation to the virtual transaction. The virtual transaction operation waits for the original group transaction to execute until the commit LSN and commit number of this transaction are obtained. The commit number of this virtual transaction is the operation number of the last operation of the corresponding active transaction, and the modification operation for the group configuration of the table to be modified is added to this virtual transaction. Those skilled in the art can specify the specific implementation of the operation number according to the specific data synchronization scenario, which is not limited here.

[0122] Steps 302 and 303, for example, construct a virtual transaction G2_TRX2 belonging to group G2 and dependent on group G1 for the new group involved in table T1 on active transaction A1. Specifically, virtual transaction G2_TRX2 is said to depend on group G1 if virtual transaction G2_TRX2 can only successfully execute and insert data into the target database updated by all sub-transactions in group G1 after all sub-transactions in group G1 have completed their execution and insertion. The commit number of this transaction is the number of the last operation of the active transaction A1 to be split. Since the active transaction to be split has not yet received a commit message, both the commit LSN and the waiting LSN are set to 0, and the modification operation of changing T1 from group G1 to group G2 is saved to this virtual transaction. The activity transaction A1 to be split is shown below. Taking this state of the activity transaction A1 to be split as an example, all the sub-transactions of the activity transaction A1 to be split include the sub-transaction G1_TRX1 to be split and the virtual transaction G2_TRX2. When the commit message of the activity transaction A1 to be split is received later, the sub-transaction G1_TRX1 to be split and the virtual transaction G2_TRX2 will be entered into the database.

[0123] A1

[0124] {

[0125] Group G1:

[0126] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0127] Group G2:

[0128] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0129] }

[0130] Step 304: Continue receiving logs and commit messages. Based on the commit messages, add all sub-transactions of the activity to be split into the database. At this point, the synchronization service is started and continues receiving logs, grouping the newly received logs according to the new grouping configuration.

[0131] To better illustrate the static modification data synchronization grouping method of the present invention, step 304 of the static modification data synchronization grouping method of the present invention will be further refined below. Specifically, as follows: Figure 3 As shown, step 304 includes:

[0132] Step 3041a: Receive a commit message, treat the sub-transactions of the active transaction corresponding to the commit message as sub-transactions to be inserted into the database, and determine whether the active transaction is an active transaction to be split. Among them, active transactions whose corresponding sub-transactions do not have an end marker are active transactions to be split. If it is not an active transaction to be split, add all sub-transactions to the committed list of the corresponding target group in ascending order of the combination of the commit number and the commit LSN of all sub-transactions to be inserted into the database, for insertion.

[0133] Step 3042a: When it is an active transaction to be split, traverse all the sub-transactions to be entered into the database from the active transaction to be split and find the corresponding virtual transaction.

[0134] Step 3043a: Use the commit LSN of the commit message as the commit LSN and wait LSN of the virtual transaction; wherein, the commit number of the virtual transaction is the operation number of the last operation in the active transaction to be split. The commit number is set when the virtual transaction is constructed; the wait LSN is the commit LSN of the group that the virtual transaction needs to wait for. By setting the wait LSN, the virtual transaction (the operation that modifies the group configuration) can only begin execution after the synchronization of sub-transactions in the waiting group that are less than or equal to the commit LSN has been completed.

[0135] Step 3044a: Add all sub-transactions to be added to the committed list of the corresponding target group in ascending order of the combination of their commit numbers and commit LSNs. Operations corresponding to the same source-end main transaction have the same commit LSN but different commit numbers. Those skilled in the art can specify the combination of commit LSNs and commit numbers according to the specific data synchronization scenario. Each group corresponds to one committed list.

[0136] Step 3045a: Based on the committed linked list and the waiting LSN, write all the sub-transactions to be written into the database.

[0137] To maintain database consistency, before synchronization, the operations to be synchronized are distinguished according to their dependent tables. This avoids database errors caused when multiple sub-transactions are executed in parallel because some sub-transactions in the dependent groups of the sub-transactions have not yet been entered into the database. To better illustrate the method of statically modifying data synchronization groups in this invention, step 3045a of the method in this embodiment of the invention will be further refined below. Specifically, as follows... Figure 4 As shown, step 3045a includes:

[0138] Step 30451: Sequentially determine whether the commit LSN of the sub-transaction to be added to the database in the committed list is less than or equal to the commit LSN of the corresponding target group, and determine whether the commit number of the sub-transaction to be added to the database is less than or equal to the commit number of the target group.

[0139] For example, the active transaction A1 that has received the commit message is as follows:

[0140] A1

[0141] {

[0142] Group G1:

[0143] G1_TRX1[Table:T1, End Marker:1, Commit LSN:25, Commit Number:1]

[0144] G1_TRX4 [Depends on G2, waiting LSN:25, end marker:1, commit LSN:25, commit number:2]

[0145] G1_TRX5[Table:T1, End Marker:0, Commit LSN:25, Commit Number:4]

[0146] Group G2:

[0147] G2_TRX2 [Depends on G1, waiting LSN: 25, end marker: 1, commit LSN: 25, commit number: 1]

[0148] G2_TRX3[Table:T1, End Marker:1, Commit LSN:25, Commit Number:2]

[0149] }

[0150] The execution thread polls the G1 and G2 groups, and executes them in the committed linked lists of the G1 and G2 groups in order of the size of the combination of the transaction's commit LSN and commit number.

[0151] Step 30452: If yes, discard the sub-transaction to be entered into the database; if no, determine whether the sub-transaction to be entered into the database is a virtual transaction.

[0152] For example, the execution thread extracts the sub-transaction G1_TRX1 to be inserted into the database from the G1 group. At this time, the G1 group has not yet executed the insertion of a certain sub-transaction into the database, so the commit LSN and commit number of the G1 group are both 0. The commit LSN25 of the sub-transaction G1_TRX1 to be inserted into the database is not less than or equal to 0, and the commit number 1 is not less than or equal to 0, so it is inserted into the database.

[0153] Step 30453: When it is a virtual transaction, based on the commit LSN and commit number corresponding to the original group to which the virtual transaction depends, selectively add or exclude all sub-transactions to be added to the database in the target group to which the virtual transaction belongs. Specifically, when it is a virtual transaction, it depends on the execution of sub-transactions in the original group before modification; when it is not a virtual transaction, it does not depend on the execution of other groups and can be executed directly.

[0154] The system sequentially determines whether the commit LSN of the original group on which the virtual transaction depends is greater than or equal to the waiting LSN of the virtual transaction, and whether the commit number of the original group is greater than or equal to the commit number of the virtual transaction. If not, all pending sub-transactions in the target group to which the virtual transaction belongs, as defined in the committed list, are not entered into the database; if yes, all pending sub-transactions in the target group to which the virtual transaction belongs are entered into the database according to the committed list.

[0155] For example, the execution thread retrieves the pending sub-transaction G1_TRX4 from group G1. It depends on the execution of group G2 and is a virtual transaction. The commit LSN25 of group G2 is equal to the waiting LSN25 of virtual transaction G1_TRX4, which satisfies the condition. The commit number 0 of group G2 (at this time, group G2 has not executed any pending sub-transaction) is not greater than or equal to the commit number 2 of virtual transaction G1_TRX4, which does not satisfy the condition. Therefore, transaction G1_TRX4 cannot be executed. The pending sub-transactions in group G1 on the currently committed list should be skipped, and the pending sub-transactions in group G2 should be retrieved.

[0156] The execution thread retrieves the pending subtransaction G2_TRX2 from group G2, which depends on the execution of group G1. At this point, group G1 has already executed G1_TRX1. Group G1's commit LSN and commit number are LSN 25 and 1, respectively. The waiting LSN 25 of the pending subtransaction G2_TRX2 equals the commit LSN 25 of group G1, satisfying the condition. The commit number 1 of group G2 equals the commit number 1 of group G1, also satisfying the condition. Therefore, the pending subtransaction G2_TRX2 is executed and inserted into the database.

[0157] Step 30454: After each sub-transaction to be inserted into the database is executed, the commit LSN of the sub-transaction is used as the commit LSN of the corresponding target group, and the commit number of the sub-transaction is used as the commit number of the target group. Each group also needs to record the commit LSN and commit number of the completed transaction during transaction execution. This is used for filtering executed transactions after a failure, ensuring data consistency.

[0158] For example, the execution thread extracts the G1_TRX1 transaction from the G1 group and stores it in the database, recording the commit LSN 25 and commit number 1 of the completed transaction as the commit LSN of the target group.

[0159] By setting up virtual transactions, when a sub-transaction involving the table to be modified is committed, the transaction dependencies between the original group and the target group, as well as the execution order of the sub-transactions formed after splitting, are guaranteed. By setting the wait LSN and commit LSN of the virtual transaction, and setting the commit number of the virtual transaction to the operation number of the last operation of the corresponding active transaction, it is ensured that during the data insertion process, the synchronization service will not encounter runtime errors caused by transaction dependencies between sub-transactions, and that the consistency of the synchronized data will be restored when the synchronization service fails and is recovered.

[0160] The target end continues to receive logs sent by the source end, generates or categorizes the operations within them into corresponding active transactions, and performs corresponding processing based on the operation type. To better illustrate the static modification of data synchronization groups method of the present invention, step 30 of the static modification of data synchronization groups method of the present invention will be further refined below. Specifically, as follows... Figure 5 As shown, step 304 further includes:

[0161] Step 3041b: Restart the synchronization service, continue receiving logs, and determine the type of operation corresponding to the logs. Among these, DML (Data Manipulation Language) operations belonging to the same main transaction at the source end, when divided into different groups for parallel execution, often have dependency conflicts. DDL (Data Definition Language) operations generally do not have dependency conflicts and do not require the static modification of data synchronization grouping method of this embodiment. After receiving a DDL operation, it is divided into the corresponding active transaction according to the tables it depends on. After receiving the corresponding commit message, it is added to the committed list according to the commit LSN and executed for database entry. When the operation is a rollback operation, all sub-transactions under the active transaction and the active transaction itself are released according to the active transaction. The rollback operation at the source end will undo all executed main transactions. To maintain the consistency between the source and target data during data synchronization, all operations on the target end involving the main transaction of the source end need to be rolled back.

[0162] Step 3042b: When the operation is a DML operation, the DML operation is divided into corresponding sub-transactions according to the main transaction ID, table information and grouping configuration of the DML operation; wherein, when the sub-transaction has added an end mark, the sub-transaction is a sub-transaction of the active transaction to be split, and a corresponding target sub-transaction is created according to the grouping configuration, and the operation number of the DML operation is used as the commit number of the target sub-transaction.

[0163] For example, when a DML operation with commit LSN 22 is received, the corresponding active transaction A1 with ID 1 is located using the primary transaction ID. Within active transaction A1, the original subtransaction G1_TRX1 and the target subtransaction G2_TRX3 corresponding to the table T1 to be modified are found. Since the original subtransaction G1_TRX1 has an end marker, meaning it was a subtransaction to be split during the last group configuration modification, according to the group configuration, the table T1 to be modified now belongs to group G2. A target subtransaction G2_TRX3 belonging to group G2 is created within the active transaction, and the DML operation is added to this target subtransaction. The operation number is used as the commit number of the target subtransaction, as detailed below:

[0164] A1

[0165] {

[0166] Group G1:

[0167] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0168] Group G2:

[0169] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0170] G2_TRX3[Table:T1, End Marker:0, Commit LSN:0, Commit Number:2]

[0171] }

[0172] When a DML operation with LSN 24 is received, the active transaction A1 with ID 1 is located through the main transaction ID. At this time, T1 belongs to the G1 group. There is already a sub-transaction G1_TRX5 belonging to the G1 group on the active transaction. Its end mark is 0, that is, no end mark has been added. Therefore, the DML operation is still added to the sub-transaction G1_TRX5.

[0173] A1

[0174] {

[0175] Group G1:

[0176] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0177] G1_TRX4 [Depends on G2, waiting LSN:0, end marker:1, commit LSN:0, commit number:2]

[0178] G1_TRX5[Table:T1, End Marker:0, Commit LSN:0, Commit Number:4]

[0179] Group G2:

[0180] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0181] G2_TRX3[Table:T1, End Marker:1, Commit LSN:0, Commit Number:2]

[0182] }

[0183] Step 3043b: When the operation is a commit operation, find the corresponding active transaction to be split according to the log corresponding to the commit operation, add commit messages to all sub-transactions to be inserted into the database for the active transaction to be split, use the commit LSN of the commit operation as the commit LSN of all sub-transactions to be inserted into the database, add all sub-transactions to be inserted into the database to the committed list of the target group, and insert all sub-transactions to be inserted into the database according to the committed list.

[0184] In the static data synchronization grouping method of this embodiment of the invention, when continuing to receive logs and dividing the corresponding operations, the group is determined based on the table information parsed from the logs corresponding to the operation. This group belongs to an active transaction or an active transaction to be split. If the table corresponding to the table information is not a table to be modified, it means that the corresponding active transaction does not need to be split before and after the group configuration modification, and the continued received operations are divided into sub-transactions of the active transaction.

[0185] For example, when a commit operation with commit LSN of 25 is received, the active transaction A1 with ID 1 is located through the main transaction ID. All sub-transactions of active transaction A1 are traversed and their commit LSNs are marked as the commit LSN of the current commit operation. If the sub-transaction is a virtual transaction, the waiting LSN of the group it is waiting for is set to 25, and then it is added to the committed list of the corresponding group.

[0186] A1

[0187] {

[0188] Group G1:

[0189] G1_TRX1[Table:T1, End Marker:1, Commit LSN:25, Commit Number:1]

[0190] G1_TRX4 [Depends on G2, waiting LSN:25, end marker:1, commit LSN:25, commit number:2]

[0191] G1_TRX5[Table:T1, End Marker:0, Commit LSN:25, Commit Number:4]

[0192] G2 group: G2_TRX2 [Depends on G1, waiting LSN: 25, end marker: 1, commit LSN: 25, commit number: 1]

[0193] G2_TRX3[Table:T1, End Marker:1, Commit LSN:25, Commit Number:2]

[0194] }

[0195] The submission message is not shown here. The specific implementation method can be specified by those skilled in the art according to the specific data synchronization scenario, and is not limited here.

[0196] It is worth noting that during the synchronization service failure restart and recovery, the following should be noted:

[0197] Based on the latest checkpoint file, restore the synchronized commit LSNs, commit numbers, and virtual transactions in each target group. During the data insertion process after recovery, determine if the commit LSN and commit number corresponding to the executed sub-transaction are less than or equal to the commit LSN and corresponding commit number of the last synchronized sub-transaction. If yes, the sub-transaction is not inserted; otherwise, the sub-transaction is executed, and the current group configuration is modified and saved according to the modification operations in the virtual transaction. Since a checkpoint action was performed before exiting the synchronization service during static modification, it is not necessary to perform a checkpoint action again. Based on the checkpoint file saved before exiting the synchronization service, filter sub-transactions whose commit LSNs are less than or equal to the target group's commit LSN and whose commit numbers are less than or equal to the target group's commit number to ensure the consistency of the fault recovery data.

[0198] Example 2:

[0199] Based on Embodiment 1 above, this embodiment of the invention provides a specific example of a method for statically modifying data synchronization groups to better understand the entire synchronization process. This embodiment of the invention will use the log information sequence shown in the table below as an example for illustration:

[0200] The source database has a table T1 (ID INT). A transaction on the source database operates on table T1, generating the following sequential log. Upon receiving this log, the receiving thread will generate the following numbered table:

[0201]

[0202] The target synchronization service is configured with two groups, G1 and G2. Initially, the table to be modified is configured in group G1. The table T1 is then modified to group G2, and then changed back to group G1 from group G2. The process is as follows:

[0203] When an operation with LSN 21 is received, an active transaction A1 with ID 1 is created using the primary transaction ID. At this moment, T1 belongs to the G1 group. A sub-transaction G1_TRX1 belonging to the G1 group is created on the active transaction, and the operation is added to the sub-transaction. The operation number of the operation is used as the commit number of the sub-transaction.

[0204] A1

[0205] {

[0206] Group G1:

[0207] G1_TRX1[Table:T1, End Marker:0, Commit LSN:0, Commit Number:1]

[0208] }

[0209] We are preparing to modify the grouping of T1, changing it from group G1 to group G2.

[0210] Stop receiving logs from the source end, traverse active transactions, and save the grouping information of their sub-transactions and the table information involved in the sub-transactions to the checkpoint file. The active transaction saved in the checkpoint file is A1, which has a sub-transaction G1_TRX1, belongs to the group G1, and the table information involved in the sub-transaction is T1.

[0211] Exit synchronization service.

[0212] Modify the synchronization group configuration of T1, changing it from group G1 to group G2.

[0213] The synchronization service is started, the active transactions saved from the last checkpoint are loaded, and the sub-transactions within the active transactions that have not been marked with an end tag are reassigned to group assignments. At this point, the sub-transaction G1_TRX1 in active transaction A1 is assigned to group G2.

[0214] Compare the original grouping information of the subtransaction in the active transaction with that of the target group. The new grouping of the subtransaction G1_TRX1 is G2, while the checkpoint saves G1, which is inconsistent. The table T1 to be modified needs to switch groups.

[0215] Add an end marker to the sub-transactions to be cut involving table T1 on active transaction A1.

[0216] A1

[0217] {

[0218] Group G1:

[0219] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0220] }

[0221] Construct a virtual transaction G2_TRX2 that depends on G1 for the new group involved in table T1 on active transaction A1, and save the modification operation of T1 from group G1 to group G2 to this virtual transaction.

[0222] A1

[0223] {

[0224] Group G1:

[0225] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0226] Group G2:

[0227] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0228] }

[0229] The synchronization service has been started.

[0230] When an operation with LSN 22 is received, the active transaction A1 with ID 1 is located through the main transaction ID. At this moment, T1 belongs to group G2. A sub-transaction G2_TRX3 belonging to group G2 is created on the active transaction, and the operation is added to the sub-transaction. The operation number of the operation is used as the commit number of the sub-transaction.

[0231] A1

[0232] {

[0233] Group G1:

[0234] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0235] Group G2:

[0236] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0237] G2_TRX3[Table:T1, End Marker:0, Commit LSN:0, Commit Number:2]

[0238] }

[0239] We are preparing to modify the grouping of T1, changing it from group G2 to group G1.

[0240] Stop receiving logs from the source end, traverse the active transactions, and save the grouping information of their sub-transactions and the table information involved in the sub-transactions to the checkpoint file. The active transaction saved in the checkpoint file is A1, which has a sub-transaction G1_TRX3 without an end marker, belongs to the group G2, and the table information involved in the sub-transaction is T1.

[0241] Exit synchronization service.

[0242] Modify the synchronization group configuration of T1, changing it from G2 group to G1 group.

[0243] The synchronization service is started, the active transactions saved from the last checkpoint are loaded, and the sub-transactions within the active transactions that have not been marked with an end tag are reassigned to the grouping. At this point, the sub-transaction G1_TRX3 in active transaction A1 is assigned to the G1 group.

[0244] Compare the grouping information of subtransactions in the active transaction to see if they are consistent. The subtransaction G1_TRX3 is grouped as G1, while the checkpoint file saves it as G2. They are inconsistent, so the grouping of table T1 to be modified needs to be switched.

[0245] Add an end marker to the sub-transactions to be cut involving table T1 on active transaction A1.

[0246] A1

[0247] {

[0248] Group G1:

[0249] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0250] Group G2:

[0251] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0252] G2_TRX3[Table:T1, End Marker:1, Commit LSN:0, Commit Number:2]

[0253] }

[0254] Construct a virtual transaction G1_TRX4 that belongs to the G1 group and depends on the G2 group for the new group involved in table T1 in active transaction A1. The commit number of this virtual transaction is the number of the last operation of active transaction A1, and save the modification operation of changing the group of T1 from G2 to G1 to this virtual transaction.

[0255] A1

[0256] {

[0257] Group G1:

[0258] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0259] G1_TRX4 [Depends on G2, waiting LSN:0, end marker:1, commit LSN:0, commit number:2]

[0260] Group G2:

[0261] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0262] G2_TRX3[Table:T1, End Marker:1, Commit LSN:0, Commit Number:2]

[0263] }

[0264] The synchronization service has been started.

[0265] When an operation with LSN 23 is received, the active transaction A1 with ID 1 is located through the main transaction ID. At this moment, T1 belongs to group G1. A sub-transaction G1_TRX5 belonging to group G1 is created on the active transaction, and the operation is added to the sub-transaction. The operation number is used as the commit number of the sub-transaction.

[0266] A1

[0267] {

[0268] Group G1:

[0269] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0270] G1_TRX4 [Depends on G2, waiting LSN:0, end marker:1, commit LSN:0, commit number:2]

[0271] G1_TRX5[Table:T1, End Marker:0, Commit LSN:0, Commit Number:3]

[0272] Group G2:

[0273] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0274] G2_TRX3[Table:T1, End Marker:1, Commit LSN:0, Commit Number:2]

[0275] }

[0276] When an operation with LSN 24 is received, the active transaction A1 with ID 1 is located through the main transaction ID. At this moment, T1 belongs to group G1. There is already a sub-transaction G1_TRX5 belonging to group G1 in the active transaction, and its end marker is 0. Therefore, the operation is still added to the sub-transaction G1_TRX5, and the operation number is used as the commit number of the sub-transaction.

[0277] A1

[0278] {

[0279] Group G1:

[0280] G1_TRX1[Table:T1, End Marker:1, Commit LSN:0, Commit Number:1]

[0281] G1_TRX4 [Depends on G2, waiting LSN:0, end marker:1, commit LSN:0, commit number:2]

[0282] G1_TRX5[Table:T1, End Marker:0, Commit LSN:0, Commit Number:4]

[0283] Group G2:

[0284] G2_TRX2 [Depends on G1, waiting LSN:0, end marker:1, commit LSN:0, commit number:1]

[0285] G2_TRX3[Table:T1, End Marker:1, Commit LSN:0, Commit Number:2]

[0286] }

[0287] When a commit operation with LSN 25 is received, the active transaction A1 with ID 1 is located through the main transaction ID. All sub-transactions on the active transaction A1 are traversed and their commit LSNs are marked as the LSN of the current commit operation. If it is a virtual transaction, its commit LSN for the waiting group is set to 25 and then added to the committed transaction list of the corresponding group.

[0288] A1

[0289] {

[0290] Group G1:

[0291] G1_TRX1[Table:T1, End Marker:1, Commit LSN:25, Commit Number:1]

[0292] G1_TRX4 [Depends on G2, waiting LSN:25, end marker:1, commit LSN:25, commit number:2]

[0293] G1_TRX5[Table:T1, End Marker:0, Commit LSN:25, Commit Number:4]

[0294] G2 group: G2_TRX2 [Depends on G1, waiting LSN: 25, end marker: 1, commit LSN: 25, commit number: 1]

[0295] G2_TRX3[Table:T1, End Marker:1, Commit LSN:25, Commit Number:2]

[0296] }

[0297] The execution thread polls groups G1 and G2, and executes them in the committed list of the group according to the order of the combination of the commit LSN and commit number of the subtransaction.

[0298] The execution thread retrieves transaction G1_TRX1 from group G1, recording its commit LSN 25 and commit number 1. The execution thread then retrieves transaction G1_TRX4 from group G1, skipping group G1 (including transaction G1_TRX4) and retrieving transactions from group G2. The execution thread retrieves transaction G2_TRX2 from group G2 and executes it for database entry. The execution thread retrieves transaction G2_TRX3 from group G2, executes it for database entry, and records its commit LSN 25 and commit number 2. The sub-transactions in group G2 awaiting database entry have been completed, and the execution thread executes transaction G1_TRX4 from group G1. Upon completion, the program terminates abnormally. At this point, the commit LSN of group G1 is 25 and the commit number is 2; the commit LSN of group G2 is also 25 and the commit number is 2. The target synchronization service restarts, and after fault recovery, the sub-transaction G1_TRX5 from group G1 is executed.

[0299] Example 3:

[0300] like Figure 6 The diagram shown is an architectural schematic of a device for statically modifying data synchronization groups according to an embodiment of the present invention. The device for statically modifying data synchronization groups in this embodiment includes one or more processors 31 and a memory 32. Figure 6 Take a processor 31 as an example.

[0301] Processor 31 and memory 32 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0302] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the method for statically modifying data synchronization packets in Embodiment 1. The processor 31 executes the method for statically modifying data synchronization packets by running the non-volatile software program and instructions stored in the memory 32.

[0303] Memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 32 may optionally include memory remotely located relative to processor 31, which can be connected to processor 31 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0304] The program instructions / modules are stored in the memory 32. When executed by one or more processors 31, they perform the static modification data synchronization grouping method described in Embodiment 1 above, for example, performing the above-described... Figures 1-5 The steps shown.

[0305] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0306] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0307] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for statically modifying data synchronization groups, characterized in that, include: Based on the table information and new grouping configuration of all sub-transactions to be split, obtain the target group corresponding to the sub-transaction to be split; The table corresponding to the table information that is different between the target group and the original group is taken as the table to be modified; Based on the original grouping and the target grouping of the table to be modified, a virtual transaction of the table to be modified is constructed. The virtual transaction is used as a sub-transaction of the corresponding active transaction to be split. A commit message is received, and the sub-transaction of the active transaction corresponding to the commit message is used as a sub-transaction to be entered into the database. When the active transaction is an active transaction to be split, the corresponding virtual transaction is found from all the sub-transactions to be entered into the database of the active transaction to be split. This includes: identifying the active transactions corresponding to the sub-transactions to be split involving the table to be modified as active transactions to be split, and adding an end marker to the sub-transactions to be split; constructing sub-transactions belonging to the target group of the active transactions to be split as virtual transactions, and adding an end marker to the virtual transactions; constructing modification operations to divide the table to be modified into the target group according to the original group and the target group, and adding the modification operations to the virtual transactions; continuing to receive logs, receiving commit messages, and storing all sub-transactions of the active transactions to be split into the database according to the commit messages; Among them, all sub-transactions of the activity to be cut include the corresponding sub-transactions to be cut and the corresponding virtual transactions.

2. The method for statically modifying data synchronization groups according to claim 1, characterized in that, Before obtaining the target group corresponding to the sub-transaction to be split based on the table information involved in all sub-transactions to be split and the new group configuration, the following steps are included: Stop the synchronization service, save the original grouping of each original sub-transaction in all current active transactions and the table information involved in the original sub-transaction to the checkpoint file, and then exit the synchronization service. Obtain the new group configuration, modify the current group configuration of the system according to the new group configuration, and save it so that the system can divide the groups according to the current group configuration after restarting the synchronization service. Restart the synchronization service and obtain the original grouping and table information for each original sub-transaction in all active transactions based on the latest checkpoint file; The original sub-transactions that have not been marked with an end tag in all current active transactions are taken as sub-transactions to be cut, and the table information involved in the sub-transactions to be cut is obtained.

3. The method for statically modifying data synchronization groups according to claim 2, characterized in that, Before stopping the synchronization service, the original groupings of each original sub-transaction in all currently active transactions and the table information involved in the original sub-transactions are saved to a checkpoint file. This process, before exiting the synchronization service, includes: Receive logs and parse the logs to obtain at least one operation, the corresponding table information, the main transaction ID, and the operation number; Determine whether a corresponding active transaction exists based on the main transaction ID. If no such transaction exists, create the active transaction corresponding to the main transaction ID. Based on the table information, determine whether there is an original sub-transaction corresponding to the original group under the active transaction. If not, create the original sub-transaction; assign the operation to the original sub-transaction. In this context, the operation number of the last operation in each original sub-transaction is used as the commit number of the corresponding original sub-transaction; Upon receiving the commit message, the active transaction adds all original sub-transactions to the committed list of the corresponding original group in ascending order of the combination of the commit number and the commit LSN of all original sub-transactions in the active transaction, and then executes and inserts all original sub-transactions into the database according to the committed list.

4. The method for statically modifying data synchronization groups according to claim 1, characterized in that, The step of continuing to receive logs, receiving commit messages, and storing all sub-transactions of the activity transaction to be split into the database according to the commit messages includes: Receive a commit message, take the sub-transaction of the active transaction corresponding to the commit message as the sub-transaction to be entered into the database, and determine whether the active transaction is an active transaction to be split. When a transaction is to be split, iterate through all the sub-transactions to be inserted into the database from the transaction to be split and find the corresponding virtual transaction. The commit LSN of the commit message is used as the commit LSN and wait LSN of the virtual transaction; wherein, the commit number of the virtual transaction is the operation number of the last operation in the active transaction to be cut. Add all the sub-transactions to be added to the committed list of the corresponding target group in ascending order of the combination of the commit number and the commit LSN of all the sub-transactions to be added to the database. Based on the committed linked list and the waiting LSN, all the sub-transactions to be entered into the database.

5. The method for statically modifying data synchronization groups according to claim 4, characterized in that, The step of loading all pending sub-transactions into the database based on the committed linked list and the waiting LSN includes: Sequentially determine whether the commit LSN of the sub-transaction to be added to the database in the committed list is less than or equal to the commit LSN of the corresponding target group, and determine whether the commit number of the sub-transaction to be added to the database is less than or equal to the commit number of the target group. If yes, discard the sub-transaction to be entered into the database; if no, determine whether the sub-transaction to be entered into the database is a virtual transaction. When it is a virtual transaction, based on the commit LSN and commit number corresponding to the original group on which the virtual transaction depends, all sub-transactions to be entered into the database in the target group to which the virtual transaction belongs may be selectively entered into the database or not entered into the database. Specifically, after each sub-transaction to be inserted into the database is executed, the commit LSN of the sub-transaction to be inserted into the database is used as the commit LSN of the corresponding target group, and the commit number of the sub-transaction to be inserted into the database is used as the commit number of the target group.

6. The method for statically modifying data synchronization groups according to claim 5, characterized in that, When it is a virtual transaction, based on the commit LSN and commit number corresponding to the original group on which the virtual transaction depends, all sub-transactions to be entered into the database in the target group to which the virtual transaction belongs are selectively entered into the database or not entered into the database, including: Sequentially determine whether the commit LSN corresponding to the original group on which the virtual transaction depends is greater than or equal to the waiting LSN of the virtual transaction, and determine whether the commit number corresponding to the original group is greater than or equal to the commit number of the virtual transaction; If not, then all pending sub-transactions in the target group to which the virtual transaction belongs in the committed list will not be entered into the database. If so, then according to the committed linked list, all sub-transactions to be entered into the database in the target group to which the virtual transaction belongs.

7. The method for statically modifying data synchronization groups according to claim 1, characterized in that, The step of continuing to receive logs, receiving commit messages, and storing all sub-transactions of the activity transaction to be split into the database according to the commit messages includes: Restart the synchronization service, continue receiving logs, and determine the type of operation corresponding to the logs; When the operation is a DML operation, the DML operation is divided into corresponding sub-transactions according to the main transaction ID, table information and grouping configuration of the DML operation; wherein, when the sub-transaction has added an end mark, the sub-transaction is a sub-transaction of the active transaction to be split, and a corresponding target sub-transaction is created according to the grouping configuration, and the operation number of the DML operation is used as the commit number of the target sub-transaction. When the operation is a commit operation, the corresponding active transaction to be split is found according to the log corresponding to the commit operation. A commit message is added to all sub-transactions to be inserted into the database for the active transaction to be split. The commit LSN of the commit operation is used as the commit LSN of all sub-transactions to be inserted into the database. All sub-transactions to be inserted into the database are added to the committed list of the target group. All sub-transactions to be inserted into the database are inserted according to the committed list.

8. The method for statically modifying data synchronization groups according to any one of claims 1-7, characterized in that, When the synchronization service fails and restarts, the following applies: Based on the latest checkpoint file, restore the synchronized commit LSN, commit number, and virtual transactions in each target group; When performing the data entry after recovery, it is determined whether the commit LSN and commit number of the executed sub-transaction are less than or equal to the commit LSN and corresponding commit number of the last synchronized sub-transaction. If yes, then the execution and storage of the sub-transaction will not be performed; otherwise, the sub-transaction will be executed, and the current group configuration will be modified and saved according to the modification operation in the virtual transaction.

9. A device for statically modifying data synchronization groups, characterized in that, The method includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. When executed by the processor, the instructions are used to perform the method for statically modifying data synchronization groups as described in any one of claims 1-7.

Images