A parallel execution method and device based on transaction merging

Abstract

The present invention relates to a parallel execution method and device based on transaction merging. The method part mainly includes: table operation information establishment step; merging a single transaction into multiple transactions and setting the commit LSN, parallel LSN and start LSN of multiple transactions, and registering multiple transactions to the execution thread linked list; single transaction processing step; entering the corresponding step by judging the commit operation, conflict detection of table operation information, judgment of parallel LSN and judgment of start LSN; thread submission step. The method adopted by the present invention can optimize the parallel processing method to improve the parallelism of multi-transaction execution, thereby improving the performance of data synchronization.

Description

Technical Field

[0001] The present invention relates to the technical field of database data processing, and in particular to a parallel execution method and device based on transaction merging. Background Art

[0002] Currently, heterogeneous database replication technology based on log parsing is widely used. This technology captures database logs at the source end, parses the INSERT, UPDATE, and DELETE operations in the logs, and then sends them to the target end. The target end reversely generates the log information and restores it to SQL statements. Then, it uses the database general interface and applies it to the target database to achieve data replication. Therefore, in the process of real-time database replication, the execution efficiency of the target end is one of the important factors affecting data synchronization performance.

[0003] When executing synchronization transactions on the target database, the transaction operations need to be executed based on the transaction submission order. Otherwise, errors will occur when the related operations are executed, thus affecting the correctness of synchronization. In order to pursue the performance of data synchronization, the transaction merging strategy is often adopted when executing synchronization transactions. Small transactions are merged into large transactions. This can not only reduce the number of executions by merging the same operations of different transactions, but also reduce the number of transaction submissions, thereby achieving the purpose of improving synchronization performance. In addition, the synchronization performance can be further improved by increasing the number of threads executing transactions and adopting a multi-threaded parallel strategy to store multiple merged transactions at the same time. However, after adopting the strategies of transaction merging and parallel execution at the same time, it is also necessary to prevent and resolve data conflicts caused by parallel execution to ensure the consistency of data synchronization, otherwise the gains will outweigh the losses.

[0004] For the convenience of description, the present invention refers to the transaction in the source log as a "single transaction", and the transaction formed by merging multiple single transactions is called a "multi-transaction".

[0005] In the prior art, when the database is running, the tables involved in the operations in each single transaction may be different. After these single transactions are merged into multiple transactions, the tables involved in the merged multiple transactions are a set of tables involved in the single transaction. This will cause data conflicts when multiple transactions are executed in parallel. If the single transactions are not merged, those single transactions involving different tables can be executed in parallel. After these single transactions that could have been executed in parallel are merged into multiple transactions, the parallelism of parallel execution is reduced, thereby reducing the efficiency of synchronization. This reduction may offset the performance improvement effect brought by transaction merging.

[0006] In view of this, how to overcome the defects of the prior art and solve the above-mentioned technical problems is a difficult problem to be solved in this technical field. Summary of the invention

[0007] In view of the above defects or improvement needs of the prior art, the present invention provides a parallel execution method and device based on transaction merging. In view of the problem in the prior art that the parallelism of parallel execution is reduced after a single transaction is merged into multiple transactions, thereby reducing the synchronization efficiency, the parallel processing method is optimized to improve the parallelism of multi-transaction execution, thereby improving the performance of data synchronization.

[0008] The embodiment of the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a parallel execution method based on transaction merging, comprising:

[0010] Table operation information creation step: Get the table name involved in the current operation, determine whether the table operation information of the table already exists in the current transaction, if not, combine the table name of the current operation with the current operation and associate them with the LSN of the current operation to form the table operation information and store it in the current transaction;

[0011] Merge a single transaction into multiple transactions and set the commit LSN, parallel LSN and start LSN of the multiple transactions, and register the multiple transactions to the execution thread list;

[0012] Single transaction processing step: the execution thread extracts the next single transaction from multiple transactions and starts executing;

[0013] Enter the corresponding steps by judging the commit operation, detecting conflicts in table operation information, judging the parallel LSN, and judging the start LSN;

[0014] Thread commit step: Determine whether the current single transaction is the last single transaction. If so, execute the thread commit operation; if not, re-enter the single transaction processing step.

[0015] Furthermore, the deployment preparation steps are also included:

[0016] Deploy data synchronization services on both the source database and the target database;

[0017] After the target-side data synchronization service is started, initialize the pending execution list, execution thread, wake-up list, and execution thread list;

[0018] After the target-side data synchronization service receives the source-side operation, it classifies and manages it according to the transaction ID of the operation. If the current operation is a commit operation, the transaction is stored in the to-be-executed linked list in the order of the LSN size of the commit operation; if it is not a commit operation, it enters the table operation information establishment step.

[0019] Further, the step of merging a single transaction into multiple transactions and setting the commit LSN, parallel LSN and start LSN of the multiple transactions, and registering the multiple transactions into the execution thread linked list specifically includes:

[0020] The execution thread extracts N single transactions from the pending execution linked list in the order of transaction commit LSNs and merges them into a multi-transaction, sets the commit LSN of the last single transaction as the commit LSN of the multi-transaction, and then registers it in the execution thread linked list according to the commit LSN of the multi-transaction;

[0021] When merging single transactions, the table operation information involved in the single transactions is also merged. If multiple single transactions have the same table operation, the table operation information of the single transaction with the smallest commit LSN shall prevail, and the commit LSN shall be set as the parallel LSN of the table operation information;

[0022] Set the LSN of the first operation involving each table in the multi-transaction as the starting LSN.

[0023] Further, the step of entering corresponding steps by judging the commit operation, detecting the conflict of table operation information, judging the parallel LSN and judging the start LSN specifically includes:

[0024] Commit operation judgment step: select to enter the conflict detection step or thread submission step according to the operation type of the next operation of the current single transaction;

[0025] Conflict detection step: Enter the commit operation judgment step or the parallel LSN judgment step according to whether there is a previous execution thread and whether there is corresponding table operation information in the previous execution thread;

[0026] Parallel LSN determination step: select to enter the conflict detection step or the starting LSN determination step according to the LSN of the current operation and the size of the parallel LSN in the table operation information;

[0027] Starting LSN determination step: Choose to enter the conflict detection step or wait for wake-up according to the LSN of the current operation and the size of the starting LSN in the table operation information.

[0028] Furthermore, the submission operation determination step specifically includes:

[0029] Extract the next operation in the current single transaction and determine whether the operation is a commit operation;

[0030] If it is a commit operation, it enters the thread commit step, if not, it enters the conflict detection step.

[0031] Furthermore, the conflict detection step specifically includes:

[0032] Extract the previous execution thread from the execution thread linked list. If there is no previous execution thread, execute the current operation and then enter the submission operation judgment step. If there is, proceed to the next step of conflict detection.

[0033] The corresponding table operation information is searched in the detected execution thread according to the table name and operation type of the current operation. If not found, the conflict detection step is re-entered; if found, the parallel LSN judgment step is entered.

[0034] Furthermore, when searching for corresponding table operation information, the search rules include:

[0035] If the current operation is an INSERT operation, then search for table operation information of UPDATE and DELETE operations in the detected execution thread;

[0036] If the current operation is an UPDATE operation, the table operation information of the INSERT, UPDATE and DELETE operations is searched in the detected execution thread;

[0037] If the current operation is a DELETE operation, the table operation information of the INSERT and UPDATE operations is searched in the detected execution thread.

[0038] Furthermore, the parallel LSN determination step specifically includes:

[0039] Determine whether the LSN of the current operation is less than the parallel LSN in the table operation information of the detected execution thread;

[0040] If it is less than, re-enter the conflict detection step; if it is not less than, enter the starting LSN determination step.

[0041] Furthermore, the starting LSN determination step specifically includes:

[0042] Determine whether the LSN of the current operation is less than the starting LSN in the table operation information of the detected execution thread;

[0043] If it is less than, re-enter the conflict detection step; if it is not less than, add the current execution thread to the wake-up linked list of the checked execution thread, and execute it after the checked execution thread is submitted.

[0044] On the other hand, the present invention provides a parallel execution device based on transaction merging, specifically: including at least one processor and a memory, at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the parallel execution method based on transaction merging in the first aspect.

[0045] Compared with the prior art, the beneficial effect of the present invention is that by establishing table operation information for each single transaction and multiple transactions, it is possible to decide whether to execute in parallel by comparing the table operation information, so as to improve the parallelism of parallel execution while avoiding data conflicts, thereby improving the performance of data synchronization.

[0046] Specifically, each single transaction in multiple transactions may operate on different tables. For each single transaction, the table operation information involved is recorded. After multiple single transactions are merged into multiple transactions, if multiple single transactions have the same table operation, the table operation information of the single transaction with the smallest commit LSN shall prevail, and the commit LSN shall be set as the parallel LSN of the table operation information. All operations smaller than the parallel LSN in the execution thread can be executed in parallel. In this way, some operations on non-conflicting tables can be excluded from the conditions for parallel execution, thereby avoiding a reduction in parallelism.

[0047] Secondly, a starting LSN is set for each table involved in multiple transactions. For the transactions that are submitted later and are arranged behind the execution thread linked list, if the LSN of the operation to be executed is smaller than the starting LSN, it means that although the transaction is submitted later than the execution thread in front, the operation is executed first. In this case, these operations can be executed in parallel. The starting LSN and the parallel LSN are combined to select the operations that can be executed in parallel, which can increase the degree of parallelism and improve the performance of data synchronization after transaction merging. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A flow chart of a parallel execution method based on transaction merging provided in Example 1 of the present invention;

[0050] Figure 2 A specific flow chart of the deployment preparation steps provided in Example 1 of the present invention;

[0051] Figure 3 A specific flow chart of step 200 provided in Example 1 of the present invention;

[0052] Figure 4 A specific flow chart of step 400 provided in Example 1 of the present invention;

[0053] Figure 5 A specific flow chart of step 500 provided in Example 1 of the present invention;

[0054] Figure 6 A specific flow chart of step 600 provided in Example 1 of the present invention;

[0055] Figure 7 A specific flow chart of step 700 provided in Example 1 of the present invention;

[0056] Figure 8 A specific transaction execution flow chart provided for Example 1 of the present invention;

[0057] Fig. 9 A schematic diagram of a parallel execution device structure based on transaction merging provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0060] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0061] Embodiment 1:

[0062] like Figure 1 As shown, an embodiment of the present invention provides a parallel execution method based on transaction merging, and the specific steps are as follows.

[0063] Step 100 (table operation information establishment step): obtain the table name involved in the current operation, determine whether the table operation information of the table already exists in the current transaction, if not, combine the table name of the current operation with the current operation and associate it with the LSN of the current operation to form table operation information and store it in the current transaction. In this step, the current operation includes INSERT (insert) operation, UPDATE (update) operation, DELETE (delete) operation, COMMIT (commit) operation and ROLLBACK (rollback) operation. This step means that the current transaction has corresponding operations on the involved tables after starting from the LSN of the current operation, and the operation types are divided into INSERT, UPDATE, DELETE, COMMIT and ROLLBACK operations.

[0064] Step 200: Merge a single transaction into multiple transactions and set the commit LSN, parallel LSN and start LSN of the multiple transactions, and register the multiple transactions into the execution thread linked list.

[0065] Step 300 (single transaction processing step): the execution thread extracts the next single transaction from the multiple transactions and starts to execute. In this step, since the execution thread executes multiple transactions merged from multiple single transactions, it executes the single transactions in the order of merging in the multiple transactions.

[0066] Step 400 to step 700: Enter the corresponding steps by judging the commit operation, detecting conflicts in table operation information, judging the parallel LSN, and judging the start LSN.

[0067] Step 800 (thread commit step): determine whether the current single transaction is the last single transaction, if so, execute the thread commit operation; if not, re-enter the single transaction processing step.

[0068] The above steps 400 to 700 specifically include step 400, step 500, step 600 and step 700. Each step is as follows:

[0069] Step 400 (commit operation determination step): select to enter the conflict detection step or the thread commit step according to the operation type of the next operation of the current single transaction.

[0070] Step 500 (conflict detection step): enter the commit operation determination step or the parallel LSN determination step according to whether there is a previous execution thread and whether there is corresponding table operation information in the previous execution thread.

[0071] Step 600 (parallel LSN determination step): select to enter the conflict detection step or the starting LSN determination step according to the LSN of the current operation and the size of the parallel LSN in the table operation information.

[0072] Step 700 (starting LSN determination step): select to enter the conflict detection step or wait for wake-up according to the LSN of the current operation and the size of the starting LSN in the table operation information.

[0073] Through the above steps 100 to 700, the embodiment of the present invention establishes table operation information for each single transaction and multiple transactions, so that conflicts can be detected by comparing the table operation information, thereby deciding whether to execute in parallel, so as to improve the parallelism of parallel execution while avoiding data conflicts, thereby improving the performance of data synchronization.

[0074] In this preferred embodiment, it also includes deployment preparation steps, such as Figure 2 As shown in the figure, the deployment preparation steps include:

[0075] Step 10: Deploy data synchronization services on both the source database and the target database. In this step, deploy the log parsing module of the data synchronization service on the source database to parse log operations, and deploy the data synchronization receiving service on the target database to receive operations sent by the source data synchronization and store them in the database.

[0076] Step 20: After the target data synchronization service is started, initialize the pending execution linked list, execution thread, wake-up linked list and execution thread linked list. In this step, the pending execution linked list is used to store the pending single transactions in the linked list in the order of the LSN size of the transaction commit operation, waiting for execution and storage. Each execution thread is used to extract multiple single transactions from the pending execution linked list and merge them into multiple transactions, set the commit LSN of the last single transaction as the commit LSN of the multiple transactions, and then execute and store them. In addition, each execution thread also needs to initialize a wake-up linked list to store those execution threads that are waiting to be woken up after they are committed. The execution thread linked list is used to register the execution threads in the linked list in the order of the size of the commit LSN of the current multiple transactions to be executed. The operation of executing multiple transactions can only be started after the registration is completed.

[0077] Step 30: After receiving the source operation, the target-side data synchronization service classifies and manages it according to the transaction ID of the operation. If the current operation is a commit operation, the transaction is stored in the to-be-executed linked list in the order of the LSN size of the commit operation; if it is not a commit operation, the table operation information establishment step is entered.

[0078] like Figure 3 As shown, in this preferred embodiment, step 200 (merging a single transaction into multiple transactions and setting the commit LSN, parallel LSN and start LSN of the multiple transactions, and registering the multiple transactions to the execution thread linked list) specifically includes:

[0079] Step 201: The execution thread extracts N single transactions from the to-be-executed linked list in the order of transaction commit LSNs and merges them into a multi-transaction, sets the commit LSN of the last single transaction as the commit LSN of the multi-transaction, and then registers the multi-transaction in the execution thread linked list according to the commit LSN.

[0080] Step 202: When merging single transactions, the table operation information involved in the single transactions is merged at the same time. If multiple single transactions have the same table operation, the table operation information of the single transaction with the smallest commit LSN shall prevail, and the commit LSN shall be set as the parallel LSN of the table operation information. It should be noted that after determining that the table operation information of the single transaction with the smallest commit LSN is the table operation information of multiple transactions, the table operation information of other single transactions in the multiple transactions shall be discarded.

[0081] Step 203: Set the LSN of the first operation of each table involved in the multi-transaction as the starting LSN. After the setting is completed, the table operation information of the multi-transaction will have a starting LSN and a parallel LSN. Specifically, the starting LSN refers to the LSN of the first operation of a certain operation type (INSERT, UPDATE or DELETE) corresponding to the table in the multi-transaction. In conjunction with the current transaction mentioned in step 100, the current transaction will only have corresponding operations on the table involved after starting from the LSN of the current operation. It should be noted that there may be operations on multiple tables in a single transaction, and when the transaction scale is large, this method can subdivide the starting LSN of each table in the transaction, so that the starting LSN of the last operation table executed in the transaction may be much larger than the starting LSN of the single transaction, and a larger starting LSN is conducive to increasing the parallelism of transactions submitted after it. Otherwise, the tables involved in the single transaction all use the first operation LSN of the single transaction as the starting LSN, so the parallelism of the single transaction and the transactions submitted after it during execution is not as good as the former.

[0082] like Figure 4 As shown, in this preferred embodiment, step 400 (submission operation determination step) specifically includes:

[0083] Step 401: extract the next operation in the current single transaction and determine whether the operation is a commit operation.

[0084] Step 402: If it is a commit operation, then enter the thread commit step; if not, then enter the conflict detection step.

[0085] like Figure 5 As shown, in this preferred embodiment, step 500 (conflict detection step) specifically includes:

[0086] Step 501: extract the previous execution thread from the execution thread linked list. If there is no previous execution thread, execute the current operation and then enter the submission operation judgment step; if there is, proceed to the next step of conflict detection. It should be noted that the previous execution thread of this step refers to the execution thread that is one before the execution thread of the current operation during the first detection. In the conflict detection of the same operation, each time the step is re-entered, the execution thread that is more previous will be extracted and detected.

[0087] Step 502: Search the corresponding table operation information in the detected execution thread through the table name and operation type of the current operation. If not found, re-enter the conflict detection step; if found, enter the parallel LSN judgment step. In this step, it is also necessary to first obtain the table name, operation type and operation LSN involved in the current operation to be executed, and then start the judgment. In addition, the detected execution thread represents the previous execution thread corresponding to each time this step is entered. Each time the conflict detection step is re-entered, the previous execution thread is re-extracted as the detected execution thread for re-judgment.

[0088] In this preferred embodiment, when searching for corresponding table operation information, the search rules include:

[0089] (A) If the current operation is an INSERT operation, then search for table operation information of UPDATE and DELETE operations in the detected execution thread. This is because INSERT and INSERT operations are compatible and will not cause conflicts, so there is no need to search for INSERT operations, while UPDATE and DELETE operations will conflict with INSERT operations, so UPDATE and DELETE operations need to be searched.

[0090] (B) If the current operation is an UPDATE operation, the table operation information of the INSERT, UPDATE, and DELETE operations is searched in the detected execution thread. This is because the UPDATE operation will conflict with these three operations, so they all need to be searched.

[0091] (C) If the current operation is a DELETE operation, then search for table operation information of INSERT and UPDATE operations in the detected execution thread. This is because DELETE and DELETE operations are compatible and will not cause conflicts, so there is no need to search for DELETE operations, while INSERT and UPDATE operations will conflict with DELETE operations, so they need to be searched.

[0092] like Figure 6 As shown, in this preferred embodiment, step 600 (parallel LSN determination step) specifically includes:

[0093] Step 601: Determine whether the LSN of the current operation is smaller than the parallel LSN in the table operation information of the detected execution thread.

[0094] Step 602: If it is less than, re-enter the conflict detection step; if it is not less than, enter the starting LSN judgment step. It should be noted that the less than in the above steps refers to the case where the operation LSN is greater than the starting LSN but less than the parallel LSN. At this time, these operations are in a state of parallel execution at the source end, so they can also be parallel at the target end. At this time, the check of this execution thread can be skipped, and the previous execution thread can be checked to see if there is a conflict.

[0095] like Figure 7 As shown, in this preferred embodiment, step 700 (starting LSN determination step) specifically includes:

[0096] Step 701: Determine whether the LSN of the current operation is smaller than the start LSN in the table operation information of the detected execution thread.

[0097] Step 702: If it is less than, re-enter the conflict detection step; if it is not less than, add the current execution thread to the wake-up linked list of the execution thread being checked, and wait for the execution thread being checked to be submitted before execution. It should be noted that in this step, when the operation LSN is less than the starting LSN of the execution thread being checked, it means that this operation is executed first on the source end, so it can also be executed in parallel on the target end. At this time, the check of this execution thread can be skipped, and the conflict detection step can be re-entered to continue to check whether there is a conflict in the previous execution thread; if it is not less than, it means that there is a possibility of conflict between the current operation and the execution thread being checked, so it is necessary to add the current execution thread to the wake-up linked list of the execution thread being checked, and wait for the execution thread being checked to be submitted before executing the current execution thread.

[0098] In this preferred embodiment, in step 800 (thread submission step), after the execution thread submission operation is performed, all the execution threads waiting in the wake-up list are also awakened, and then the execution thread is removed from the execution thread list to complete the execution of multiple transactions.

[0099] The above is a detailed description of each step of the method provided in this embodiment, and the specific transaction execution process after these steps are combined into one is as follows: Figure 8 As shown, Figure 8The specific transaction execution process is as follows: start executing transactions; extract the next single transaction in the current multi-transaction; extract the next operation of the current single transaction, and determine whether the current operation is a commit operation; if it is a commit operation, determine whether it is the last single transaction. If it is the last single transaction, execute the commit and wake up all the execution threads in the wake-up linked list, and then end the execution of the multi-transaction. If it is not the last single transaction, return to the step of taking the next single transaction of the current multi-transaction; if the current operation is not a commit operation, take the previous execution thread in the execution thread linked list, and determine whether the operation LSN is less than the parallel LSN of the table operation information of the previous execution thread. If it is less, take the previous execution thread for judgment. If it is not less than, determine whether the operation LSN is less than the starting LSN of the table operation information. If it is less than, take the previous execution thread for judgment. If it is not less than, add the current execution thread to the wake-up linked list of the previous execution thread and wait for the other party to submit and wake up; when taking the previous execution thread in the execution thread linked list, if there is no previous execution thread, execute the current operation, and then return to the step of taking the next operation of the current single transaction.

[0100] In summary, in this embodiment, the table operated by each single transaction in each multi-transaction may be different, and the table operation information involved is recorded for each single transaction. After multiple single transactions are merged into multiple transactions, if there are multiple single transactions with the same table operation, the table operation information of the single transaction with the smallest commit LSN shall prevail, and the commit LSN shall be set as the parallel LSN of the table operation information, and the operations smaller than the parallel LSN in the execution thread can be executed in parallel. In this way, it is possible to avoid excluding the operations of some non-conflicting tables from the conditions for parallel execution, thereby avoiding the reduction of parallelism. If this is not the case, but all tables are uniformly using the commit LSN of the first transaction in the multi-transaction as the parallel LSN, the operations of some non-conflicting tables will be excluded from the conditions for parallel execution, which greatly reduces the parallelism.

[0101] Secondly, this embodiment sets a starting LSN for each table involved in multiple transactions. For transactions that are submitted later and are arranged behind the execution thread linked list, if the LSN of the operation to be executed is smaller than the starting LSN, it means that although the transaction is submitted later than the execution thread in front, the operation is executed first. In this case, these operations can be executed in parallel. By combining the starting LSN and the parallel LSN to select the operations that can be executed in parallel, the degree of parallelism can be increased, thereby improving the performance of data synchronization after transaction merging.

[0102] Embodiment 2:

[0103] Based on the parallel execution method based on transaction merging provided in Example 1, this Example 2 describes the present invention in more detail through a specific application scenario.

[0104] In this embodiment, take the source database table T1 (ID VARCHAR (100)) and table T2 (ID VARCHAR (100)) as examples. They each have 2 rows of data, and their data is distributed in the database as follows:

[0105] T1 table ID column T2 table ID column T1_1 T2_1 T1_2 T2_2

[0106] The source side has the following log operations:

[0107] LSN Transaction ID operate 1 1 UPDATE T1 SET ID='T1_11' WHERE ID="T1_1'; 2 3 UPDATE T2 SET ID='T2_11'WHERE ID='T2_1'; 3 1 COMMIT; 4 2 UPDATE T2 SET ID='T2_22'WHERE ID='T2_2'; 5 3 UPDATE T1 SET ID='T1_111' WHERE ID='T1_11'; 6 4 UPDATE T1 SET ID='T1_22'WHERE ID='T1_2'; 7 2 COMMIT; 8 3 COMMIT; 9 4 COMMIT;

[0108] Specifically, based on the above application scenario, the steps for establishing the execution thread embodied in Embodiment 1 in Embodiment 2 are as follows:

[0109] 1. Create two execution threads A and B.

[0110] 2. The target-side data synchronization service receives logs, manages them by transaction ID, and records the table operation information of each table. The following results are obtained:

[0111]

[0112] 3. Execution thread A extracts transactions with single transaction IDs 1 and 2 and merges them into a multi-transaction, uses the commit LSN 7 of single transaction 2 as the commit LSN of the multi-transaction, and registers it in the execution thread linked list. When merging transactions, table operation information must also be merged, and the parallel LSN of the table operation information must be set.

[0113] 4. Execution thread B extracts transactions with single transaction IDs 3 and 4 and merges them into a multi-transaction. It uses the commit LSN 9 of single transaction 4 as the commit LSN of the multi-transaction and registers it in the execution thread linked list. At this time, execution thread A already exists in the execution thread linked list. Since B's commit LSN is larger than A's, A is ranked ahead of B in the execution thread linked list. When merging transactions, table operation information must also be merged and the parallel LSN of the table operation information must be set.

[0114] The multi-transaction table operation information formed by the combined execution of threads A and B is as follows:

[0115]

[0116] 5. Execution threads A and B start executing in parallel.

[0117] After the above execution threads A and B are established and the preparation work is completed, they officially enter the parallel execution stage. The execution process of this stage is shown in the following table:

[0118]

[0119]

[0120]

[0121] In the above example, in the existing solution, when a single transaction is merged into multiple transactions, since both multiple transactions contain UPDATE operations of table T1 and table T2, execution thread B needs to wait until execution thread A commits before it can start execution, because execution thread B cannot determine whether the data it updates conflicts with execution thread A. Serial execution can ensure data synchronization consistency but sacrifices synchronization performance.

[0122] As can be seen from the above parallel execution process of this embodiment, when the execution thread B executes the table T2, the operation type is UPDATE, and the operation LSN is 2, it can be executed in parallel because it is smaller than the parallel LSN (7) in the operation information of the T2 table in the execution thread A, which is a point to improve the synchronization performance. The next operation involves the table T1, the operation type is UPDATE, and the operation LSN is 5, which is larger than the parallel LSN (3) in the operation information of the T1 table in the previous execution thread A, and is also larger than the starting LSN (1) in the operation information of the T1 table, so it cannot be executed in parallel, so the execution thread B adds itself to the wake-up linked list of the execution thread A, and waits for the execution thread A to submit before executing, thereby ensuring the consistency of parallel execution data synchronization.

[0123] Embodiment 3:

[0124] Based on the parallel execution method based on transaction merging provided in Example 1, this Example 3 describes the present invention in more detail through another specific application scenario different from Example 2.

[0125] In this embodiment, the source database has a table T1 (ID VARCHAR(100)).

[0126] The source side has the following log operations:

[0127]

[0128]

[0129] Specifically, based on the above application scenario, the steps for establishing the execution thread embodied in Embodiment 1 in Embodiment 3 are as follows:

[0130] 1. Create two execution threads A and B.

[0131] 2. The target-side data synchronization service receives logs, manages them by transaction ID, and records the table operation information of each table. The following results are obtained:

[0132] Single transaction ID Table name Operation Type Starting LSN Parallel LSN 1 T1 INSERT 1 / 2 T1 UPDATE 4 / 3 T1 INSERT 2 / 4 T1 INSERT 5 /

[0133] 3. Execution thread A extracts transactions with single transaction IDs 1 and 2 and merges them into a multi-transaction, uses the commit LSN 6 of single transaction 2 as the commit LSN of the multi-transaction, and registers it in the execution thread linked list. When merging transactions, table operation information must also be merged, and the parallel LSN of the table operation information must be set.

[0134] 4. Execution thread B extracts transactions with single transaction IDs 3 and 4 and merges them into a multi-transaction. It uses the commit LSN 8 of single transaction 4 as the commit LSN of the multi-transaction and registers it in the execution linked list. At this time, execution thread A already exists in the execution thread linked list. Since B's commit LSN is larger than A's, A is ranked ahead of B in the execution thread linked list. When merging transactions, table operation information must also be merged and the parallel LSN of the table operation information must be set.

[0135] The multi-transaction table operation information formed by the combined execution of threads A and B is as follows:

[0136]

[0137] As can be seen from the above table, execution thread A has table operation information of two types of operations, INSERT and UPDATE, for table T1, which correspond to different parallel LSNs, respectively. This creates conditions for subsequent parallelism.

[0138] 5. Execution threads A and B start executing in parallel.

[0139] After the above execution threads A and B are established and the preparation work is completed, they officially enter the parallel execution stage. The execution process of this stage is shown in the following table:

[0140]

[0141]

[0142] In the above example, in the existing solution, when a single transaction is merged into multiple transactions, since both transactions include table T1 operations, execution thread B needs to wait until execution thread A commits before it can start execution. This is because execution thread B cannot determine whether the data it inserts conflicts with the data updated by execution thread A. Serial execution can ensure data synchronization consistency but sacrifices synchronization performance.

[0143] As can be seen from the above parallel execution process, when the execution thread B executes the operation of table T1, the operation type is INSERT, and the operation LSN is 2, it can be executed in parallel because it is smaller than the parallel LSN (6) in the UPDATE operation information of the T1 table in the execution thread A, which is a point to improve the synchronization performance. The next operation involves table T1, the operation type is INSERT, and the operation LSN is 5, which is also smaller than the parallel LSN (6) in the UPDATE operation information of the T1 table in the previous execution thread A, and can be executed in parallel, which is another point to improve the synchronization performance. From the above execution process, it can be seen that the execution thread A and the execution thread B are completely executed in parallel, which greatly improves the efficiency of data synchronization.

[0144] Embodiment 4:

[0145] Based on the parallel execution methods based on transaction merging provided in the above-mentioned embodiments 1 to 3, the present invention further provides a parallel execution device based on transaction merging that can be used to implement the above-mentioned methods, such as Fig. 9 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The parallel execution device based on transaction merging of this embodiment includes one or more processors 21 and a memory 22. Fig. 9 A processor 21 is taken as an example.

[0146] The processor 21 and the memory 22 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0147] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the parallel execution method based on transaction merging in Embodiments 1 to 3. The processor 21 executes various functional applications and data processing of the parallel execution device based on transaction merging by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, implements the parallel execution method based on transaction merging in Embodiments 1 to 3.

[0148] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the parallel execution method based on transaction merging in the above-mentioned embodiments 1 to 2 is executed. For example, the above-described Figures 1 to 8 The steps shown.

[0150] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A parallel execution method based on transaction merging, characterized in that: include: Table operation information creation step: Get the table name involved in the current operation, determine whether the table operation information of the table already exists in the current transaction, if not, combine the table name of the current operation with the current operation and associate them with the LSN of the current operation to form the table operation information and store it in the current transaction; Merge a single transaction into multiple transactions and set the commit LSN, parallel LSN and start LSN of the multiple transactions, and register the multiple transactions to the execution thread list; Specifically, the execution thread extracts N single transactions from the to-be-executed linked list in the order of transaction commit LSNs and merges them into a multi-transaction, and sets the commit LSN of the last single transaction as the commit LSN of the multi-transaction, and then registers the multi-transactions in the execution thread linked list according to the commit LSNs; when merging single transactions, the table operation information involved in the single transaction is merged at the same time. If multiple single transactions have the same table operation, the table operation information of the single transaction with the smallest commit LSN shall prevail, and the commit LSN shall be set as the parallel LSN of the table operation information; the LSN of the first operation of each table involved in the multi-transaction is set as the starting LSN; Single transaction processing step: the execution thread extracts the next single transaction from multiple transactions and starts executing; Enter the corresponding steps by judging the commit operation, detecting conflicts in table operation information, judging the parallel LSN, and judging the start LSN; Thread commit step: Determine whether the current single transaction is the last single transaction. If so, execute the thread commit operation; if not, re-enter the single transaction processing step.

2. The parallel execution method based on transaction merging according to claim 1, characterized in that: It also includes deployment preparation steps: Deploy data synchronization services on both the source database and the target database; After the target-side data synchronization service is started, initialize the pending execution list, execution thread, wake-up list, and execution thread list; After receiving the source operation, the target data synchronization service classifies and manages the operations according to the transaction ID. If the current operation is a commit operation, the transaction is stored in the pending execution list in the order of the LSN size of the commit operation. If it is not a submit operation, enter the table operation information creation step.

3. The parallel execution method based on transaction merging according to claim 1, characterized in that: The steps of entering corresponding steps by judging the commit operation, detecting the conflict of table operation information, judging the parallel LSN and judging the start LSN specifically include: Commit operation judgment step: select to enter the conflict detection step or thread submission step according to the operation type of the next operation of the current single transaction; Conflict detection step: Enter the commit operation judgment step or the parallel LSN judgment step according to whether there is a previous execution thread and whether there is corresponding table operation information in the previous execution thread; Parallel LSN determination step: select to enter the conflict detection step or the starting LSN determination step according to the LSN of the current operation and the size of the parallel LSN in the table operation information; Starting LSN determination step: Choose to enter the conflict detection step or wait for wake-up according to the LSN of the current operation and the size of the starting LSN in the table operation information.

4. The parallel execution method based on transaction merging according to claim 3, characterized in that: The submission operation determination step specifically includes: Extract the next operation in the current single transaction and determine whether the operation is a commit operation; If it is a commit operation, it enters the thread commit step, if not, it enters the conflict detection step.

5. The parallel execution method based on transaction merging according to claim 4, characterized in that: The conflict detection step specifically includes: Extract the previous execution thread from the execution thread linked list. If there is no previous execution thread, execute the current operation and then enter the submission operation judgment step. If there is, proceed to the next step of conflict detection. The corresponding table operation information is searched in the detected execution thread according to the table name and operation type of the current operation. If not found, the conflict detection step is re-entered; if found, the parallel LSN judgment step is entered.

6. The parallel execution method based on transaction merging according to claim 5, characterized in that: When searching for corresponding table operation information, the search rules include: If the current operation is an INSERT operation, search for table operation information of UPDATE and DELETE operations in the detected execution thread; If the current operation is an UPDATE operation, the table operation information of the INSERT, UPDATE and DELETE operations is searched in the detected execution thread; If the current operation is a DELETE operation, the table operation information of the INSERT and UPDATE operations is searched in the detected execution thread.

7. The parallel execution method based on transaction merging according to claim 5, characterized in that: The parallel LSN determination step specifically includes: Determine whether the LSN of the current operation is less than the parallel LSN in the table operation information of the detected execution thread; If it is less than, re-enter the conflict detection step; if it is not less than, enter the starting LSN determination step.

8. The parallel execution method based on transaction merging according to claim 7, characterized in that: The starting LSN determination step specifically includes: Determine whether the LSN of the current operation is less than the starting LSN in the table operation information of the detected execution thread; If it is less than, re-enter the conflict detection step; if it is not less than, add the current execution thread to the wake-up linked list of the checked execution thread, and execute it after the checked execution thread is submitted.

9. A parallel execution device based on transaction merging, characterized in that: The method comprises at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after being executed by the processor, the instructions are used to complete the parallel execution method based on transaction merging as described in any one of claims 1 to 8.

Images