A method, device and system for parallel execution of operations based on a conflicting transaction linked list

Abstract

The present invention relates to a method, device, and system for parallel execution of operations based on a conflicting transaction list. The method portion of the method primarily comprises: a working thread extracts a transaction from a committed transaction list in the order in which the transaction is committed, determines whether the current transaction is a rollback transaction, and if so, waits for all conflicting transactions in the conflicting transaction list of the current transaction to be executed before waking up the working thread; the working thread uses the conflicting TRXID of the current pending operation to search a hash table P for a transaction with the conflicting TRXID; if so, waits for wakeup; otherwise, the working thread uses the information in the current pending operation table to search a hash table G for a transaction operating on the same table, where the transaction commit LSN is smaller than the current transaction and the transaction start LSN is smaller than the conflicting LSN of the current operation; if so, waits for wakeup; otherwise, determines that no conflict is detected and executes the current operation. The present invention can effectively increase the concurrency of operation execution, thereby improving synchronization performance.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, device and system for executing operations in parallel based on a conflicting transaction linked list. Background Art

[0002] In a database real-time synchronization system (supporting heterogeneous architecture) based on a log parsing architecture, the source-side data synchronization service is responsible for capturing the operation logs of the source database, parsing the logs to restore the corresponding database operations, and then sending them to the target-side data synchronization service. The target-side data synchronization service is responsible for executing these operations in the target database, thereby completing the data synchronization process.

[0003] When the target-side data synchronization service synchronizes operations on the target-side database, it often needs to execute them in the order in which the transactions were committed. This is because when the previous transaction has not yet completed synchronization, the operation of the next transaction cannot be executed and stored concurrently with the previous transaction operation without clarifying whether it conflicts with the previous transaction. Execution can only begin after all transactions preceding the current operation have been committed. This greatly affects the concurrency of transaction storage. The current effective method is to determine whether parallel execution is possible by judging the size of the log LSN of the operation and the LSN of the previously submitted transaction commit operation. If the LSN of the currently executing transaction operation is smaller than the LSN of the commit operation of another transaction currently being executed, then this operation can be executed concurrently with that transaction. This strategy helps improve concurrency, but the performance improvement is small.

[0004] In view of this, how to overcome the defects of existing technologies and how to improve the parallel execution of target-side data synchronization services to improve data synchronization performance have become important technical issues that need to be urgently addressed in the industry. Summary of the Invention

[0005] In view of the defects or improvement needs in the existing technology: How to improve the parallel execution strength of the target-side data synchronization service and thus improve the performance of data synchronization. In order to improve the performance of synchronization, the target-side data synchronization service sets its dependency information for each operation when receiving the operation, so that it can execute concurrently with a finer granularity during execution, thereby improving performance. Based on this, the present invention provides a method, device and system for parallel execution of operations based on a conflicting transaction linked list. When the target-side data synchronization service of this method receives the synchronization transaction operation, it receives it in the order in which the source-side database log is generated. Then, the order of each received operation is the basis for them to detect whether there is a conflict. The present invention extracts the primary key or index or custom selection column on each received operation as the KEY based on the order in which the operation is received, and searches for whether this KEY has the same KEY as the operation of other transactions received previously within the set range. If so, the execution of this operation needs to wait for the conflicting transaction to be submitted before it can start execution; otherwise, it can be executed concurrently with the transaction submitted before itself, maximizing the parallelism of transaction execution, thereby effectively improving the performance of synchronization.

[0006] The present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for executing operations in parallel based on a conflicting transaction linked list, comprising:

[0008] The target-side data synchronization service creates hash tables P and G and several worker threads.

[0009] The worker thread extracts a transaction from the committed transaction list in the order in which it was committed, registers the transaction ID in the hash table P, and registers the table information involved in the transaction and the transaction's starting LSN in the hash table G;

[0010] Determine whether the current transaction is a rollback transaction. If so, wait until all conflicting transactions in the conflicting transaction list of the transaction are committed before waking up the worker thread waiting for the current transaction, then remove the registration information of the current transaction from hash tables P and G, and complete the execution of the current transaction; otherwise, perform the operation extraction step;

[0011] Operation extraction step: The working thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it processes the operation according to whether there is an operation rollback mark in the transaction. Otherwise, the operation is determined to be a DML operation. The working thread uses the conflict TRXID of the current operation to be executed to search the hash table P for a transaction with the conflicting TRXID. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, the working thread uses the information of the current operation table to search the hash table G for a transaction operating on the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, it determines that no conflict is detected and executes the current operation.

[0012] In some embodiments, when the target-side data synchronization service starts, a hash table R is initialized, and the record address in the hash item of the hash table R is initialized to empty, which is used to store and locate the received operations; the target-side data synchronization service creates two files to store the KEY in the hash table R, and the file name of each file is named incrementally in the order of creation, and the two files are added to the cache file list.

[0013] In some embodiments, obtaining the conflicting TRXID and the conflicting LSN includes:

[0014] After receiving the DML operation from the source, the target-side data synchronization service extracts the table ID and its primary key, index, or custom selected column in the operation as the key, then extracts the transaction ID in the operation as the TRXID, combines the key and TRXID into a record and saves it to the current file. If the record is the first record in the current file, the log LSN corresponding to the operation is marked on the file as the starting LSN of the file.

[0015] Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict.

[0016] The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first file corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

[0017] In some embodiments, the conflict detection determination includes:

[0018] Calculate the hash value of the KEY of the current detection record X and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, backfill the hash item with the address of the detection record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection and judgment;

[0019] Determine whether the file ID in the address recorded in the hash item is less than the ID value of the first file in the cache file list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and backfill the hash item with the address of the detection record X. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment;

[0020] According to the address recorded in the hash item, the corresponding record Y is loaded from the file. It is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation. Record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item. The record address in the hash item is set to the next address space of the current detection record X, the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflicting TRXID for return; if not, the next step of conflict detection is carried out;

[0021] Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record list.

[0022] In some embodiments, the cache file linked list includes two files. When the first file is full, it switches to the second file. When the second file is full, the first file in the linked list is eliminated first, and then a new file is created with an incremented file ID and added to the cache file linked list.

[0023] In some embodiments, when the target-side data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation; specifically:

[0024] Locate the transaction that needs to be rolled back using the transaction ID of the rollback or partial rollback operation, extract the number of rows included in the transaction in the rollback operation, or the number of rows that need to be rolled back in the partial rollback operation; and mark the corresponding transaction with a rollback flag.

[0025] Create a conflicting transaction list to collect conflicting TRXIDs saved in rollback operations. Extract the rolled-back operations from the end of the transaction operation list, extract the conflicting TRXIDs saved in the operations, and add them to the conflicting transaction list. Continue extracting the previous operation to be rolled back and collecting the same conflicting TRXIDs until the specified number of rows is rolled back.

[0026] In some embodiments, the processing based on whether a transaction has an operation rollback mark specifically includes:

[0027] If there is no rollback mark, the current transaction is committed and the waiting worker thread is woken up. The registration information of the current transaction is then removed from the hash table P and the hash table G to complete the execution of the current transaction.

[0028] If there is a rollback mark, it will wait until all conflicting transactions on the conflicting transaction list of this transaction are committed before waking up its own waiting worker thread, and then remove the registration information of the current transaction from hash table P and hash table G to complete the execution of the current transaction.

[0029] In a second aspect, the present invention further provides a device for parallel execution of operations based on a conflicting transaction list, the device comprising:

[0030] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for parallel execution of operations based on a conflicting transaction linked list as described in the first aspect.

[0031] In a third aspect, the present invention further provides an operation parallel execution system based on a conflicting transaction linked list, applying the operation parallel execution method based on a conflicting transaction linked list as described in the first aspect, wherein the system includes a hash table and worker thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module, wherein:

[0032] The hash table and worker thread creation module is used to create a hash table P, a hash table G and several worker threads through the target end data synchronization service;

[0033] The transaction extraction module is used to extract a transaction from the committed transaction list in the transaction submission order through a working thread, and register the transaction ID into the hash table P, and register the table information involved in the transaction and the starting LSN of the transaction into the hash table G;

[0034] The rollback judgment module is used to determine whether the current transaction is a rollback transaction. If so, it will wait until all conflicting transactions in the conflicting transaction list of the transaction are committed before waking up the worker thread waiting for the current transaction, and then remove the registration information of the current transaction from the hash table P and the hash table G to complete the execution of the current transaction; otherwise, it will call the operation extraction and conflict detection module;

[0035] The operation extraction and conflict detection module is used to extract an operation of a transaction through a working thread, determine whether the current operation is a commit operation, and if so, process it according to whether the transaction has an operation rollback mark; otherwise, the operation is determined to be a DML operation, and the working thread uses the conflict TRXID of the current operation to be executed to search the hash table P for a transaction with the conflicting TRXID. If so, the working thread waits for the conflicting transaction to be executed and committed before waking itself up; otherwise, the working thread uses the information of the current operation table to be executed to search the hash table G for a transaction operating the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation. If so, the working thread waits for the conflicting transaction to be executed and committed before waking itself up; otherwise, it is determined that no conflict is detected and the current operation is executed.

[0036] In a fourth aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the parallel execution method of operations based on the conflicting transaction linked list described in the first aspect.

[0037] Compared with the prior art, the present invention provides a method, device, and system for executing operations in parallel based on a conflicting transaction list, which have the following beneficial effects:

[0038] First, the target-side data synchronization service receives operations sent from the source side in sequence. If there are conflicting operations, the operations received first will be executed first, and the operations received later will be executed later. This order is also the basis for conflicting operation detection. The currently received operation can be compared with the previously received operation. If there are transaction operations with the same table and the same KEY, then it means that the current operation and a previous transaction are in conflict. When comparing the current operation with the previously received operation, a range must be set. The larger the range, the higher the accuracy, but at the same time, the more resources are consumed. This requires determining the size of the detection range based on the current hardware conditions. The embodiment of the present invention performs conflict detection on the currently received operation and the previously received operation within a set limited range. If the KEYs involved in the two operations are the same, it indicates that there is a conflicting transaction within the limited range for the current operation. When executing this operation, it needs to wait for the transaction of the conflicting operation to be executed first.

[0039] Secondly, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to mark the corresponding transaction to indicate that an operation on the transaction has been rolled back. If the transaction is a rollback transaction, it must also be added to the queue for execution and cannot be discarded directly. Otherwise, subsequent transactions that depend on it will be executed prematurely because the dependent transaction cannot be found.

[0040] Finally, when the target-side data synchronization service receives a rollback or partial rollback operation, it not only retains the transaction information of the rolled-back transaction, but also collects the conflicting TRXIDs that the rolled-back operation depends on. This helps to clearly identify the conflicting transactions that the rolled-back transaction depends on when it is released. It only needs to wait for the conflicting transaction to complete before releasing itself and then wake up and wait for its own transaction to continue executing, without having to wait for the non-conflicting transaction to commit before releasing itself, thereby improving synchronization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0042] Figure 1 A flowchart of a method for parallel execution of operations based on a conflicting transaction linked list provided in Example 1 of the present invention;

[0043] Figure 2 A flowchart of obtaining conflicting TRXIDs and conflicting LSNs provided in Example 1 of the present invention;

[0044] Figure 3 This is a flowchart of conflict detection and judgment provided in Example 1 of the present invention;

[0045] Figure 4 This is a flowchart of processing based on whether a transaction has an operation rollback mark provided in Example 1 of the present invention;

[0046] Figure 5 A schematic diagram of a module of a system for parallel execution of operations based on a conflicting transaction linked list provided in Example 1 of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of a device for parallel execution of operations based on a conflicting transaction list provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention. These all fall within the scope of protection of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other and are all within the scope of protection of this application. In addition, although the functional modules may be divided in the device schematic diagram and the logical order may be shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow chart.

[0049] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this invention belongs. The terms used in this specification and in the description of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. Furthermore, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0050] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as meaning open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that the specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0051] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate this application and are not intended to limit this application. The present invention will be described in detail below with reference to the accompanying drawings and examples.

[0052] Example 1:

[0053] like Figure 1 As shown, an embodiment of the present invention provides a method for parallel execution of operations based on a conflicting transaction linked list, and the method includes the following steps.

[0054] Step 100: The target-side data synchronization service creates hash tables P and G, as well as several worker threads. In this embodiment, LSN stands for log sequence number; TRXID stands for the ID that identifies a transaction; DML stands for insert, update, and delete operations; and KEY stands for a key that can be used to identify a specific row in a table.

[0055] Step 200: The worker thread extracts a transaction from the committed transaction list in the order in which the transactions are committed, registers the transaction ID into the hash table P, and registers the table information involved in the transaction and the transaction start LSN into the hash table G.

[0056] Step 300: Determine whether the current transaction is a rollback transaction. If so, wait until all conflicting transactions in the conflicting transaction list of the transaction are committed before waking up the working thread waiting for the current transaction, then remove the registration information of the current transaction from hash table P and hash table G, and complete the execution of the current transaction; otherwise, perform the operation extraction step.

[0057] Step 400 (operation extraction step): The working thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it processes the operation according to whether there is an operation rollback mark in the transaction. Otherwise, it determines that the operation is a DML operation. The working thread uses the conflict TRXID of the current operation to be executed to search the hash table P for a transaction with the conflicting TRXID. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, the working thread uses the information of the current operation table to be executed to search the hash table G for a transaction operating on the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, it determines that no conflict is detected and executes the current operation.

[0058] When receiving a transaction operation, the embodiment of the present invention extracts the table ID of the operation, combines the primary key, index, or custom selected column of the operation as the KEY, and searches for the same KEY in other transaction operations within the set range received previously to determine conflicts. Conflicting operations need to wait until the conflicting transaction is submitted before they can be executed; non-conflicting transactions can be executed in parallel with transactions submitted before receiving the current operation, thereby maximizing storage performance.

[0059] This embodiment will now explain the above method in more detail.

[0060] In one embodiment, a synchronization system needs to be deployed on the source database and the target database. The source database synchronization system reads logs from the source database and parses them into operations, while the target database synchronization system is responsible for classifying and collecting synchronization operations sent from the source by transaction and setting a dependency relationship between each received operation and the conflicting transactions received before it.

[0061] In one embodiment, after the target-side data synchronization service is started, a hash table R of size N needs to be initialized, and the record address in the hash item is initialized to empty to store and locate the received operations; and two files of size M are created to store the KEY in the hash table R. The file names are named incrementally in the order of creation, and the two files are added to the cache file list.

[0062] For hash table R, the structure of the HASH item in its HASH table is a 4-byte file ID, set as K; a 4-byte file offset, set as F. K and F constitute a HASH item in the HASH table, as shown in the following table:

[0063] Document ID(K) File offset (F)

[0064] It indicates that the first KEY in the HASH item is stored in the file with ID K, and its offset in the file is F.

[0065] refer to Figure 2 As shown, in one embodiment, obtaining the conflicting TRXID and the conflicting LSN includes the following steps.

[0066] Step 101: After receiving the DML operation from the source, the target-side data synchronization service extracts the table ID and its primary key, index, or custom selected column in the operation as the KEY, and then extracts the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved at offset F of file K. If the record is the first record of the current file, the log LSN corresponding to the operation is marked on the file as the starting LSN of the file, and the LSN is saved in the file name. The complete file name format is "File ID_Start LSN.dat".

[0067] Since the hash record needs to save the address of the next record, the record saved in the file also needs to reserve the address of the next record, as shown in the table below.

[0068] KEY TRXID Next record file ID (K) Next record file offset (F)

[0069] Step 102: Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict.

[0070] Step 103: Set the TRXID obtained by conflict detection as the conflict TRXID, and write the starting LSN of the first file corresponding to the current hash table R into the current operation and set it as the conflict LSN, thereby obtaining the conflict information of this operation.

[0071] It should be noted that when this operation is executed, it searches for transactions involving the same table ID in all transactions whose committed operations are currently being stored and whose LSN is smaller than the LSN of this operation log, and whose operation LSN is currently being stored and whose LSN is smaller than the conflict LSN of this operation (indicating that operations in this transaction with a smaller LSN than the conflict LSN are not within the scope of the conflict check, and it cannot be determined whether there is a conflict). If so, this operation needs to wait for the completion of the transaction before it can be executed; if not, it continues to search for transactions in this batch with transaction IDs equal to the conflict TRXID. If so, this operation needs to wait for the completion of the transaction before it can be executed; if not, the operation can be immediately stored.

[0072] Based on the above steps, the target-side data synchronization service receives the next message and saves it to file K according to step 101. The cache file linked list includes two files. When the first file is full (the written data exceeds the set size M), it switches to the second file. When the second file is full, the first file in the linked list is eliminated first. Then, a new file is created by incrementing the file ID in the system, added to the linked list, and saved to the new file. When a record is saved to the new file, if it is the first record in the file, the operation LSN corresponding to the record is marked as the starting LSN of the file.

[0073] refer to Figure 3 As shown, in one embodiment, the conflict detection and judgment in the above step 102 specifically includes the following steps.

[0074] Step 201: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current record X, backfill the hash item with the address of the record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection and judgment.

[0075] Step 202: Determine the file ID K in the address recorded in the hash item and whether the file ID is less than the ID value of the first file in the cache file list. If so, it indicates that this record Y has exceeded the inspection range. The address recorded in the hash item is set to the next address space of the current record X, and the address of the record X is backfilled into the hash item. 0 is returned as the conflict TRXID to indicate that no conflict is detected. If not, proceed to the next step of conflict detection.

[0076] Step 203: Load the corresponding record Y from the file according to the address recorded in the hash item, and determine whether the KEY of record Y is equal to the KEY of the detection record X and the TRXID is not equal to the TRXID of the current operation. If so, it indicates that there is an operation conflict. Remove record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, set the record address in the hash item to the next address space of the current record X, fill the address of record X back into the hash item, and use the TRXID stored in record Y as the conflicting TRXID for return; if not, proceed to the next step of conflict detection and judgment.

[0077] Step 204: traverse the next record of record Y in sequence and perform conflict detection and judgment until the record linked list ends.

[0078] In one embodiment, when the target-side data synchronization service receives a commit operation, it adds the transaction to a queue of pending transactions.

[0079] In one embodiment, when the target-side data synchronization service receives a rollback or partial rollback operation, it is necessary to collect the conflicting TRXIDs stored in the rolled-back operation. If it is a rollback operation, the transaction cannot be discarded and needs to be added to the queue of pending transactions. Specifically, collecting the conflicting TRXIDs stored in the rolled-back operation includes: locating the transaction that needs to be rolled back by the transaction ID of the rollback or partial rollback operation, extracting the number of rows included in the transaction in the rollback operation or the number of rows that need to be rolled back in the partial rollback operation; at the same time, marking the corresponding transaction with an operation rollback mark; creating a conflicting transaction linked list L to collect the conflicting TRXIDs stored in the rollback operation; extracting the rolled-back operation from the end of the transaction operation linked list, extracting the conflicting TRXIDs stored in the operation, and adding them to the conflicting transaction linked list L; continuing to extract the previous operation to be rolled back and perform the same conflicting TRXID collection until the specified number of rows is rolled back.

[0080] It should be noted that when a received transaction is rolled back, since the rolled back transaction operations will not be executed when the data is stored, there may be subsequent transactions that depend on these rolled back operations. If the rolled back transaction is directly discarded, the dependent transactions will not be able to find the dependent transactions and will assume that there is no conflict. This will cause the subsequent transactions to execute prematurely, causing errors. For example, if three transactions A, B, and C all modify the same row in the same table, but transaction B is rolled back, when the target-side data synchronization service receives the operation, the conflict detection process will form a dependency order relationship in which B depends on A, and C depends on B. If transaction B is directly discarded and not executed due to the rollback, then when C is executed, it will not be able to find the dependent transaction B and will execute prematurely. Ultimately, transactions A and C will execute concurrently, causing errors.

[0081] It should also be noted that when processing rolled-back transactions and partial rollback operations, if only the rolled-back transaction is retained without collecting the conflicting TRXIDs and conflicting LSNs that the rollback operation depends on, then the transaction cannot determine the specific transactions it depends on during execution. It can only be released after all transactions with commit LSNs smaller than the transaction have completed synchronization, which will affect synchronization performance. The embodiment of the present invention collects the conflicting TRXIDs that the rolled-back operation depends on. During execution, the transaction only needs to detect that the transactions corresponding to these conflicting TRXIDs have completed synchronization before it can be released, without waiting for the submission of other non-conflicting transactions, thereby improving synchronization performance.

[0082] In one embodiment, after a system failure and restart, files are initialized into the target-side data synchronization service's cache file list based on their file ID sizes. It should be noted that when initializing the cache file list, the starting LSN of each file must be recovered from the file name. The stored records are then read sequentially and inserted into hash table R. Only after both files are loaded can new operations be accepted and conflict checks performed on them.

[0083] In one embodiment, the target-side data synchronization service creates H worker threads, executes transactions in order of their commit LSNs, and stores the transaction operations. It should be noted that the target-side data synchronization service also creates a hash table P to record the currently executing transaction information (transaction ID) and a hash table G to record the current transaction's table information and the transaction's start LSN.

[0084] In one embodiment, the worker thread extracts a transaction from the committed transaction list in the order in which the transactions were committed, registers the transaction ID in the hash table P, and registers the table information involved in the transaction and the transaction start LSN in the hash table G.

[0085] In one embodiment, it is determined whether the current transaction is a rollback transaction. If so, it is necessary to wait for all conflicting transactions in the conflicting transaction list L of the current transaction to commit before waking up the waiting worker thread. The current transaction's registration information is then removed from hash tables P and G to complete the execution of the current transaction. Otherwise, the next step of operation extraction is performed. It should be noted that in this step, a conflicting TRXID is sequentially extracted from the conflicting transaction list L of the current transaction and checked in hash table P to see if it exists. If it does, it is necessary to wait for the transaction to complete before waking up itself. After waking up, the next conflicting TRXID is extracted from the conflicting transaction list L and the same check is performed until all conflicting transactions in the conflicting transaction list L are completed. It should also be noted that since the operations in the rolled back transaction do not need to be executed and are discarded, the embodiment of the present invention collects the conflicting TRXIDs of the operations when discarding the rollback operations. Therefore, the transaction information on which they depend is clear. Therefore, when releasing the operations, it only needs to wait for the conflicting transactions to commit to ensure the correctness of the transaction dependency order.

[0086] In one embodiment, the working thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it processes the operation based on whether there is an operation rollback marker in the transaction; otherwise, the operation is a DML operation and proceeds to the next step.

[0087] In one embodiment, the worker thread uses the conflicting TRXID of the currently pending operation to search hash table P for a transaction with the conflicting TRXID. If so, the worker thread suspends the current execution process and waits for the conflicting transaction to complete and commit before waking up again. Otherwise, the worker thread jumps to the next step. It should be noted that the presence of a transaction with the conflicting TRXID in hash table P indicates that the data involved in the current operation conflicts with the transaction with the conflicting TRXID. Therefore, the current operation cannot be executed in parallel with the conflicting transaction and must wait until the conflicting transaction commits before it can begin execution.

[0088] In one embodiment, the worker thread uses the information of the current pending operation table to search the hash table G for transactions that operate on the same table, and whose commit LSN is smaller than the current transaction, and whose start LSN is smaller than the conflict LSN of the current operation. If so, the worker thread suspends the current execution process and waits for the conflicting transaction to be executed and committed before waking up itself. Otherwise, if the conflict detection of the worker thread's current pending operation does not detect a conflict, the worker thread executes the current operation and then continues with the detection and judgment of the next operation. It should be noted that this step mainly checks whether there are transactions in the current concurrently executed transactions that exceed the conflict check LSN range, because the conflict transaction information on the pending operation has a valid interval range, that is, the conflict transaction information is only valid within the interval from the conflict LSN saved in the operation to the current operation LSN. The operation does not perform conflict detection on the operations before the conflict LSN. That is to say, when there is a transaction whose commit LSN is smaller than the commit LSN of the transaction to which the currently pending operation belongs, and whose start LSN is smaller than the conflict LSN saved by the currently pending operation, it is unknown whether the operation on this transaction conflicts with the currently pending operation. In this case, the currently pending operation cannot be executed rashly and must wait until the transaction is committed before it can be executed.

[0089] refer to Figure 4 As shown, in one embodiment, processing based on whether a transaction has an operation rollback mark specifically includes the following steps.

[0090] Step 301: If there is no rollback flag, the current transaction is committed and the waiting worker thread is awakened. The current transaction's registration information is then removed from hash tables P and G, completing the execution of the current transaction. It should be noted that since the transactions that each operation in the transaction depends on have been determined during conflict detection, the transaction can be committed directly after executing all operations (indicating that all dependent transactions have been committed), without having to wait for transactions that submitted earlier than it, thereby effectively improving synchronization performance.

[0091] Step 302: If a rollback marker exists, the worker thread waiting for itself must wait until all conflicting transactions in the conflicting transaction list L of the current transaction are committed before waking up. The worker thread then removes the registration information of the current transaction from hash table P and hash table G to complete the execution of the current transaction. Specifically, each conflicting TRXID is extracted from the conflicting transaction list L of the current transaction and checked in hash table P to see if it exists. If it does, the worker thread must wait for the transaction to complete before waking up. After waking up, the worker thread then extracts the next conflicting TRXID from the conflicting transaction list L and performs the same check until all conflicting transactions in the conflicting transaction list L are completed. It should be noted that since the operations on the rolled back transaction do not need to be executed and are discarded, the embodiment of the present invention collects the conflicting TRXIDs of the operations when discarding the rollback operations. Therefore, the transaction information on which they depend is clear. Therefore, when releasing the operations, the worker thread only needs to wait for the conflicting transactions to commit to ensure the correctness of the transaction dependency order.

[0092] Based on the above process, this embodiment proposes a method for parallel execution of operations based on a conflicting transaction linked list. The method mainly performs conflict detection on the currently received operation and the previously received operation within a set limited range. If the table ID and the primary key or index or the key composed of the custom selected columns involved in the two operations are the same, it indicates that the current operation has a conflicting transaction within the limited range. When executing this operation, it must wait until the conflicting transaction is executed first.

[0093] refer to Figure 5As shown, this embodiment also provides an operation parallel execution system based on a conflicting transaction linked list, which applies the aforementioned operation parallel execution method based on a conflicting transaction linked list. The system includes a hash table and worker thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module, wherein: the hash table and worker thread creation module is used to create a hash table P, a hash table G, and several worker threads through the target end data synchronization service; the transaction extraction module is used to extract a transaction from the committed transaction linked list according to the transaction submission order through the worker thread, and register the transaction ID to the hash table P, and register the table information involved in the transaction and the starting LSN of the transaction to the hash table G; the rollback judgment module is used to determine whether the current transaction is a rollback transaction. If so, it will wait until all conflicting transactions on the conflicting transaction linked list of the transaction are committed before waking up the worker thread waiting for the current transaction, and then remove the transaction from the hash table P and the hash table G. In addition to the registration information of the current transaction, the execution of the current transaction is completed; otherwise, the operation extraction and conflict detection module is called; the operation extraction and conflict detection module is used to extract an operation of the transaction through the working thread, determine whether the current operation is a commit operation, and if so, process it according to whether the transaction has an operation rollback mark; otherwise, if the operation is determined to be a DML operation, the working thread uses the conflict TRXID of the current operation to be executed to search the hash table P for a transaction with the conflicting TRXID. If so, it waits for the conflicting transaction to be executed and committed before waking itself up; otherwise, the working thread uses the information of the current operation table to be executed to search the hash table G for a transaction operating on the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation. If so, it waits for the conflicting transaction to be executed and committed before waking itself up; otherwise, it determines that no conflict is detected and executes the current operation. The specific functions and principles of this system function module correspond to the specific methods in the method part and will not be repeated here.

[0094] In summary, this embodiment provides a method and system for parallel execution of operations based on a conflicting transaction list, which has the following advantages: First, the target-side data synchronization service receives operations sent by the source in sequence. If there are conflicting operations, the first received operation is executed first, followed by the later received operation. This order is the basis for conflict detection. The currently received operation can be compared with the previously received operation. If there are transactions with the same table and the same key, then the current operation conflicts with a previous transaction. When comparing the current operation with the previously received operation, a range must be set. A larger range increases accuracy, but also consumes more resources. The size of the detection range must be determined based on current hardware conditions. This embodiment of the present invention performs conflict detection on the currently received operation and the previously received operation within a limited range. If the keys involved in the two operations are the same, it indicates that the current operation has a conflicting transaction within the limited range. The current operation must wait until the conflicting transaction executes before executing.

[0095] Secondly, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to mark the corresponding transaction to indicate that an operation on the transaction has been rolled back. If the transaction is a rollback transaction, it must also be added to the queue for execution and cannot be discarded directly. Otherwise, subsequent transactions that depend on it will be executed prematurely because the dependent transaction cannot be found.

[0096] Finally, when the target-side data synchronization service receives a rollback or partial rollback operation, it not only retains the transaction information of the rolled-back transaction, but also collects the conflicting TRXIDs that the rolled-back operation depends on. This helps to clearly identify the conflicting transactions that the rolled-back transaction depends on when it is released. It only needs to wait for the conflicting transaction to complete before releasing itself and then wake up and wait for its own transaction to continue executing, without having to wait for the non-conflicting transaction to commit before releasing itself, thereby improving synchronization performance.

[0097] Example 2:

[0098] Based on the method for parallel execution of operations based on the conflicting transaction linked list provided in the above-mentioned embodiment 1, embodiment 2 of the present invention provides a specific example for illustration.

[0099] For example: The source database has table T (ID INT PRIMARY KEY, C1 INT);

[0100] The source application has three transactions that perform the following operations on table T in the following order:

[0101] TRX1: INSERT INTO T(ID, C1)VALUES(1, 1);

[0102] TRX1:COMMIT;

[0103] TRX2: INSERT INTO T(ID, C1)VALUES(2, 2);

[0104] TRX2: UPDATE T SET C1=1WHERE ID=1;

[0105] TRX2:ROLLBACK;

[0106] TRX3: UPDATE T SET C1=2WHERE ID=1;

[0107] TRX3:COMMIT;

[0108] The order of the above operations represents the order of these operations in the log stream, and the resulting log stream will form the following numbered table:

[0109]

[0110]

[0111] After the target-side data synchronization service is started, it needs to initialize a hash table R of size N and two files of size M. The files are numbered 1 and 2 in sequence.

[0112] The receiving process is as follows:

[0113] 1. The log receiving thread receives the operation with LSN 1, extracts its table ID and primary key as the key, assuming it is 1. The length after combining the TRXID and the space for the next record is 32. First, its key and transaction ID are stored in the first file, expressed as [file ID (K): file offset (F)]. Its offset in the file is [1:0].

[0114] KEY: 1TRXID: 1NEXT: empty

[0115] Since the KEY information is the first operation information of the current file, the LSN value of the operation should be set to the starting LSN of the file, and the starting LSN of the file should be set to 1.

[0116] 2. Insert the KEY into the hash table R. The returned TRXID is 0, indicating no conflict. The TRXID 0 indicating no conflict and the minimum starting LSN of the file is 1, which is written back to the operation.

[0117] 3. The log receiving thread receives the operation with LSN 3, extracts its table ID and primary key as the key, assuming it is 2. The length after combining the TRXID and the space for the next record is 32. First, its key and transaction ID are stored in the first cache file. Its offset in the file is [1:32].

[0118] KEY: 1TRXID: 1NEXT: empty KEY: 2TRXID: 2NEXT: empty

[0119] 4. Insert the KEY into the hash table R. The returned TRXID is 0, indicating no conflict. The TRXID 0 indicating no conflict and the minimum starting LSN of the cached file is 1, which is written back to the operation.

[0120] 5. The log receiving thread receives the operation with LSN 4, extracts its table ID and primary key as the KEY, assuming it is 1, and combines the TRXID and the space for the next record to make the length 32. First, store its KEY and transaction ID in the first cache file. Its offset in the file is [1:64].

[0121]

[0122] 6. Insert the KEY into the hash table R. Since the operation information with KEY 1 already exists in the HASH, the conflicting TRXID returned is 1, indicating that the current operation conflicts with the transaction with transaction ID 1. The conflicting TRXID 1 and the minimum starting LSN of the file 1 are written back to the operation.

[0123] 7. Remove the conflicting KEY information with KEY 1 from the HASH table.

[0124] 8. Set the address in the hash table R with KEY 1 to the next record of the current KEY information to obtain the following table.

[0125] KEY: 1TRXID: 1NEXT: empty KEY: 2TRXID: 2NEXT: empty KEY:1TRXID:2NEXT:1:0

[0126] 9. Restart the target data synchronization service and load cache file 1 and cache file 2 in sequence.

[0127] 10. Load and merge the KEY information in cache file 1 into hash table R to restore the HASH state before the failure.

[0128] 11. The log receiving thread receives the rollback operation with LSN 5, marks the corresponding TRX2, indicating that a rollback operation has occurred in the transaction, and then adds TRX2 to the execution queue. During the rollback operation, TRX2 creates a conflicting transaction list L, collects the conflicting TRXID 1 information for the transaction update operation, and adds it to the conflicting transaction list L.

[0129] 12. The log receiving thread receives the operation with LSN 6, extracts its table ID and primary key as the KEY, assuming it is 3. The length after combining the TRXID and the space for the next record is 32. First, its KEY and transaction ID are stored in the first file. Its offset in the file is [1:96].

[0130] KEY: 1TRXID: 1NEXT: empty KEY: 2TRXID: 2NEXT: empty KEY:1TRXID:2NEXT:1:0 KEY:3TRXID:3NEXT:empty

[0131] 13. Insert the KEY into the hash table R. The returned TRXID is 2, indicating that there is a conflict between the current operation and the transaction with transaction ID 2. The conflicting TRXID 2 and the minimum starting LSN of the cache file 1 are written back to the operation.

[0132] After all operation conflicts are detected, the following detection results are formed.

[0133]

[0134]

[0135] 14. Create three worker threads and execute TRX1, TRX2, and TRX3 respectively in the order in which the transactions were submitted.

[0136] 15. Worker thread 1 extracts the TRX1 transaction, registers the transaction information TRXID as 1 in hash table P, and registers the start LSN of the transaction associated with table information T as 1 in hash table G.

[0137] 16. Worker thread 2 extracts the TRX2 transaction, registers the transaction information TRXID 2 into hash table P, and registers the starting LSN 3 of the transaction associated with table information T into hash table G.

[0138] 17. Worker thread 3 extracts the TRX3 transaction, registers the transaction information TRXID 3 into hash table P, and registers the starting LSN 6 of the transaction associated with table information T into hash table G.

[0139] 18. Worker thread 1 extracts the insert operation with LSN 1 on TRX1, obtains the conflicting TRXID and conflicting LSN on the operation, and performs conflict detection.

[0140] (1) This operation conflicts. A TRXID of 0 indicates no conflict.

[0141] (2) Obtain the table information T and the conflict LSN of the operation as 1. Search the hash table G for conflicting transactions involving table T and with a commit LSN less than 2 and a minimum transaction start LSN less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0142] 19. Worker thread 2 extracts TRX2. This transaction is marked as a rollback transaction, so it does not need to execute. However, it needs to wait for TRX1 (a transaction with a smaller LSN than its own) in the conflicting transaction list L to commit before removing its own transaction information from the registered hash tables P and G. It then wakes up the transaction waiting for its completion. Specifically, it extracts the conflicting transaction information TRX1 from the conflicting transaction list L, searches for it in the hash table P, suspends the current execution, and waits for the conflicting transaction TRX1 to complete before waking itself up.

[0143] 20. Work thread 3 extracts the update operation with LSN 6 on TRX3, obtains the conflicting TRXID and conflicting LSN on the operation, and performs conflict detection.

[0144] (1) The conflicting TRXID is 2, indicating that it conflicts with TRX2. The current working thread is suspended and the thread is awakened after TRX2 is finished executing.

[0145] As can be seen here, TRX3 depends on TRX2, but TRX2 itself is rolled back and will not perform any operations. If TRX2 is directly discarded and not executed, the update of TRX3 will not find the conflicting transaction and will be executed concurrently with the insert operation of TRX1 in advance. However, these two transactions operate on the same row. If the update operation of TRX3 is executed before the insert operation of TRX1, an error may be reported.

[0146] 21. After worker thread 1 executes the commit operation of TRX1, it deletes the current transaction information from hash table P and hash table G respectively, and then wakes up the execution of TRX2 transaction on worker thread 2.

[0147] 22. After transaction TRX2 of worker thread 2 is awakened, it continues to traverse the conflicting TRXID on the conflicting transaction list L. All conflicting transactions have been executed. Since this transaction is a rolled back transaction, it does not need to commit any operations. It only needs to delete the current transaction information from hash table P and hash table G and then wake up to wait for its own transaction TRX3 to be executed.

[0148] 23. After worker thread 3 is awakened, it executes the update operation of TRX3, and then executes the commit operation and deletes the current transaction information from hash table P and hash table G to complete the execution of the transaction.

[0149] As can be seen from the above process, since transaction TRX1 is received first, followed by the operations of transaction TRX2, TRX2 will discover during conflict detection that its operations depend on transaction TRX1. Although TRX2 ultimately receives a rollback operation, meaning that the transaction will not be executed, TRX2's operations have already overwritten the dependencies in the HASH table during conflict detection. As a result, the operations of the last received transaction, TRX3, depend on TRX2 rather than TRX1 during conflict detection. Therefore, TRX2 must be marked as having a rollback operation and added to the execution chain. This ensures that the cascading transaction dependency relationships (TRX3 depends on TRX2, and TRX2 depends on TRX1) are not interrupted. Although TRX2 is a rollback transaction, it must still be registered during execution to ensure dependency continuity. Since the transaction information of TRX2's dependencies is collected in the conflict transaction chain L, TRX2 only needs to ensure that all transactions in the conflict transaction chain L have completed execution before it can be released, without having to wait for transactions with a lower LSN than its own to commit, thereby improving synchronization performance.

[0150] Example 3:

[0151] It should be noted that for rollback transactions, since the rollback transaction will not be executed, there are three ways to handle the rollback transaction operation in the hash table R.

[0152] The first approach is to not roll back the impact of the rolled-back transaction's operations on the operation dependencies in hash table R, nor collect the conflicting TRXIDs of the rolled-back transaction's dependencies. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies. When this approach is adopted, the rolled-back transaction's operations are directly released as a whole, which can improve the processing performance of rolled-back transactions. However, operations that depend on the rolled-back transaction must wait until the rolled-back transaction is released from the pending execution list before they can be executed. Since conflicting transactions with operation dependencies were not collected before the rollback transaction was released, the rollback transaction must wait until all transactions with commit LSNs or rollback LSNs smaller than its own rollback LSN have completed execution before it can be released, which affects synchronization efficiency. This approach corresponds to the following: when the target-side data synchronization service receives a rolled-back transaction, all operations in the transaction are discarded without execution, and only the retained transaction information is added to the pending execution queue. For example, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to locate the transaction based on the transaction ID in the operation, and then mark the transaction to indicate that some operations in the transaction are rolled back. If the operation is a rollback operation, then all operations in the transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed. The transaction information of the rolled back transaction cannot be discarded.

[0153] The second approach does not roll back the impact of the rolled-back transaction on the dependencies in hash table R. Instead, the conflicting TRXIDs of the rolled-back transaction's dependencies are collected, for example, recorded in the conflicting TRXID dependency list L. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending list for execution to ensure the continuity of the dependencies. When this approach is adopted, the conflicting TRXIDs in the rolled-back transaction are collected before the operation is released. Although this reduces the processing performance of the rolled-back transaction, it improves the release performance of the rolled-back transaction during execution. Operations that depend on the rolled-back transaction must wait until the rolled-back transaction is released from the pending list before they can be executed. Since the conflicting transactions of the rolled-back transaction were collected during the release operation, the rolled-back transaction can be released only after the execution of these conflicting transactions, without having to wait for the completion of all transactions with a commit LSN or a rollback LSN smaller than the rollback LSN. This partially improves synchronization efficiency compared to the first approach. This approach corresponds to the following: when the target-side data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs stored in the rolled-back operation. Furthermore, the corresponding transaction is not discarded but added to the pending transaction queue. The description of the rollback portion in Example 1 of the present invention is an example of this approach.

[0154] The third method is to roll back the impact of the rolled-back transaction on the dependencies of the operations in hash table R. In this way, the rolled-back transaction will not be dependent on other operations, and the rolled-back transaction does not need to be added to the pending execution list. When this solution is adopted, after receiving the rollback operation, the transaction needs to roll back the impact of all operations in the transaction on the dependencies of the operations in hash table R. Because the transaction scale is uncontrollable, when a large transaction is rolled back, the rollback action will affect the progress of receiving the operation, thereby affecting the efficiency of synchronization. Corresponding to this method, when the target-side data synchronization service receives a rollback or partial rollback operation, it writes the conflicting TRXID saved in the rolled-back operation back to hash table R.

[0155] The three aforementioned transaction rollback solutions each have their advantages and disadvantages. The first solution is suitable for large-scale transaction rollbacks, the second for medium-scale transactions, and the third for small-scale transactions. Therefore, two parameters, W1 and W2, are set: W1 represents the minimum number of DML operations in large-scale transactions, and W2 represents the minimum number of DML operations in medium-scale transactions. Based on this, after receiving DML operations from the source, the target-side data synchronization service must manage these operations by transaction classification and calculate the number of DML operations received per transaction, S (S being the number of DML operations received in a single transaction). When the target-side data synchronization service receives a rollback operation, it takes appropriate action based on the number of DML operations in the transaction, S.

[0156] When the size S of the rolled-back transaction is greater than or equal to W1, it corresponds to the large-scale transaction rollback in the first solution. In this case, the impact of the rolled-back transaction's operations on the operation dependencies in hash table R is not rolled back, and the conflicting TRXIDs of the rolled-back transaction's operation dependencies are not collected. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations and must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies.

[0157] When the scale S of the rolled-back transaction is greater than or equal to W2 and less than W1, this corresponds to the medium-sized transaction rollback in the second solution. In this case, the impact of the rolled-back transaction's operations on the operation dependencies in hash table R is not rolled back. However, the conflicting TRXIDs of the rolled-back transaction's operation dependencies are collected and can be optionally recorded in the conflicting TRXID dependency list L. This rolled-back transaction will be dependent on other operations and must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies. The description of the rollback portion in Example 1 of the present invention is an example description when the scale S of the rolled-back transaction is greater than or equal to W2 and less than W1.

[0158] When the size S of the rolled-back transaction is smaller than W2, it corresponds to the small-scale transaction rollback in the third solution. In this case, the impact of the rolled-back transaction operation on the operation dependencies in the hash table R is rolled back, and the rolled-back transaction does not need to be added to the pending execution list and is directly released.

[0159] Furthermore, for partial rollback operations, due to their special nature, they inherently require traversing the rolled-back operations. Therefore, adding the process of collecting conflicting TRXIDs for these operations does not significantly reduce the performance of handling partial rollbacks. Therefore, when the target-side data synchronization service receives a partial rollback operation, it counts the total number of rolled-back rows S1 and, based on the size of S1, performs the following processing: When the number of rolled-back rows S1 is less than W2, the impact of the rollback operation on the operation dependencies in hash table R is rolled back, that is, the conflicting TRXIDs stored in the rolled-back operation are written back to hash table R. When the number of rolled-back rows S1 is greater than W2, the impact of the rolled-back transaction on the operation dependencies in hash table R is not rolled back, that is, the conflicting TRXIDs stored in the rolled-back operation are not written back to hash table R. However, the conflicting TRXIDs of the rolled-back transaction operation dependencies are collected and recorded in the conflicting TRXID dependency list L.

[0160] Based on the above settings, after the target-side data synchronization service creates a worker thread, the worker thread extracts a transaction from the committed transaction list in the order in which the transactions were committed, and determines whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to determine whether the rollback transaction has a conflicting TRXID list L.

[0161] If the rolled-back transaction does not conflict with the TRXID linked list L, it must wait for all transactions with a commit LSN lower than the current transaction to complete before waking up the waiting worker thread. It then removes the current transaction's registration information from hash tables P and G, completing the current transaction. This is because the operations on the rolled-back transaction are no longer needed and are discarded, so the information on the transactions they depend on is unclear. Therefore, when committing, it must wait until all transactions submitted earlier than the current transaction have completed their commit to ensure the correct order of transaction dependencies.

[0162] If the rolled back transaction has a conflicting TRXID linked list L, it is necessary to wait for all conflicting transactions on the conflicting transaction linked list L to commit before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and hash table G to complete the execution of the current transaction. Specifically, extract a conflicting TRXID from the current transaction conflicting transaction linked list L in turn, and check whether it exists in the hash table P. If it exists, it is necessary to wait for the transaction to be completed before waking itself up. After waking up, it extracts the next conflicting TRXID from the conflicting transaction linked list L and performs the same check until all conflicting transactions in the conflicting transaction linked list L are completed. This is because the operations on the rolled back transaction do not need to be executed and are discarded. When the rollback operation is discarded, the conflicting TRXIDs on the operations are collected, so the transaction information they depend on is very clear. Then, when releasing, it only needs to wait for these conflicting transactions to be committed to ensure the correctness of the transaction dependency order.

[0163] In summary, since the rollback transaction will not be executed, it will generate three schemes during conflict detection. Each scheme has its own advantages. It is necessary to set appropriate values ​​to give full play to the advantages of the three schemes in order to achieve the purpose of improving the overall synchronization performance. In addition, for the partial rollback operations of the committed transactions, the conflicting TRXIDs referenced by them are collected in a targeted manner during the rollback operation, which can effectively reduce the performance loss caused by the partial rollback operation writing back the conflict detection hash table R, so as to achieve the purpose of improving the overall synchronization performance. This embodiment processes the rollback transaction and partial rollback operations in a targeted manner according to the agreed operation scale, and finds the balance point between the cost of the operation rollback and the execution of the rollback transaction, thereby improving the overall synchronization performance.

[0164] In addition, in some embodiments, the KEY in the hash table R can be stored in a file or in a memory chip. Specifically, the difference between the two is that when using files for storage, although the storage performance is low and affects the efficiency of conflict detection, after a data synchronization failure, the state of the hash table R before the failure can be restored by storing the KEY value in the file. In this way, when performing conflict detection on the received operations after the failure, there is continuity with the failure, and the synchronization performance will not drop drastically due to the failure. When using memory chips for storage, the conflict detection performance will be greatly improved compared to the file storage solution, but the data of the previous conflict detection will be lost after the failure. When restoring the detection, it can only start from the currently received LSN, which will cause a discontinuity between the subsequent conflict detection and the failure. During synchronous execution, the synchronization performance will drop drastically at the failure location. When the storage performance is high (for example, using SSD fixed storage), it can meet the performance requirements of conflict detection, and the efficiency of operation reception can be balanced with synchronous execution. Then, the method of storing the KEY in the hash table R through files can effectively avoid the loss of synchronization performance caused by failures. When the storage performance affects the efficiency of conflict detection, the bottleneck of synchronization exists in the operation reception. At this time, the method of storing the KEY in the hash table R through a memory chip can effectively improve the reception performance of the operation, thereby improving the overall synchronization performance. Example 1 of the present application is an example of a method of storing the KEY in the hash table R through a file.

[0165] In some embodiments, you can choose to calculate the MD5 value as the operation key on either the source or target side. The source-side data synchronization service detects conflicts for each operation, allowing for a more accurate calculation of the conflicting MD5 value. However, if the target-side data synchronization service calculates the MD5 value based on the target database table's primary key information, errors may occur if the target database table's primary key structure or column order differs from that on the source side. For this reason, the MD5 value is typically calculated on the source side as the operation key. Upon receiving the DML operation from the source side, the target-side data synchronization service directly extracts the MD5 value calculated by the source side as the key.

[0166] In some embodiments, when calculating the KEY of an operation, since the order of columns in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the KEY is calculated according to the fixed order of the primary key or unique index combination columns. After the source-side data synchronization service captures a DML log operation, it obtains the table ID in the log and uses the table ID to obtain the primary key information and unique index of the table in the source-side database. After parsing the current DML log operation, the source-side data synchronization service determines whether the table has a primary key or unique index. If there is a primary key or unique index, the table ID and the column values ​​of the primary key or unique index are combined to calculate an MD5 value as the KEY for this operation. This value is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when a table has both a primary key and a unique index, the table ID and the primary key are preferably combined to calculate the MD5 value as the KEY; when a table has no primary key but has multiple unique indexes, the table ID and any unique index are combined to calculate the MD5 value. If there is no primary key or unique index, after parsing the current DML log operation, the source-side data synchronization service combines the table ID and the row's ROWID value or custom selected columns to calculate an MD5 value as the key for this operation. This value is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when calculating the key for a table without a primary key or unique constraint, if it is clear that all columns in the table have no duplicate values, the table's ROWID can be used directly to calculate the key, effectively reducing the cost of calculating the key. If the table has duplicate rows, custom selected columns are required to calculate the key. The key can be calculated based on the distribution of column values, which columns or all column values ​​are used to calculate the key. This allows the target-side synchronization service to use the key to prevent deadlocks caused by concurrent synchronization transactions operating on rows with duplicate values.

[0167] Example 4:

[0168] Based on the method for parallel execution of operations based on a conflicting transaction list provided in the above embodiment 1, the present invention further provides a device for parallel execution of operations based on a conflicting transaction list that can be used to implement the above method and system, such as Figure 6 FIG. 1 is a schematic diagram of the device architecture of an embodiment of the present invention. The device for parallel execution of operations based on a conflicting transaction list of this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0169] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0170] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the method for parallel execution of operations based on a conflicting transaction list in Example 1. The processor 21 executes the various functional applications and data processing of the device for parallel execution of operations based on a conflicting transaction list by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the method for parallel execution of operations based on a conflicting transaction list in Example 1.

[0171] The memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21, and such remote memory may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the method for executing operations in parallel based on the conflicting transaction list in the above embodiment 1 is executed. For example, the above described method is executed. Figures 1-4 The steps shown.

[0173] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0174] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0175] Through the description of the above embodiments, it can be clearly understood by those skilled in the art that each embodiment can be implemented by means of software plus a general hardware platform, or of course by hardware. It can be understood by those skilled in the art that all or part of the processes in the above embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above. For the sake of simplicity, they are not provided in detail. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for parallel execution of operations based on a conflicting transaction list, characterized in that: include: The target-side data synchronization service creates hash tables P and G and several worker threads. The worker thread extracts a transaction from the committed transaction list in the order in which it was committed, registers the transaction ID in the hash table P, and registers the table information involved in the transaction and the transaction's starting LSN in the hash table G; Determine whether the current transaction is a rollback transaction. If so, wait until all conflicting transactions in the conflicting transaction list of the transaction are committed before waking up the worker thread waiting for the current transaction, then remove the registration information of the current transaction from hash tables P and G, and complete the execution of the current transaction; otherwise, perform the operation extraction step; Operation extraction step: The worker thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it processes the operation based on whether there is an operation rollback marker in the transaction. Otherwise, the operation is determined to be a DML operation. The worker thread uses the conflicting TRXID of the currently executed operation to search the hash table P for transactions with conflicting TRXIDs. If so, the worker thread waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, the worker thread uses the information of the current pending operation table to search the hash table G to see if there is a transaction operating on the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflicting LSN of the current operation. If so, the worker thread waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, it is determined that no conflict is detected and the current operation is performed.

2. The method for parallel execution of operations based on a conflicting transaction list according to claim 1, characterized in that: When the target-side data synchronization service starts, it initializes a hash table R and initializes the record address in the hash item of the hash table R to empty, which is used to store and locate the received operations; the target-side data synchronization service creates two files to store the KEY in the hash table R, and the file name of each file is named incrementally in the order of creation, and the two files are added to the cache file list.

3. The method for parallel execution of operations based on a conflicting transaction list according to claim 2, characterized in that: The acquisition of the conflicting TRXID and the conflicting LSN includes: After receiving the DML operation from the source, the target-side data synchronization service extracts the table ID and its primary key, index, or custom selected column in the operation as the key, then extracts the transaction ID in the operation as the TRXID, combines the key and TRXID into a record and saves it to the current file. If the record is the first record in the current file, the log LSN corresponding to the operation is marked on the file as the starting LSN of the file. Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict. The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first file corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

4. The method for parallel execution of operations based on a conflicting transaction list according to claim 3, characterized in that: The conflict detection and judgment includes: Calculate the hash value of the KEY of the current detection record X and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, backfill the hash item with the address of the detection record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection and judgment; Determine whether the file ID in the address recorded in the hash item is less than the ID value of the first file in the cache file list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and backfill the hash item with the address of the detection record X. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment; According to the address recorded in the hash item, the corresponding record Y is loaded from the file. It is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation. Record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item. The record address in the hash item is set to the next address space of the current detection record X, the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflicting TRXID for return; if not, the next step of conflict detection is carried out; Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record list.

5. The method for parallel execution of operations based on a conflicting transaction list according to claim 2, characterized in that: The cache file linked list includes two files. When the first file is full, it switches to the second file. When the second file is full, the first file in the linked list is eliminated first, and then a new file is created with an incremented file ID and added to the cache file linked list.

6. The method for parallel execution of operations based on a conflicting transaction list according to any one of claims 1 to 5, characterized in that: When the target-side data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation; specifically: Locate the transaction that needs to be rolled back using the transaction ID of the rollback or partial rollback operation, extract the number of rows included in the transaction in the rollback operation, or the number of rows that need to be rolled back in the partial rollback operation; and mark the corresponding transaction with a rollback flag. Create a conflicting transaction list to collect conflicting TRXIDs saved in rollback operations. Extract the rolled-back operations from the end of the transaction operation list, extract the conflicting TRXIDs saved in the operations, and add them to the conflicting transaction list. Continue extracting the previous operation to be rolled back and collecting the same conflicting TRXIDs until the specified number of rows is rolled back.

7. The method for parallel execution of operations based on a conflicting transaction list according to any one of claims 1 to 5, characterized in that: The processing according to whether the transaction has an operation rollback mark specifically includes: If there is no rollback mark, the current transaction is committed and the waiting worker thread is woken up. The registration information of the current transaction is then removed from the hash table P and the hash table G to complete the execution of the current transaction. If there is a rollback mark, it will wait until all conflicting transactions on the conflicting transaction list of this transaction are committed before waking up its own waiting worker thread, and then remove the registration information of the current transaction from hash table P and hash table G to complete the execution of the current transaction.

8. A device for parallel execution of operations based on a conflicting transaction list, characterized in that: The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for parallel execution of operations based on a conflicting transaction linked list as described in any one of claims 1-7.

9. A system for parallel execution of operations based on a conflicting transaction list, applying the method for parallel execution of operations based on a conflicting transaction list according to any one of claims 1 to 7, characterized in that: The system includes a hash table and worker thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module, wherein: The hash table and worker thread creation module is used to create a hash table P, a hash table G and several worker threads through the target end data synchronization service; The transaction extraction module is used to extract a transaction from the committed transaction list in the transaction submission order through a working thread, and register the transaction ID into the hash table P, and register the table information involved in the transaction and the starting LSN of the transaction into the hash table G; The rollback judgment module is used to determine whether the current transaction is a rollback transaction. If so, it will wait until all conflicting transactions in the conflicting transaction list of the transaction are committed before waking up the worker thread waiting for the current transaction, and then remove the registration information of the current transaction from the hash table P and the hash table G to complete the execution of the current transaction; otherwise, it will call the operation extraction and conflict detection module; The operation extraction and conflict detection module is used to extract an operation of a transaction through a working thread, determine whether the current operation is a commit operation, and if so, process it according to whether the transaction has an operation rollback mark; otherwise, the operation is determined to be a DML operation, and the working thread uses the conflict TRXID of the current operation to be executed to search the hash table P for a transaction with the conflicting TRXID. If so, the working thread waits for the conflicting transaction to be executed and committed before waking itself up; otherwise, the working thread uses the information of the current operation table to be executed to search the hash table G for a transaction operating the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation. If so, the working thread waits for the conflicting transaction to be executed and committed before waking itself up; otherwise, it is determined that no conflict is detected and the current operation is executed.

10. A non-volatile computer storage medium, characterized in that The computer storage medium stores computer-executable instructions, which are executed by one or more processors to complete the method for parallel execution of operations based on a conflicting transaction linked list according to any one of claims 1 to 7.

Images