Transaction parallel execution method, device and system based on operation rollback mark

Abstract

The invention relates to a transaction parallel execution method, device and system based on an operation rollback mark. The method mainly comprises the following steps: extracting a transaction from a submitted transaction chain table according to a transaction submission sequence by a working thread, judging whether the current transaction is a rollback transaction or not, if so, waiting for all transactions of which the submission LSN is smaller than that of the current transaction to be submitted, and then waking up the working thread; the working thread uses the conflicting TRXID of the current to-be-executed operation to search whether a transaction of the conflicting TRXID exists in the hash table P, and if yes, waits for awakening; otherwise, the working thread uses the current to-be-executed operation table information to search whether a transaction for operating the same table exists in the hash table G or not, the submission LSN of the transaction is smaller than that of the current transaction, and the initial LSN of the transaction is smaller than the conflict LSN of the current operation, and if yes, wait for waking up; otherwise, judging that no conflict is detected, and executing the current operation. According to the method, the concurrency of operation execution can be effectively increased, and the purpose of improving the synchronization performance is achieved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a transaction parallel execution method, device and system based on operation rollback marking. Background Art

[0002] In the database real-time synchronization system (supporting heterogeneity) based on the log parsing architecture, the source-side data synchronization service is responsible for capturing the operation log of the source-side database, parsing the log and restoring the corresponding database operations, and then sending them to the target-side data synchronization service. The target-side data synchronization service is responsible for executing these operations in the target-side database, thereby completing the data synchronization process.

[0003] When the target-side data synchronization service synchronizes operations on the target-side database, it often needs to execute in the order of transaction submission. Because when the previous transaction is not synchronized, the operation of the next transaction cannot be executed concurrently with the previous transaction operation without clarifying whether it conflicts with the previous transaction. It can only wait until all transactions before the current operation are submitted before it can start to execute, which greatly affects the concurrency of transaction operation storage. The effective method currently used is to determine whether it can be executed in parallel by judging the size of the log LSN where the operation is located and the LSN of the previously submitted transaction submission operation. If the LSN of the currently executed transaction operation is smaller than the LSN of the submission operation of another transaction currently being executed, then this operation can be executed concurrently with the transaction. This strategy helps to improve concurrency, but the performance improvement is small.

[0004] In view of this, how to overcome the defects of the existing technology and how to improve the parallel execution of the target-side data synchronization service to improve the performance of data synchronization have become important technical issues that need to be urgently solved in the industry. Summary of the invention

[0005] In view of the defects or improvement needs in the prior art: how to improve the parallel execution strength of the target-side data synchronization service to improve the performance of data synchronization. In order to improve the performance of synchronization, the target-side data synchronization service sets its dependency information for each operation when receiving the operation, so that it can be executed concurrently with a finer granularity when executing, thereby improving performance. Based on this, the present invention provides a transaction parallel execution method, device and system based on operation rollback mark. When the target-side data synchronization service of the method receives the synchronization transaction operation, it receives it in the order of generation of the source-side database log. Then the order of each operation received is the basis for them to detect whether there is a conflict. The present invention extracts the primary key or index or custom selection column on each received operation as the KEY based on the order of operation reception, and finds whether this KEY has the same KEY as the operation of other transactions received before within the set range. If so, the execution of this operation needs to wait for the conflicting transaction to be submitted before it can start to execute; otherwise, it can be executed concurrently with the transaction submitted before itself, maximizing the parallelism of transaction execution, thereby effectively improving the performance of synchronization.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a transaction parallel execution method based on operation rollback markers, comprising:

[0008] The target-side data synchronization service creates hash table P, hash table G, and several worker threads;

[0009] The worker thread extracts a transaction from the committed transaction list in the order of transaction submission, registers the transaction ID into the hash table P, and registers the table information involved in the transaction and the transaction start LSN into the hash table G;

[0010] Determine whether the current transaction is a rollback transaction. If so, wait until all transactions with a commit LSN smaller than the current transaction are committed before waking up the worker thread waiting for the current transaction; otherwise, perform the operation extraction step;

[0011] Operation extraction step: The working thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it processes the operation according to whether there is an operation rollback mark in the transaction. Otherwise, it determines that the operation is a DML operation. The working thread uses the conflicting TRXID of the current operation to be executed to search for a transaction with the conflicting TRXID in the hash table P. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, the working thread uses the information of the current operation table to be executed to search for a transaction operating the same table in the hash table G, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, it determines that no conflict is detected and executes the current operation.

[0012] In some embodiments, when the target-side data synchronization service is started, a hash table R is initialized, and the record address in the hash item of the hash table R is initialized to empty, which is used to store and locate the received operations; the target-side data synchronization service creates two cache files to store the KEY in the hash table R, and the file name of each cache file is named incrementally in the order of creation, and the two cache files are added to the cache file linked list.

[0013] In some embodiments, obtaining the conflicting TRXID and the conflicting LSN includes:

[0014] After receiving the DML operation from the source, the target-side data synchronization service extracts the table ID and its primary key or index or custom selected column in the operation as the KEY, and then extracts the transaction ID in the operation as the TRXID, and combines the KEY and TRXID into a record and saves it to the current cache file. If the record is the first record of the current cache file, the log LSN corresponding to the operation is marked on the cache file as the starting LSN of the cache file;

[0015] Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection and judgment during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict;

[0016] The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first cache file corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

[0017] In some embodiments, the conflict detection determination includes:

[0018] Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item, and return 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next step of conflict detection and judgment;

[0019] Determine whether the file ID in the address recorded in the hash item is less than the ID value of the first file in the cache file list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment;

[0020] According to the address recorded in the hash item, load the corresponding record Y from the file, determine whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, remove record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, set the record address in the hash item to the next address space of the current detection record X, fill the address of the current detection record X back into the hash item, and use the TRXID stored in record Y as the conflicting TRXID to return; if not, proceed to the next step of conflict detection and judgment;

[0021] Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record linked list.

[0022] In some embodiments, the cache file linked list includes two cache files. When the first cache file is full, it switches to the second cache file. When the second cache file is full, the first cache file in the linked list is eliminated first, and then a new cache file is created with an incremented file ID and added to the cache file linked list.

[0023] In some embodiments, when the target-side data synchronization service receives a rollback or partial rollback operation, it locates the transaction based on the transaction ID in the operation, and then marks the transaction with an operation rollback mark, indicating that some operations in the transaction have been rolled back. If the operation is a rollback operation, then all operations in the transaction do not need to be executed and are directly discarded, and only the retained transaction information is added to the queue to be executed.

[0024] In some embodiments, the processing according to whether a transaction has an operation rollback mark specifically includes:

[0025] If there is no operation rollback mark, after committing the current transaction, wake up the waiting worker thread, remove the registration information of the current transaction from hash table P and hash table G, and complete the execution of the current transaction;

[0026] If there is a rollback mark, wait for all transactions with a committed LSN smaller than the current transaction to be committed before waking up the waiting worker thread, and then remove the registration information of the current transaction from hash tables P and G to complete the execution of the current transaction.

[0027] In a second aspect, the present invention further provides a transaction parallel execution device based on operation rollback markers, the device comprising:

[0028] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the transaction parallel execution method based on operation rollback marking as described in the first aspect.

[0029] In a third aspect, the present invention further provides a transaction parallel execution system based on operation rollback marks, which applies the transaction parallel execution method based on operation rollback marks as described in the first aspect, and the system includes a hash table and worker thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module, wherein:

[0030] The hash table and worker thread creation module is used to create a hash table P, a hash table G and several worker threads through the target end data synchronization service;

[0031] The transaction extraction module is used to extract a transaction from the committed transaction linked list according to the transaction submission order through a working thread, and register the transaction ID into a hash table P, and register the table information involved in the transaction and the start LSN of the transaction into a hash table G;

[0032] The rollback judgment module is used to judge whether the current transaction is a rollback transaction. If so, it will wait for all transactions with a commit LSN smaller than the current transaction to complete before waking up the working thread waiting for the current transaction; otherwise, it will call the operation extraction and conflict detection module;

[0033] The operation extraction and conflict detection module is used to extract an operation of a transaction through a working thread, determine whether the current operation is a commit operation, and if so, process it according to whether the transaction has an operation rollback mark; otherwise, the operation is determined to be a DML operation, and the working thread uses the conflict TRXID of the current operation to be executed to search whether there is a transaction with the conflict TRXID in the hash table P, and if so, wait for the conflicting transaction to be executed and committed before waking itself up; otherwise, the working thread uses the information of the current operation table to be executed to search in the hash table G whether there is a transaction operating the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation, and if so, wait for the conflicting transaction to be executed and committed before waking itself up; otherwise, it is determined that no conflict is detected and the current operation is executed.

[0034] In a fourth aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the transaction parallel execution method based on operation rollback markers described in the first aspect.

[0035] Compared with the prior art, the present invention provides a transaction parallel execution method, device and system based on operation rollback marking, which has the following beneficial effects:

[0036] First of all, the target-side data synchronization service receives operations sent from the source side in sequence. If there are conflicting operations, the operations received first will be executed first, and the operations received later will be executed later. This order is also the basis for conflicting operation detection. The currently received operation can be compared with the previously received operation. If there are transaction operations with the same table and the same KEY, it means that the current operation conflicts with a previous transaction. When comparing the current operation with the previously received operation, a range must be set. The larger the range, the higher the accuracy, but at the same time, the more resources are consumed. This requires determining the size of the detection range based on the current hardware conditions. The embodiment of the present invention performs conflict detection on the currently received operation and the previously received operation within a set limited range. If the KEYs involved in the two operations are the same, it indicates that there is a conflicting transaction within the limited range for the current operation. When executing this operation, it is necessary to wait for the transaction of the conflicting operation to be executed before it can be executed.

[0037] Secondly, using the file system to save the KEY information of the received operations can effectively ensure the continuity of the conflict detection of the received operations. After the target-side data synchronization service is restarted after a failure, the KEY information of the previously received operations can be restored. In this way, the conflict relationship of the newly received operations is continuous with that before the failure, ensuring that the failure will not affect the dependency relationship of the positioning operation, thereby ensuring that the parallelism of the operation execution will not be affected after the system failure recovery.

[0038] Finally, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to mark the corresponding transaction to indicate that an operation on the transaction has been rolled back. If the transaction is a rollback transaction, it must also be added to the queue for execution and cannot be discarded directly. Otherwise, subsequent transactions that depend on it will be executed early because the dependent transaction cannot be found. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of a transaction parallel execution method based on operation rollback marking provided in Example 1 of the present invention;

[0041] Figure 2 A flow chart of obtaining conflicting TRXIDs and conflicting LSNs provided in Example 1 of the present invention;

[0042] Figure 3 A flowchart of conflict detection and judgment provided in Example 1 of the present invention;

[0043] Figure 4 A flowchart of processing according to whether a transaction has an operation rollback mark provided in Embodiment 1 of the present invention;

[0044] Figure 5 A module diagram of a transaction parallel execution system based on operation rollback marking provided in Example 1 of the present invention;

[0045] Figure 6 A schematic diagram of the structure of a transaction parallel execution device based on operation rollback marking provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present invention, several variations and improvements can also be made. These all belong to the protection scope of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the protection scope of this application. In addition, although the functional module division may be performed in the device schematic diagram, and the logical order may be shown in the flow chart, in some cases, it may be different from the module division in the device, or the steps shown or described in the order in the flow chart may be performed.

[0047] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0049] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0050] Embodiment 1:

[0051] like Figure 1 As shown, an embodiment of the present invention provides a transaction parallel execution method based on operation rollback marking, and the method includes the following steps.

[0052] Step 100: The target end data synchronization service creates hash table P, hash table G and several worker threads. In this embodiment, LSN is the log sequence number; TRXID is the id that identifies a transaction; DML is the insert, update and delete operation of data; KEY is a key that can be used to identify a specific row of records in the table.

[0053] Step 200: The working thread extracts a transaction from the committed transaction linked list in the transaction commit order, and registers the transaction ID into the hash table P, and registers the table information involved in the transaction and the start LSN of the transaction into the hash table G.

[0054] Step 300: Determine whether the current transaction is a rollback transaction. If so, wait until all transactions with a commit LSN smaller than the current transaction are committed before waking up the working thread waiting for the current transaction; otherwise, perform the operation extraction step.

[0055] Step 400 (operation extraction step): the working thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it processes the operation according to whether there is an operation rollback mark in the transaction. Otherwise, it determines that the operation is a DML operation, and the working thread uses the conflicting TRXID of the current operation to be executed to search whether there is a transaction with the conflicting TRXID in the hash table P. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, the working thread uses the information of the current operation table to be executed to search for a transaction operating the same table in the hash table G, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, it is determined that no conflict is detected and the current operation is executed.

[0056] When receiving a transaction operation, the embodiment of the present invention extracts the table ID of the operation, combines the primary key or index of the operation or a custom selected column as the KEY, and searches for the same KEY in other transaction operations within the set range previously received to determine conflicts. Conflicting operations need to wait for the conflicting transaction to be submitted before they can be executed; non-conflicting transactions can be executed in parallel with transactions submitted before receiving the current operation, thereby maximizing storage performance.

[0057] This embodiment will now explain the above method in more detail.

[0058] In one embodiment, a synchronization system needs to be deployed on the source database and the target database. The source database synchronization system reads logs from the source database and parses them into operations, while the target database synchronization system is responsible for classifying and collecting the synchronization operations sent from the source by transaction, and setting a dependency relationship between each received operation and the conflicting transaction received before it.

[0059] In one embodiment, after the target-side data synchronization service is started, a hash table R of size N needs to be initialized, and the record address in the hash item is initialized to empty, which is used to store and locate the received operations; and two files of size M are created to store the KEY in the hash table R. The file names are named incrementally in the order of creation, and the two files are added to the cache file linked list.

[0060] For hash table R, the structure of the HASH item in its HASH table is a 4-byte file ID, set as K; a 4-byte file offset, set as F, K and F constitute a HASH item in the HASH table, as shown in the following table:

[0061] Document ID(K) File offset (F)

[0062] It means that the first KEY in the HASH item is stored in the file with ID K and its offset in the file is F.

[0063] refer to Figure 2 As shown, in one embodiment, obtaining the conflicting TRXID and the conflicting LSN includes the following steps.

[0064] Step 101: After receiving the DML operation from the source, the target-side data synchronization service extracts the table ID and its primary key or index or custom selected column in the operation as the KEY, and then extracts the transaction ID in the operation as the TRXID, and combines the KEY and TRXID into a record and saves it to the offset F of the cache file K. If the record is the first record of the current file, the log LSN corresponding to the operation is marked on the file as the starting LSN of the file, and the LSN is saved in the file name. The complete cache file name format is "File ID_Starting LSN.daT".

[0065] Since the hash record needs to save the address of the next record, the record saved in the file also needs to reserve the address of the next record, as shown in the following table.

[0066] KEY TRXID Next record file ID (K) Next record file offset (F)

[0067] Step 102: Use the KEY of the current operation to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection and judgment during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict.

[0068] Step 103: Set the TRXID obtained by conflict detection as the conflict TRXID, write the start LSN of the first cache file corresponding to the current hash table R into the current operation, and set it as the conflict LSN, thereby obtaining the conflict information of this operation.

[0069] It should be noted that when this operation is executed, it searches for transactions involving the same table ID in transactions whose committed operations LSN is smaller than the LSN of this operation log and whose operation LSN is smaller than the conflict LSN of this operation (indicating that operations in this transaction with a smaller LSN than the conflict LSN are not within the scope of the conflict check and it cannot be determined whether there is a conflict). If so, this operation needs to wait for the completion of the transaction before it can be executed. If not, it continues to search for transactions with transaction IDs equal to the conflict TRXID in this batch of transactions. If so, this operation needs to wait for the completion of the transaction before it can be executed. If not, this operation can be executed immediately.

[0070] Based on the above steps, the target end data synchronization service receives the next message and saves it to the cache file K in the manner of step 101. The cache file linked list includes two files. When the first cache file is full (the written data exceeds the set size M), it switches to the second cache file. When the second is full, the first cache file in the linked list is eliminated first, and then a new cache file is created by incrementing the file ID in the system, added to the linked list, and saved in the new file. When a record is saved to a new file, if it is the first record in the file, the operation LSN corresponding to the record should be marked as the starting LSN of the file.

[0071] refer to Figure 3 As shown, in one embodiment, the conflict detection and judgment in the above step 102 specifically includes the following steps.

[0072] Step 201: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current record X, and fill the address of the record X back into the hash item, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection and judgment.

[0073] Step 202: Determine the file ID K in the address recorded in the hash item, and determine whether the file ID is less than the ID value of the first file in the cache file list. If so, it indicates that the record Y has exceeded the inspection range. The address recorded in the hash item is set to the next address space of the current record X, and the address of the record X is backfilled into the hash item. 0 is returned as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next step of conflict detection and judgment.

[0074] Step 203: Load the corresponding record Y from the file according to the address recorded in the hash item, and determine whether the KEY of record Y is equal to the KEY of the detection record X and the TRXID is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation. Remove record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, set the record address in the hash item to the next address space of the current record X, fill the address of record X back into the hash item, and return the TRXID stored in record Y as the conflicting TRXID; if not, proceed to the next step of conflict detection and judgment.

[0075] Step 204: traverse the next record of record Y in sequence and perform conflict detection and judgment until the record linked list ends.

[0076] In one embodiment, when the target-side data synchronization service receives a rollback or partial rollback operation, it locates the transaction according to the transaction ID in the operation, and then marks the transaction with an operation rollback mark, indicating that part of the transaction is rolled back. If the operation is a rollback operation, then all operations in the transaction do not need to be executed and are directly discarded, and only the retained transaction information is added to the queue to be executed. The transaction information of the rolled back transaction cannot be discarded.

[0077] It should be noted that when a received transaction is rolled back, since the rolled back transaction operation will not be executed when the data is entered into the database, there may be transactions behind it that depend on these rolled back operations. If the rolled back transaction is directly discarded, the dependent transaction will not find the dependent transaction and will think there is no conflict, causing the subsequent transaction to be executed in advance, causing an error. For example, there are three transactions A, B, and C that have modified the same row of the same table, but transaction B is rolled back. When the target-side data synchronization service receives the operation, the operation conflict detection process will form a dependency order relationship in which B depends on A and C depends on B. If transaction B is directly discarded and not executed because it is rolled back, then C will not find the dependent transaction B when it is executed and will be executed in advance. In the end, transaction A and transaction C will be executed concurrently, causing an error.

[0078] In one embodiment, when the system restarts after a failure, the file is initialized to the cache file linked list of the target end data synchronization service according to the size of the cache file ID. It should be noted that when initializing the cache file linked list, it is necessary to recover the starting LSN of each cache file from the cache file name, and then read the records stored therein in turn and insert them into the hash table R. Only when both cache files are loaded can new operations be received and conflicts are checked for the new operations.

[0079] In one embodiment, the target-side data synchronization service creates H working threads, executes transactions in order of the size of the submitted LSN of the received transactions, and stores the transaction operations. It should be noted that the target-side data synchronization service also needs to create a hash table P to register the transaction information (transaction ID) currently being executed, and create a hash table G to register the table information of the current transaction and the start LSN of the transaction.

[0080] In one embodiment, the working thread extracts a transaction from the committed transaction linked list according to the transaction commit order, registers the transaction ID into the hash table P, and registers the table information involved in the transaction and the start LSN of the transaction into the hash table G.

[0081] In one embodiment, it is determined whether the current transaction is a rollback transaction. If it is, it is necessary to wait for all transactions with a commit LSN smaller than the current transaction to complete before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and the hash table G to complete the execution of the current transaction; otherwise, proceed to the next step. It should be noted that since the operations on the rolled back transaction do not need to be executed and are discarded, the transaction information they depend on is also unclear. Therefore, when submitting, it is necessary to wait until all transactions submitted earlier than the current transaction are submitted to ensure the correctness of the transaction dependency order.

[0082] In one embodiment, the working thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it is processed according to whether there is an operation rollback mark in the transaction; otherwise, the operation is a DML operation and proceeds to the next step.

[0083] In one embodiment, the working thread uses the conflicting TRXID of the current operation to be executed to search whether there is a transaction with the conflicting TRXID in the hash table P. If there is, the current execution process is suspended, and the conflicting transaction is woken up after being executed and submitted; otherwise, it jumps to the next step. It should be noted that the existence of a transaction with the conflicting TRXID in the hash table P indicates that the data involved in the current operation and the transaction with the conflicting TRXID are in conflict, and the current operation cannot be executed in parallel with the conflicting transaction, and the conflicting transaction needs to be submitted before it can be executed.

[0084] In one embodiment, the working thread uses the information of the current pending operation table to search in the hash table G whether there is a transaction operating the same table, and the transaction commit LSN is smaller than the current transaction, and the transaction start LSN is smaller than the conflict LSN of the current operation. If so, the current execution process is suspended, and the conflicting transaction is awakened after execution and submission; otherwise, if the conflict detection of the current pending operation of the working thread does not detect a conflict, the current operation is executed, and then the detection and judgment of the next operation is continued. It should be noted that this step is mainly to check whether there is a transaction beyond the conflict check LSN range in the current concurrently executed transactions, because the conflict transaction information on the pending operation has a valid interval range, that is, the conflict transaction information is only valid within the interval from the conflict LSN saved in the operation to the current operation LSN, and the operation does not perform conflict detection on the operation before the conflict LSN. That is to say, when there is a transaction whose commit LSN is smaller than the commit LSN of the transaction to which the current pending operation belongs, and whose start LSN is smaller than the conflict LSN saved by the current pending operation, it is unknown whether the operation on this transaction conflicts with the current pending operation. In this case, the current pending operation cannot be executed rashly and must wait until the transaction is committed before it can be executed.

[0085] refer to Figure 4 As shown, in one embodiment, processing according to whether there is an operation rollback mark in the transaction specifically includes the following steps.

[0086] Step 301: If there is no operation rollback mark, after committing the current transaction, wake up the waiting worker thread, remove the registration information of the current transaction from the hash table P and the hash table G, and complete the execution of the current transaction. It should be noted that since the transaction that each operation on the transaction depends on has been determined in the conflict detection, the transaction can be directly committed after executing all operations (indicating that the transactions it depends on have been committed), without waiting for the transaction submitted before itself to be committed first, thereby effectively improving the synchronization performance.

[0087] Step 302: If there is a rollback mark, wait for all transactions with a commit LSN smaller than the current transaction to complete before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and the hash table G to complete the execution of the current transaction. It should be noted that since there are operations that have been rolled back on the transaction, these operations have not been executed, so the transaction information they depend on is also unclear, so when releasing, it is necessary to wait until all transactions that were submitted earlier than itself are submitted to ensure the correctness of the transaction dependency order.

[0088] Based on the above process, this embodiment proposes to provide a transaction parallel execution method based on operation rollback mark, which mainly performs conflict detection on the currently received operation and the previously received operation within a set limited range. If the table ID and the primary key or the index or the KEY composed of the custom selected columns involved in the two operations are the same, it indicates that the current operation has a conflicting transaction within the limited range. When executing this operation, it is necessary to wait for the transaction of the conflicting operation to be executed first.

[0089] refer to Figure 5 As shown, the present embodiment further provides a transaction parallel execution system based on operation rollback mark, which applies the aforementioned transaction parallel execution method based on operation rollback mark, and the system includes a hash table and worker thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module, wherein: the hash table and worker thread creation module is used to create a hash table P, a hash table G, and a plurality of worker threads through the target end data synchronization service; the transaction extraction module is used to extract a transaction from the committed transaction linked list according to the transaction submission order through the worker thread, and register the transaction ID to the hash table P, and register the table information involved in the transaction and the starting LSN of the transaction to the hash table G; the rollback judgment module is used to determine whether the current transaction is a rollback transaction, and if so, wait until all transactions with a submitted LSN smaller than the current transaction are submitted before waking up the worker waiting for the current transaction Work thread; otherwise, call the operation extraction and conflict detection module; the operation extraction and conflict detection module is used to extract an operation of the transaction through the working thread, determine whether the current operation is a commit operation, and if so, process it according to whether the transaction has an operation rollback mark; otherwise, the operation is determined to be a DML operation, and the working thread uses the conflict TRXID of the current operation to be executed to search in the hash table P whether there is a transaction with the conflict TRXID, if so, wait for the conflicting transaction to be executed and committed before waking itself up; otherwise, the working thread uses the current operation table information to be executed to search in the hash table G whether there is a transaction operating the same table, and the transaction commit LSN is smaller than the current transaction, and the transaction start LSN is smaller than the conflict LSN of the current operation, if so, wait for the conflicting transaction to be executed and committed before waking itself up; otherwise, it is determined that no conflict is detected and the current operation is executed. The specific functions and principles of the system function module correspond to the specific methods in the method part, and will not be repeated here.

[0090] In summary, this embodiment proposes to provide a transaction parallel execution method and system based on operation rollback mark, including the following advantages: First, the target end data synchronization service receives the operations sent by the source end in sequence. If there are conflicting operations, the first received one should be executed first, and the later received one should be executed later. This order is also the basis for conflict operation detection. The currently received operation can be compared with the previously received operation. If there are transaction operations with the same table and the same KEY, it means that the current operation and a previous transaction are in conflict. When the current operation is compared with the previously received operation, a range must be set. The larger the range, the higher the accuracy, but at the same time, the more resources are consumed. This requires determining the size of the detection range according to the current hardware conditions. The embodiment of the present invention performs conflict detection on the currently received operation and the previously received operation within a limited range. If the KEYs involved in the two operations are the same, it indicates that there is a conflicting transaction within the limited range for the current operation. When the operation is executed, it needs to wait for the transaction of the conflicting operation to be executed before it can be executed.

[0091] Secondly, using the file system to save the KEY information of the received operations can effectively ensure the continuity of the conflict detection of the received operations. After the target-side data synchronization service is restarted after a failure, the KEY information of the previously received operations can be restored. In this way, the conflict relationship of the newly received operations is continuous with that before the failure, ensuring that the failure will not affect the dependency relationship of the positioning operation, thereby ensuring that the parallelism of the operation execution will not be affected after the system failure recovery.

[0092] Finally, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to mark the corresponding transaction to indicate that an operation on the transaction has been rolled back. If the transaction is a rollback transaction, it must also be added to the queue for execution and cannot be discarded directly. Otherwise, subsequent transactions that depend on it will be executed early because the dependent transaction cannot be found.

[0093] Embodiment 2:

[0094] Based on the transaction parallel execution method based on operation rollback marker provided in the above-mentioned embodiment 1, embodiment 2 of the present invention provides a specific example for illustration.

[0095] For example: The source database has table T (ID INT PRIMARY KEY, C1 INT);

[0096] The source application has three transactions that perform the following operations on table T in the following order:

[0097] TRX1: INSERT INTO T(ID, C1)VALUES(1, 1);

[0098] TRX1:COMMIT;

[0099] TRX2: INSERT INTO T(ID, C1)VALUES(2, 2);

[0100] TRX2: UPDATE T SET C1=1WHERE ID=1;

[0101] TRX2:ROLLBACK;

[0102] TRX3: UPDATE T SET C1=2WHERE ID=1;

[0103] TRX3:COMMIT;

[0104] The order of the above operations represents the order of these operations in the log stream, and the resulting log stream will form the following numbered table:

[0105]

[0106]

[0107] After the target-side data synchronization service is started, a hash table R of size N and two cache files of size M need to be initialized. The cache files are numbered 1 and 2 in sequence.

[0108] The receiving process is as follows:

[0109] 1. The log receiving thread receives the operation with LSN 1, extracts its table ID and primary key as KEY, assuming it is 1, and combines TRXID and the space of the next record to make the length 32. First, store its KEY and transaction ID in the first cache file, expressed as [file ID (K): file offset (F)], and its offset in the file is [1:0].

[0110] KEY:1TRXID:1NEXT:empty

[0111] Since the KEY information is the first operation information of the current cache file, the LSN value of the operation should be set to the starting LSN of the file, and the starting LSN of the file should be set to 1.

[0112] 2. Insert the KEY into the hash table R. The returned TRXID is 0, indicating no conflict. The TRXID 0 indicating no conflict and the minimum starting LSN of the cached file is 1, which is written back to the operation.

[0113] 3. The log receiving thread receives the operation with LSN 3, extracts its table ID and primary key as KEY, assuming it is 2, and combines the TRXID and the space for the next record to make the length 32. First, store its KEY and transaction ID in the first cache file. Its offset in the file is [1:32].

[0114] KEY:1TRXID:1NEXT:empty KEY: 2TRXID: 2NEXT: empty

[0115] 4. Insert the KEY into the hash table R. The returned TRXID is 0, indicating no conflict. The TRXID 0 indicating no conflict and the minimum starting LSN of the cache file is 1, which is written back to the operation.

[0116] 5. The log receiving thread receives the operation with LSN 4, extracts its table ID and primary key as KEY, assuming it is 1, and combines the TRXID and the space for the next record to make the length 32. First, store its KEY and transaction ID in the first cache file. Its offset in the file is [1:64].

[0117] KEY:1TRXID:1NEXT:empty KEY: 2TRXID: 2NEXT: empty

[0118] KEY: 1TRXID: 2NEXT: empty

[0119] 6. Insert the KEY into the hash table R. Since the operation information with KEY 1 already exists in the HASH, the conflicting TRXID returned is 1, which means that there is a conflict between the current operation and the transaction with transaction ID 1. The conflicting TRXID 1 and the minimum starting LSN 1 of the cache file are written back to the operation.

[0120] 7. Remove the conflicting KEY information with KEY 1 from the HASH table.

[0121] 8. Set the address in the hash table R with KEY 1 to the next record of the current KEY information to obtain the following table.

[0122] KEY:1TRXID:1NEXT:empty KEY: 2TRXID: 2NEXT: empty KEY:1TRXID:2NEXT:1:0

[0123] 9. Restart the target data synchronization service and load cache file 1 and cache file 2 in sequence.

[0124] 10. Load and merge the KEY information in cache file 1 into hash table R to restore the HASH state before the failure.

[0125] 11. The log receiving thread receives the rollback operation with LSN 5, marks its corresponding TRX2, indicating that there is a rolled back operation on the transaction, and then adds TRX2 to the execution queue.

[0126] 12. The log receiving thread receives the operation with LSN 6, extracts its table ID and primary key as KEY, assuming it is 3, and combines the TRXID and the space for the next record to make the length 32. First, store its KEY and transaction ID in the first cache file. Its offset in the file is [1:96].

[0127] KEY:1TRXID:1NEXT:empty KEY: 2TRXID: 2NEXT: empty KEY:1TRXID:2NEXT:1:0 KEY:3TRXID:3NEXT:empty

[0128] 13. Insert the KEY into hash table R. The returned TRXID is 2, which means that there is a conflict between the current operation and the transaction with transaction ID 2. The conflicting TRXID 2 and the minimum starting LSN 1 of the cache file are written back to the operation.

[0129] After all operation conflicts are detected, the following detection results are formed.

[0130]

[0131]

[0132] 14. Create three working threads and execute TRX1, TRX2 and TRX3 respectively in the order in which the transactions were submitted.

[0133] 15. Worker thread 1 extracts the TRX1 transaction, registers the transaction information TRXID as 1 in the hash table P, and registers the starting LSN of the table information T associated transaction as 1 in the hash table G.

[0134] 16. Worker thread 2 extracts the TRX2 transaction, registers the transaction information TRXID 2 into hash table P, and registers the starting LSN 3 of the transaction associated with table information T into hash table G.

[0135] 17. Worker thread 3 extracts the TRX3 transaction, registers the transaction information TRXID 3 into hash table P, and registers the starting LSN 6 of the table information T associated transaction into hash table G.

[0136] 18. Worker thread 1 extracts the insert operation with LSN 1 on TRX1, obtains the conflicting TRXID and conflicting LSN on the operation for conflict detection.

[0137] (1) This operation conflicts. TRXID is 0, indicating no conflict.

[0138] (2) Get the table information T and the conflict LSN of the operation as 1, and search the hash table G for conflicting transactions involving table T and with commit LSN less than 2 and transaction minimum start LSN less than 1. If no conflicting transaction is found, it can be executed directly.

[0139] 19. Worker thread 2 extracts TRX2. This transaction is marked with a rollback transaction, and it is a rollback transaction, so it does not need to be executed. However, it needs to wait for TRX1 with a smaller LSN than it to be committed before it can remove its own transaction information from the registered hash tables P and G, and then wake up the transaction waiting for it to complete execution.

[0140] 20. Work thread 3 extracts the update operation with LSN 6 on TRX3, obtains the conflicting TRXID and conflicting LSN on the operation for conflict detection.

[0141] (1) The conflicting operation TRXID is 2, indicating that it conflicts with TRX2. The execution of the current working thread is suspended, and it will wake itself up after TRX2 is executed.

[0142] From here we can see that TRX3 depends on TRX2, but TRX2 itself is rolled back and will not perform any operations. If TRX2 is directly discarded and not executed, the update of TRX3 will not find the conflicting transaction and will be executed in advance concurrently with the insert operation of TRX1. However, these two transactions operate on the same row. If the update operation of TRX3 is executed before the insert operation of TRX1, an error will be reported.

[0143] 21. After worker thread 1 executes the commit operation of TRX1, it deletes the current transaction information from hash table P and hash table G respectively, and then wakes up the execution of TRX2 transaction on worker thread 2.

[0144] 22. After transaction TRX2 of worker thread 2 is awakened, it does not need to commit any operations because it is a rolled back transaction. It only needs to delete the current transaction information from hash table P and hash table G and then wake up and wait for its own transaction TRX3 to be executed.

[0145] 23. After worker thread 3 is awakened, it executes the update operation of TRX3, and then executes the commit operation and deletes the current transaction information from hash table P and hash table G to complete the execution of the transaction.

[0146] From the above process, we can see that transaction TRX1 is received first, followed by the operation of transaction TRX2. TRX2 will find that its operation depends on transaction TRX1 during the operation conflict detection. Although TRX2 finally receives the rollback operation, which means that this transaction will not be executed, the operation of TRX2 has covered the dependency relationship of the operation in the HASH table during the detection, resulting in the operation of the last received TRX3 transaction relying on TRX2 instead of TRX1 during the conflict detection. Therefore, TRX2 should be marked with a rollback operation and added to the execution chain list. Ensure that the cascade relationship of TRX3 relying on TRX2 and TRX2 relying on TRX1 in the transaction dependency relationship will not be interrupted. Although TRX2 is a rollback transaction, it also needs to be registered during execution to ensure the continuity of the dependency relationship. Since there is no operation that needs to be executed in the TRX2 transaction, TRX2 has no dependent transaction information during execution. Its release can only be released after all transactions with a commit LSN smaller than it have been committed.

[0147] Embodiment 3:

[0148] It should be noted that for rollback transactions, since the rollback transactions will not be executed, there are three ways to process the rollback transaction operations in the hash table R.

[0149] The first method: do not roll back the impact of the rollback transaction operation on the operation dependency in the hash table R, and do not collect the conflicting TRXIDs of the rollback transaction operation dependency. Directly release all the operations of the rollback transaction. This rollback transaction will be dependent on other operations. The rollback transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependency. When this solution is adopted, the operations of the rollback transaction are directly released as a whole, which can improve the processing performance of the rollback transaction. However, the operations that depend on the rollback transaction must wait until the rollback transaction is released in the pending execution list before they can be executed. Since the conflicting transactions of the operation dependency were not collected before the release of the rollback transaction, it is necessary to wait until all transactions whose commit LSN or rollback LSN is less than its own rollback LSN are executed before they can be released, which will affect the synchronization efficiency. This method corresponds to that when the target-side data synchronization service receives the rollback transaction, all operations on this transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the pending execution queue. For example, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to locate the transaction according to the transaction ID in the operation, and then mark the transaction to indicate that part of the transaction is rolled back. If the operation is a rollback operation, then all operations on the transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed, and the transaction information of the rolled back transaction cannot be discarded. The description of the rollback part in Embodiment 1 of the present invention is an example description under this method.

[0150] The second method: do not roll back the impact of the rollback transaction operation on the operation dependency in the hash table R, but collect the conflicting TRXIDs of the rollback transaction operation dependency, for example, record it in the conflicting TRXID dependency list L. This rollback transaction will be relied upon by other operations, and the rollback transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependency. When this solution is adopted, the conflicting TRXIDs in the operation must be collected before the rollback transaction operation is released. Although the processing performance of the rollback transaction is reduced, the release performance of the rollback transaction will be improved during execution. The operation that depends on the rollback transaction must wait until the rollback transaction is released in the pending execution list before it can be executed. Since the conflicting transactions of the operation dependency were collected before the release operation, the rollback transaction can be released only after these conflicting transactions are executed, without waiting for all transactions whose commit LSN or rollback LSN is less than its own rollback LSN to be executed. Compared with the first solution, this will partially improve the efficiency of synchronization. This method corresponds to that when the target-side data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation, and for the rollback operation, the corresponding transaction cannot be discarded, but is added to the queue of pending transactions.

[0151] The third method: roll back the impact of the rolled-back transaction on the operation dependencies in hash table R. Then the rolled-back transaction will not be dependent on other operations, and the rolled-back transaction does not need to be added to the pending execution list. When this solution is adopted, after receiving the rollback operation, the transaction needs to roll back the impact of all operations in the transaction on the operation dependencies in hash table R. Since the transaction scale is uncontrollable, when a large transaction is rolled back, the rollback action will affect the progress of receiving the operation, thereby affecting the efficiency of synchronization. This method corresponds to that when the target-side data synchronization service receives a rollback or partial rollback operation, the conflicting TRXID saved in the rolled-back operation is written back to hash table R.

[0152] The above three solutions for rolling back transactions have their own advantages and disadvantages. The first solution is suitable for large-scale transaction rollback, the second solution is suitable for medium-scale transaction rollback, and the third solution is suitable for small-scale transaction rollback. Therefore, two parameters W1 and W2 are set. W1 represents the minimum number of DML operations in large-scale transactions, and W2 represents the minimum number of DML operations in medium-scale transactions. Based on this, after the target-side data synchronization service receives the DML operation from the source, it is also necessary to manage the received DML operations by transaction classification, and count the scale S of DML operations received by each transaction (S is the number of DML operations received in a single transaction). When the target-side data synchronization service receives a rollback operation, it takes corresponding actions according to the scale S of DML operations in the transaction.

[0153] When the scale S of the rolled-back transaction is greater than or equal to W1, it corresponds to the large-scale transaction rollback in the first solution. In this case, the impact of the rolled-back transaction operation on the operation dependency in the hash table R is not rolled back, and the conflicting TRXID of the rolled-back transaction operation dependency is not collected. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending execution linked list for execution to ensure the continuity of the operation dependency. The description of the rollback part in Embodiment 1 of the present invention is an example description when the scale S of the rolled-back transaction is greater than or equal to W1.

[0154] When the scale S of the rolled-back transaction is greater than or equal to W2 and less than W1, it corresponds to the medium-scale transaction rollback in the second solution. In this case, the impact of the rolled-back transaction operation on the operation dependency in the hash table R is not rolled back, but the conflicting TRXIDs of the rolled-back transaction operation dependency are collected, and it can be selected to be recorded in the conflicting TRXID dependency linked list L. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending execution linked list for execution to ensure the continuity of the operation dependency.

[0155] When the scale S of the rolled-back transaction is less than W2, it corresponds to the small-scale transaction rollback in the third solution. In this case, the impact of the rolled-back transaction operation on the operation dependency in the hash table R is rolled back, and the rolled-back transaction does not need to be added to the pending linked list and is directly released.

[0156] Furthermore, for partial rollback operations, due to the particularity of partial rollback, partial rollback itself needs to traverse the operations to be rolled back, so adding the process of collecting conflicting TRXIDs on these operations will not significantly reduce the performance of processing partial rollbacks. Based on this, when the target-side data synchronization service receives a partial rollback operation, it is necessary to count the total number of rolled back rows S1, and perform the following processing according to the size of S1: When the number of rollback operation rows S1 is less than W2, the impact of the rollback operation on the operation dependency in the hash table R is rolled back, that is, the conflicting TRXIDs saved in the rolled back operation are written back to the hash table R. When the number of rollback operation rows S1 is greater than W2, the impact of the rollback transaction operation on the operation dependency in the hash table R is not rolled back, that is, the conflicting TRXIDs saved in the rolled back operation are not written back to the hash table R, but the conflicting TRXIDs of the rollback transaction operation dependency are collected, and the conflicting TRXIDs can be recorded in the conflicting TRXID dependency linked list L.

[0157] Based on the above settings, after the target-side data synchronization service creates a working thread, the working thread extracts a transaction from the committed transaction list in the order in which the transaction is committed, and determines whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to determine whether the rollback transaction has a conflicting TRXID linked list L.

[0158] If the rolled back transaction does not have a conflicting TRXID linked list L, it is necessary to wait for all transactions with a smaller LSN than the current transaction to be committed before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and hash table G to complete the execution of the current transaction. This is because the operations on the rolled back transaction do not need to be executed and are discarded, so the transaction information they depend on is also unclear. Therefore, when submitting, it is necessary to wait until all transactions submitted earlier than itself are submitted to ensure the correctness of the transaction dependency order.

[0159] If the rolled back transaction has a conflicting TRXID linked list L, it is necessary to wait for all conflicting transactions on the conflicting transaction linked list L to commit before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and hash table G to complete the execution of the current transaction. Specifically, extract a conflicting TRXID from the current transaction conflicting transaction linked list L in turn, and check whether it exists in the hash table P. If it exists, it is necessary to wait for the transaction to be completed before waking itself up. After waking up, the next conflicting TRXID is extracted from the conflicting transaction linked list L and the same check is performed until all conflicting transactions in the conflicting transaction linked list L are completed. This is because the operations on the rolled back transaction do not need to be executed and are discarded. When the rollback operation is discarded, the conflicting TRXIDs on the operation are collected, so the transaction information they depend on is very clear. Then, when releasing, you only need to wait for these conflicting transactions to be committed to ensure the correctness of the transaction dependency order.

[0160] In summary, since the rollback transaction will not be executed, it will generate three schemes during conflict detection. Each scheme has its own advantages, and appropriate values ​​need to be set to give full play to the advantages of the three schemes, so as to achieve the purpose of improving the overall synchronization performance. In addition, for the partial rollback operation of the committed transaction, the conflicting TRXIDs referenced by them are collected in a targeted manner during the rollback operation, which can effectively reduce the performance loss caused by the partial rollback operation writing back the conflict detection hash table R, so as to achieve the purpose of improving the overall synchronization performance. This embodiment processes the rollback transaction and partial rollback operation in a targeted manner according to the agreed operation scale, and finds the balance point between the cost of operation rollback and rollback transaction execution, thereby improving the overall synchronization performance.

[0161] In addition, in some embodiments, the KEY in the hash table R can be stored in a file or in a memory chip. Specifically, the difference between the two is that when using files for storage, although the storage performance is low and affects the efficiency of conflict detection, after a data synchronization failure, the state of the hash table R between failures can be restored by storing the KEY value in the file, so that when performing conflict detection on the received operations after the failure, there is continuity with the failure, and the synchronization performance will not drop drastically due to the failure; when using memory chips for storage, the performance of conflict detection will be greatly improved compared to the file storage solution, but the data of the previous conflict detection will be lost after the failure, and the recovery detection can only start from the currently received LSN, which will cause a discontinuity between the subsequent conflict detection and the failure, and the synchronization performance will drop drastically at the failure location during synchronization execution. When the storage performance is high (for example, using SSD fixed storage), the performance requirements of conflict detection can be met, and the efficiency of operation reception can be balanced with synchronous execution. Then, the method of storing the KEY in the hash table R through files can effectively avoid the loss of synchronization performance caused by failures; when the storage performance affects the efficiency of conflict detection, the bottleneck of synchronization exists in the operation reception. At this time, the method of storing the KEY in the hash table R through a memory chip can effectively improve the reception performance of the operation, thereby improving the overall synchronization performance. Embodiment 1 of the present application is an example of a method of storing the KEY in the hash table R through files.

[0162] In some embodiments, you can choose to calculate the MD5 value at the source or target end as the KEY of the operation. The data synchronization service at the source end detects conflicts for each operation and can calculate the MD5 value of the conflict more accurately. If the primary key structure or column order of the target database table is inconsistent with the source end, the MD5 value calculated by the primary key information of the target end table will have errors. Based on this, it is generally chosen to calculate the MD5 value at the source end as the KEY of the operation. After the target end data synchronization service receives the DML operation at the source end, it can directly extract the MD5 value calculated by the source end as the KEY.

[0163] In some embodiments, when calculating the KEY of an operation, since the order of columns in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the KEY is calculated according to the fixed order of the primary key or unique index combination columns. After the source-side data synchronization service captures a DML log operation, it obtains the table ID in the log, and obtains the primary key information and unique index of the table in the source-side database through the table ID. After parsing the current DML log operation, the source-side data synchronization service determines whether the table has a primary key or unique index. If there is a primary key or unique index, the table ID and the column value of the primary key or unique index are combined to calculate an MD5 value as the KEY of this operation, which is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when a table has both a primary key and a unique index, the table ID and the primary key are preferably selected to jointly calculate the MD5 value as the KEY; when a table has no primary key but has multiple unique indexes, the table ID and any unique index are selected to jointly calculate the MD5 value. If there is no primary key or unique index, after parsing the current DML log operation, the source-side data synchronization service combines the table ID and the row data's ROWID value or custom selected column to calculate an MD5 value as the KEY for this operation, stores it in the current DML log operation, and sends it to the target-side synchronization service. It should be noted that when calculating the KEY for a table without a primary key or unique constraint, if it is clear that all columns of the table have no duplicate values ​​when combined, the table's ROWID can be used directly to calculate the KEY, which can effectively reduce the cost of calculating the KEY; if the table has duplicate rows, then it is necessary to use custom selected columns to calculate the KEY, and which columns or all column values ​​to use to calculate the KEY can be selected based on the distribution of column values, so that the target-side synchronization service can use the KEY to prevent deadlocks caused by concurrently executed synchronization transactions operating rows with duplicate values ​​at the same time.

[0164] Embodiment 4:

[0165] Based on the transaction parallel execution method based on operation rollback markers provided in the above embodiment 1, the present invention also provides a transaction parallel execution device based on operation rollback markers that can be used to implement the above method and system, such as Figure 6 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The transaction parallel execution device based on operation rollback mark of this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0166] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0167] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the transaction parallel execution method based on operation rollback markers in Example 1. The processor 21 executes various functional applications and data processing of the transaction parallel execution device based on operation rollback markers by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, implements the transaction parallel execution method based on operation rollback markers in Example 1.

[0168] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the transaction parallel execution method based on the operation rollback mark in the above embodiment 1 is executed, for example, the above described Figure 1-Figure 4 The steps shown.

[0170] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0171] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A transaction parallel execution method based on operation rollback mark, characterized in that: include: The target-side data synchronization service creates hash table P, hash table G, and several worker threads; The worker thread extracts a transaction from the committed transaction list in the order of transaction submission, registers the transaction ID into the hash table P, and registers the table information involved in the transaction and the transaction start LSN into the hash table G; Determine whether the current transaction is a rollback transaction. If so, wait until all transactions with a commit LSN smaller than the current transaction are committed before waking up the worker thread waiting for the current transaction. Otherwise, perform the operation extraction step; Operation extraction step: The working thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, it processes the operation based on whether there is an operation rollback mark in the transaction. Otherwise, the operation is judged to be a DML operation. The worker thread uses the conflicting TRXID of the current operation to be executed to search whether there is a transaction with the conflicting TRXID in the hash table P. If so, it waits for the conflicting transaction to be executed and committed before waking itself up. Otherwise, the worker thread uses the information of the current pending operation table to search in the hash table G to see if there is a transaction operating the same table, and the transaction commit LSN is smaller than the current transaction, and the transaction start LSN is smaller than the conflict LSN of the current operation. If there is, it will wait for the conflicting transaction to be executed and committed before waking itself up; Otherwise, it is determined that no conflict is detected and the current operation is performed.

2. The transaction parallel execution method based on operation rollback mark according to claim 1 is characterized in that: When the target-side data synchronization service starts, a hash table R is initialized, and the record address in the hash item of the hash table R is initialized to empty, which is used to store and locate the received operations; the target-side data synchronization service creates two cache files to store the KEY in the hash table R, and the file name of each cache file is named incrementally in the order of creation, and the two cache files are added to the cache file linked list.

3. The transaction parallel execution method based on operation rollback mark according to claim 2 is characterized in that: The acquisition of the conflicting TRXID and the conflicting LSN includes: After receiving the DML operation from the source, the target-side data synchronization service extracts the table ID and its primary key or index or custom selected column in the operation as the KEY, and then extracts the transaction ID in the operation as the TRXID, and combines the KEY and TRXID into a record and saves it to the current cache file. If the record is the first record of the current cache file, the log LSN corresponding to the operation is marked on the cache file as the starting LSN of the cache file; Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection and judgment during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict; The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first cache file corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

4. The transaction parallel execution method based on operation rollback mark according to claim 3 is characterized in that: The conflict detection and judgment comprises: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item, and return 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next step of conflict detection and judgment; Determine whether the file ID in the address recorded in the hash item is less than the ID value of the first file in the cache file list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment; According to the address recorded in the hash item, load the corresponding record Y from the file, determine whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, remove record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, set the record address in the hash item to the next address space of the current detection record X, fill the address of the current detection record X back into the hash item, and use the TRXID stored in record Y as the conflicting TRXID to return; if not, proceed to the next step of conflict detection and judgment; Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record linked list.

5. The transaction parallel execution method based on operation rollback mark according to claim 2, characterized in that: The cache file linked list includes two cache files. When the first cache file is full, it switches to the second cache file. When the second cache file is full, the first cache file in the linked list is eliminated first, and then a new cache file is created with an incremented file ID and added to the cache file linked list.

6. The transaction parallel execution method based on operation rollback mark according to claim 2, characterized in that: When the target-side data synchronization service receives a rollback or partial rollback operation, it locates the transaction according to the transaction ID on the operation, and then marks the transaction with an operation rollback mark, indicating that some operations of the transaction are rolled back. If the operation is a rollback operation, then all operations on this transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed.

7. The transaction parallel execution method based on operation rollback marker according to any one of claims 1 to 6, characterized in that: The processing according to whether there is an operation rollback mark in the transaction specifically includes: If there is no operation rollback mark, after committing the current transaction, wake up the waiting worker thread, remove the registration information of the current transaction from hash table P and hash table G, and complete the execution of the current transaction; If there is a rollback mark, wait for all transactions with a committed LSN smaller than the current transaction to be committed before waking up the waiting worker thread, and then remove the registration information of the current transaction from hash tables P and G to complete the execution of the current transaction.

8. A transaction parallel execution device based on operation rollback mark, characterized in that: The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the transaction parallel execution method based on operation rollback marking as described in any one of claims 1-7.

9. A transaction parallel execution system based on operation rollback markers, applying the transaction parallel execution method based on operation rollback markers as claimed in any one of claims 1 to 7, characterized in that: The system includes a hash table and work thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module, wherein: The hash table and worker thread creation module is used to create a hash table P, a hash table G and several worker threads through the target end data synchronization service; The transaction extraction module is used to extract a transaction from the committed transaction linked list according to the transaction submission order through a working thread, and register the transaction ID into a hash table P, and register the table information involved in the transaction and the start LSN of the transaction into a hash table G; The rollback judgment module is used to judge whether the current transaction is a rollback transaction. If so, it will wait for all transactions with a commit LSN smaller than the current transaction to complete before waking up the working thread waiting for the current transaction; otherwise, it will call the operation extraction and conflict detection module; The operation extraction and conflict detection module is used to extract an operation of a transaction through a working thread, determine whether the current operation is a commit operation, and if so, process it according to whether the transaction has an operation rollback mark; otherwise, the operation is determined to be a DML operation, and the working thread uses the conflict TRXID of the current operation to be executed to search whether there is a transaction with the conflict TRXID in the hash table P, and if so, wait for the conflicting transaction to be executed and committed before waking itself up; otherwise, the working thread uses the information of the current operation table to be executed to search in the hash table G whether there is a transaction operating the same table, and the commit LSN of this transaction is smaller than the current transaction, and the start LSN of the transaction is smaller than the conflict LSN of the current operation, and if so, wait for the conflicting transaction to be executed and committed before waking itself up; otherwise, it is determined that no conflict is detected and the current operation is executed.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the transaction parallel execution method based on operation rollback markers as described in any one of claims 1-7.

Images