A transaction parallel execution method, device and system based on operation rollback marking

Abstract

The application relates to a transaction parallel execution method, device and system based on operation rollback marking. The method mainly comprises the following steps: a work thread extracts a transaction from a submitted transaction chain table according to a transaction submission sequence, judges whether the current transaction is a rollback transaction, if yes, waits for all transactions with a submission LSN smaller than that of the current transaction to be submitted, and then wakes up the work thread; the work thread searches a hash table P using a conflict TRXID of a current operation to be executed to check whether a transaction with the conflict TRXID exists, if yes, waits for being woken up; otherwise, the work thread searches a hash table G using table information of the current operation to be executed to check whether a transaction with the same table exists, and the transaction has a submission LSN smaller than that of the current transaction and a starting LSN smaller than a conflict LSN of the current operation, if yes, waits for being woken up; otherwise, it is judged that no conflict is detected, and the current operation is executed. The application can effectively increase the concurrency of operation execution, and improves the synchronization performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of database technology, and particularly to a transaction parallel execution method, device and system based on operation rollback marking. BACKGROUND

[0002] In a database real-time synchronization system based on log analysis architecture (supporting heterogeneous), a source data synchronization service is responsible for capturing operation logs of a source database, analyzing the logs to restore corresponding database operations, and then sending the operations to a target data synchronization service. The target data synchronization service is responsible for executing the operations in a target database, thereby completing the data synchronization process.

[0003] When the target data synchronization service synchronizes operations on the target database, it often needs to execute according to the commit order of transactions. Because when the previous transaction is not synchronized, the operation of the next transaction cannot be executed into the database concurrently with the operation of the previous transaction without explicitly determining whether there is a conflict with the previous transaction, but can only start to execute after all transactions before the current operation are committed, which greatly affects the concurrency of transaction operation into the database. The effective method currently adopted is to determine whether the operation can be executed in parallel by judging the size of the log LSN of the operation and the LSN of the commit operation of the previous transaction. If the LSN of the current transaction operation is smaller than the LSN of the commit operation of the other transaction being executed, the operation can be executed concurrently with the transaction. This strategy helps to improve concurrency, but the performance improvement is small.

[0004] Therefore, how to overcome the defects of the prior art and improve the parallel execution of the target data synchronization service to improve the performance of data synchronization has become an important technical problem to be solved in the industry. SUMMARY

[0005] In view of defects in the prior art or improvement needs: how to improve the strength of parallel execution of target end data synchronization service to improve the performance of data synchronization. In order to improve the performance of synchronization, the target end data synchronization service sets the dependency information of each operation when receiving the operation, so as to facilitate concurrent execution with finer granularity when executing, and improve the performance. Based on this, the application provides a transaction parallel execution method, device and system based on operation rollback mark. When the target end data synchronization service receives a synchronization transaction operation, it is received in the generation order of the source end database log, so the order of each operation received is the basis for detecting whether there is a conflict. The application extracts the primary key or index or custom selected column on each received operation as KEY based on the order of operation reception, finds whether the KEY in the set range is the same as the operation of other transactions received before, if yes, the execution of the operation needs to wait for the submission of the conflicting transaction before starting to execute; otherwise, it can be executed concurrently with the transaction submitted before itself, maximizing the parallel degree of transaction execution, thereby effectively improving the performance of synchronization.

[0006] The application adopts the following technical solutions:

[0007] In a first aspect, the application provides a transaction parallel execution method based on operation rollback mark, comprising:

[0008] The target end data synchronization service creates a hash table P, a hash table G and a plurality of working threads;

[0009] The working thread extracts a transaction from the committed transaction list in the transaction submission order, and registers the transaction ID to the hash table P, and registers the table information involved in the transaction and the starting LSN of the transaction to the hash table G;

[0010] It is judged whether the current transaction is a rollback transaction, if yes, the working thread waiting for the current transaction is awakened after all transactions with a smaller LSN than the current transaction are submitted; otherwise, the operation extraction step is performed;

[0011] Operation extraction step: the worker thread extracts an operation of a transaction, judges whether the current operation is a commit operation, if yes, handles according to whether there is an operation rollback mark in the transaction; otherwise, judges whether the operation is a DML operation, the worker thread searches in the hash table P whether there is a transaction with the conflict TRXID of the current operation to be executed, if yes, waits until the conflict transaction is executed and committed, and then wakes up itself; otherwise, the worker thread searches in the hash table G whether there is a transaction with the same table as the current operation, and the transaction has a commit LSN less than the current transaction, and the start LSN of the transaction is less than the conflict LSN of the current operation, if yes, waits until the conflict transaction is executed and committed, and then wakes up itself; otherwise, judges that no conflict is detected, and executes the current operation.

[0012] In some embodiments, the target-side data synchronization service initializes a hash table R when starting, and initializes the record address in the hash item of the hash table R as empty for storing and locating the received operation; the target-side data synchronization service creates two cache files for storing the KEY in the hash table R, and the file names of each cache file are named in increasing order according to the creation order, and adds the two cache files to the cache file linked list.

[0013] In some embodiments, the obtaining of the conflict TRXID and the conflict LSN comprises:

[0014] After the target-side data synchronization service receives the DML operation of the source side, the table ID and the primary key or the index or the custom selected column in the operation are extracted as the KEY, and the transaction ID in the operation is extracted as the TRXID, the KEY and the TRXID are combined into a record and saved to the current cache file, if the record is the first record of the current cache file, the log LSN corresponding to the operation is marked on the cache file as the start LSN of the cache file;

[0015] A detection record X is constructed using the KEY of the current operation, the detection record X is merged into the hash table R, and conflict detection judgment is performed in the merging process, if there is a conflict KEY, the TRXID value corresponding to the conflict KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict;

[0016] The TRXID obtained by the conflict detection judgment is set as the conflict TRXID, and the start LSN of the first cache file corresponding to the current hash table R is written into the current operation as the conflict LSN.

[0017] In some embodiments, the conflict detection judgment comprises:

[0018] Hashing the KEY of the current detection record X, locating to the hash item through the hash value; judging whether the record address in the hash item is empty, if yes, setting the record address in the hash item to the next address space of the current detection record X, backfilling the address of the detection record X into the hash item, and returning 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceeding to the next conflict detection judgment;

[0019] judging whether the file ID in the record address in the hash item is less than the ID value of the first file in the cache file linked list, if yes, indicating that the record Y has exceeded the checking range, setting the record address in the hash item to the next address space of the current detection record X, backfilling the address of the detection record X into the hash item, and returning 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceeding to the next conflict detection judgment;

[0020] loading the corresponding record Y from the file according to the record address in the hash item, judging whether the KEY of the record Y is equal to the KEY of the current detection record X and the TRXID of the record Y is not equal to the TRXID of the current operation, if yes, indicating that the operation exists conflict, removing the record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, setting the record address in the hash item to the next address space of the current detection record X, backfilling the address of the current detection record X into the hash item, and returning the TRXID stored in the record Y as the conflict TRXID; if not, proceeding to the next conflict detection judgment;

[0021] sequentially traversing the next record of the record Y and performing conflict detection judgment until the end of the record linked list.

[0022] In some embodiments, the cache file linked list includes two cache files, when the first cache file is full, switching to the second cache file, when the second cache file is full, discarding the first cache file in the linked list first, and then creating a new cache file with an incremental file ID to join the cache file linked list.

[0023] In some embodiments, when the target end data synchronization service receives a rollback or partial rollback operation, locating to the transaction according to the transaction ID on the operation, and then marking the operation rollback on the transaction, indicating that the transaction has partial operation rolled back, if the operation is a rollback operation, the entire operation on the transaction does not need to be executed, and is discarded directly, only the reserved transaction information is added to the execution queue.

[0024] In some embodiments, the processing according to whether the transaction has the operation rollback mark specifically includes:

[0025] If there is no operation rollback mark, the current transaction is woken up after being committed, and the registration information of the current transaction is removed from the hash table P and the hash table G to complete the execution of the current transaction;

[0026] If there is a rollback mark, the current transaction is woken up after all transactions with a smaller commit LSN are committed, and the registration information of the current transaction is removed from the hash table P and the hash table G to complete the execution of the current transaction.

[0027] In a second aspect, the present application further provides a transaction parallel execution device based on operation rollback marks, the device comprising:

[0028] at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the transaction parallel execution method based on operation rollback marks in the first aspect.

[0029] In a third aspect, the present application further provides a transaction parallel execution system based on operation rollback marks, which applies the transaction parallel execution method based on operation rollback marks in the first aspect, and the system comprises a hash table and a work thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module, wherein:

[0030] The hash table and the work thread creation module are used to create a hash table P, a hash table G, and a plurality of work threads through a target end data synchronization service;

[0031] The transaction extraction module is used to extract a transaction from the committed transaction linked list in the order of transaction submission through the work thread, and register the transaction ID to the hash table P, and register the table information involved in the transaction and the start LSN of the transaction to the hash table G;

[0032] The rollback judgment module is used to judge whether the current transaction is a rollback transaction, if yes, the work thread waiting for the current transaction is woken up after all transactions with a smaller commit LSN are committed; otherwise, the operation extraction and conflict detection module is called;

[0033] The operation extraction and conflict detection module is configured to extract one operation of a transaction by a worker thread, determine whether the current operation is a commit operation, if yes, handle according to whether the transaction has an operation rollback mark; otherwise, determine whether the operation is a DML operation, and the worker thread searches the hash table P using the conflict TRXID of the current operation to be executed to determine whether there is a transaction with the conflict TRXID, if yes, wait until the conflict transaction is executed and committed, and then wake up the worker thread; otherwise, the worker thread searches the hash table G using the table information of the current operation to be executed to determine whether there is a transaction with the same table, and the transaction has a commit LSN smaller than the current transaction and a start LSN smaller than the conflict LSN of the current operation, if yes, wait until the conflict transaction is executed and committed, and then wake up the worker thread; otherwise, determine that no conflict is detected, and execute the current operation.

[0034] In a fourth aspect, the application further provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the transaction parallel execution method based on operation rollback marks.

[0035] Compared with the prior art, the application provides a transaction parallel execution method, device and system based on operation rollback marks, which has the following beneficial effects:

[0036] Firstly, the target end data synchronization service receives the operations from the source end, and the operations have sequence and conflict, the operations received first are executed first, and the operations received later are executed later, and the sequence is also the basis for conflict operation detection. The current received operation can be compared with the previously received operation, if there is a transaction operation with the same table and the same KEY, it is indicated that the current operation and the previous transaction have conflict. The current operation must be compared with the previously received operation within a range, the larger the range, the higher the accuracy, but the more resources consumed, which needs to be determined according to the current hardware condition to determine the size of the detection range. In the embodiment of the application, the current received operation is compared with the previously received operation within the set limited range, if the KEY involved in the two operations is the same, it is indicated that the current operation has a conflict transaction within the limited range, and the operation needs to be executed after the conflict operation transaction is executed.

[0037] Secondly, the KEY information of the received operation is saved by using the file system, which can effectively ensure the continuity of the conflict detection of the received operation, the KEY information of the previously received operation can be recovered after the target end data synchronization service fails and restarts, so that the conflict relationship of the newly received operation is continuous before the failure, and the failure does not affect the positioning operation dependency relationship, thereby ensuring that the parallelism of the operation execution is not affected after the system failure is recovered.

[0038] Finally, when the target end data synchronization service receives a rollback or partial rollback operation, it needs to mark the corresponding transaction to indicate that there is an operation that has been rolled back on the transaction, and if the transaction is a rollback transaction, it also needs to be added to the to-be-executed queue and cannot be discarded directly, otherwise the transaction that depends on it will be executed in advance because the dependent transaction cannot be found. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0040] Figure 1 A flowchart of a transaction parallel execution method based on operation rollback marking provided for the embodiment 1 of the present application;

[0041] Figure 2 A flowchart of conflict TRXID and conflict LSN acquisition provided for the embodiment 1 of the present application;

[0042] Figure 3 A flowchart of conflict detection and judgment provided for the embodiment 1 of the present application;

[0043] Figure 4 A flowchart of processing according to whether there is an operation rollback mark provided for the embodiment 1 of the present application;

[0044] Figure 5 A module schematic diagram of a transaction parallel execution system based on operation rollback marking provided for the embodiment 1 of the present application;

[0045] Figure 6 A structure schematic diagram of a transaction parallel execution device based on operation rollback marking provided for the embodiment 3 of the present application. DETAILED DESCRIPTION

[0046] The application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that, for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made. These are within the scope of the application. It should be noted that, if there is no conflict, the features of the embodiments of the application can be combined with each other, and all within the scope of the application. In addition, although the functional modules may be divided in the device schematic diagram, and the steps shown or described in the flowchart may be executed in a different order from the module division in the device or the order in the flowchart in some cases.

[0047] Unless otherwise defined, all technical and scientific terms used in the specification are the same as those commonly understood by those skilled in the art to which the application belongs. The terms used in the specification of the application are only for the purpose of describing the specific embodiments and are not intended to limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as there is no conflict.

[0048] Unless otherwise required by context, the term "comprising" in the specification and claims is interpreted to be open, i.e. "including but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" are intended to mean that the specific feature, structure, material or characteristic associated with that embodiment or example is included in at least one embodiment or example of the disclosure. The illustrative representation of the above terms does not necessarily mean the same embodiment or example. In addition, the specific features, structures, materials or characteristics described can be included in any one or more embodiments or examples in any appropriate manner, i.e. although they may be carried by the embodiments or examples of the above terms due to the order of appearance and location, they are not limited to being carried by one embodiment or example in a combined manner.

[0049] In order to make the purpose, technical scheme and advantages of the present application clearer, the application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application. The application will be described in detail below with reference to the drawings and examples.

[0050] Example 1:

[0051] As shown in Figure 1 The embodiment of the application provides a transaction parallel execution method based on operation rollback marking, which comprises the following steps.

[0052] Step 100: The target data synchronization service creates hash table P, hash table G and several working threads. In this embodiment, LSN is a log sequence number; TRXID is an id identifying a transaction; DML is a data insertion, update and deletion operation; and KEY is a key that can be used to determine a specific row record in a table.

[0053] Step 200: The working thread extracts a transaction from the committed transaction list in the transaction submission order, and registers the transaction ID to the hash table P, and registers the table information involved in the transaction and the starting LSN of the transaction to the hash table G.

[0054] Step 300: It is judged whether the current transaction is a rollback transaction. If yes, the working thread waiting for the current transaction is woken up after all transactions with a submission LSN smaller than the current transaction are submitted; otherwise, the operation extraction step is performed.

[0055] Step 400 (operation extraction step): The working thread extracts an operation of the transaction, judges whether the current operation is a submission operation, and if yes, processes according to whether the transaction has an operation rollback mark; otherwise, judges whether the operation is a DML operation, and the working thread searches for a transaction with a conflict TRXID in the hash table P using the conflict TRXID of the current operation to be executed, and if the transaction exists, waits for the conflict transaction to be executed and submitted before waking up itself; otherwise, the working thread searches for a transaction with the same table in the hash table G using the table information of the current operation to be executed, and if the transaction exists and the submission LSN of the transaction is smaller than the current transaction and the starting LSN of the transaction is smaller than the conflict LSN of the current operation, waits for the conflict transaction to be executed and submitted before waking up itself; otherwise, it is judged that no conflict is detected, and the current operation is executed.

[0056] In the embodiment, when a transaction operation is received, the table ID of the operation is extracted, the primary key or index or custom selected column of the operation is used as KEY, and it is judged whether the same KEY exists in other transaction operations received within a set range before the operation is received, and the operation with a conflict needs to wait for the conflict transaction to be submitted before being executed; the transaction without a conflict can be executed in parallel with the transactions submitted before the operation is received, so that the performance of the database is maximally improved.

[0057] The above method is described in more detail in the embodiment.

[0058] In one embodiment, a synchronization system needs to be deployed in the source database and the target database, the source database synchronization system reads logs from the source database and parses the logs into operations, and the target database synchronization system is responsible for classifying and collecting the synchronization operations from the source database according to transactions, and setting a dependency relationship between each received operation and a conflict transaction received before the operation.

[0059] In one embodiment, the target end data synchronization service needs to initialize a hash table R with a size of N after starting, and initialize the record address in the hash item as empty to store and locate the received operation; and create two files with a size of M to store the KEY in the hash table R, the file name is named in increasing order according to the creation order, and the two files are added to the cache file linked list.

[0060] For the hash table R, the structure of the hash item in the HASH table is 4 bytes of file ID, which is set as K; 4 bytes of file offset, which is set as F, K and F constitute a hash item in the HASH table, as shown in the following table:

[0061] File ID (K) File offset (F)

[0062] It indicates that the first KEY in the hash item is stored in the file with ID K, and the offset of the file is F.

[0063] Referring to Figure 2 In one embodiment, the obtaining of the conflict TRXID and the conflict LSN includes the following steps.

[0064] Step 101: After the target end data synchronization service receives the DML operation of the source end, the table ID in the operation and its primary key or index or custom selection column are extracted as KEY, and the transaction ID in the operation is extracted as TRXID, and KEY and TRXID are combined into a record saved to the offset F of the cache file K, if the record is the first record of the current file, the log LSN corresponding to the operation is marked on the file as the starting LSN of the file, and the LSN is saved in the file name, and the complete cache file name format is “file ID_starting LSN.dat”.

[0065] Since the hash record needs to save the address of the next record, the record saved to the file also needs to reserve the address of the next record, as shown in the following table.

[0066] KEY TRXID Next record file ID (K) Next record file offset (F)

[0067] Step 102: Detecting record X is constructed using the KEY of the current operation, and detecting record X is merged into the hash table R, and conflict detection is performed in the merging process, if there is a conflict KEY, the TRXID value corresponding to the conflict KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict.

[0068] Step 103: Set the TRXID obtained by the conflict detection judgment as the conflict TRXID, and write the starting LSN of the first cache file corresponding to the current hash table R into the current operation as the conflict LSN. Thus, the conflict information of this operation is obtained.

[0069] It should be noted that when the operation is executed, all transactions in which the LSN of the currently executed operation is less than the log LSN of the operation are searched to find whether there is a transaction involving the same table ID, and the LSN of the currently executed operation of the transaction is less than the conflict LSN of the operation (which indicates that the operation less than the conflict LSN in the transaction is not within the scope of the conflict check, and it cannot be determined whether there is a conflict), if there is, the operation needs to wait for the transaction to be executed before it can be executed; if not, continue to search for transactions in which the transaction ID and the conflict TRXID are equal, if there is, the operation needs to wait for the transaction to be executed before it can be executed; if not, the operation can be immediately executed and stored.

[0070] Based on the above steps, the target end data synchronization service receives the next message and saves it to the cache file K in the manner of step 101. The cache file chain table includes two files. When the first cache file is full (the written data exceeds the set size M), the second cache file is switched to. When the second cache file is full, the first cache file in the chain table is discarded first, and then a new cache file is created by incrementing the file ID in the system and added to the chain table, and the record is saved to the new file. When the record is saved to the new file, if it is the first record in the file, the operation LSN corresponding to the record is marked as the starting LSN of the file.

[0071] Reference Figure 3 As shown in one embodiment, the conflict detection judgment in step 102 above specifically includes the following steps.

[0072] Step 201: Hash the KEY of the current detection record X to locate the hash item; judge whether the record address in the hash item is empty, if yes, set the record address in the hash item to the next address space of the current record X, backfill the address of the record X to the hash item, and return 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next conflict detection judgment.

[0073] Step 202: judge the file ID K in the address recorded in the hash item, judge whether the file ID is less than the ID value of the first file in the cache file linked list, if yes, it indicates that the record Y has exceeded the checking range, set the address recorded in the hash item to the next address space of the current record X, backfill the address of the record X into the hash item, return 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next conflict detection judgment.

[0074] Step 203: load the record Y corresponding to the address recorded in the hash item from the file, judge whether the KEY of the record Y is equal to the KEY of the detection record X and the TRXID is not equal to the TRXID of the current operation, if yes, it indicates that the operation exists conflict, remove the record Y from the record linked list in the hash item to shorten the length of the linked list in the hash item, set the address recorded in the hash item to the next address space of the current record X, backfill the address of the record X into the hash item, and return the TRXID stored in the record Y as the conflict TRXID; if not, proceed to the next conflict detection judgment.

[0075] Step 204: sequentially traverse the next record of the record Y and perform conflict detection judgment until the end of the record linked list.

[0076] In an embodiment, when the target-side data synchronization service receives a rollback or partial rollback operation, it locates a transaction according to the transaction ID on the operation, and then marks the transaction with an operation rollback mark, indicating that the transaction has some operations rolled back, if the operation is a rollback operation, then all operations on the transaction do not need to be executed, and are discarded directly, and only the retained transaction information is added to the execution queue. The transaction information of the rolled back transaction cannot be discarded.

[0077] It should be noted that when a transaction rollback is received, since the rolled back transaction operation will not be executed when it is executed into the database, there may be transactions that depend on these rolled back operations behind it. If the rolled back transaction is discarded directly, then the transactions that depend on it will be considered to have no conflict because the dependent transaction cannot be found, resulting in the execution of the subsequent transactions in advance and causing errors. For example, there are three transactions A, B and C that all modify the same row of the same table, but the B transaction is rolled back. When the target-side data synchronization service receives the operation, the operation conflict detection process will form a dependent order relationship of B depending on A and C depending on B. If the transaction B is discarded directly because of the rollback, then C will execute in advance because it cannot find the dependent transaction B, and finally transactions A and C will execute concurrently, resulting in an error.

[0078] In one embodiment, when the system is restarted after a failure, the files are initialized into the cache file linked list of the target data synchronization service according to the size of the cache file ID. It should be noted that when the cache file linked list is initialized, the starting LSN of each cache file needs to be recovered from the cache file name, and then the records saved therein are read in sequence and inserted into the hash table R, and when both cache files are loaded, new operations can be received and conflict checking is performed on the new operations.

[0079] In one embodiment, the target data synchronization service creates H worker threads, executes transactions according to the size of the commit LSN of the received transaction in sequence, and stores the operations of the transaction. It should be noted that the target data synchronization service also needs to create a hash table P to register the transaction information (transaction ID) of the current transaction being executed, and a hash table G to register the table information and the starting LSN of the current transaction.

[0080] In one embodiment, the worker thread extracts a transaction from the committed transaction linked list according to the commit order of the transaction, and registers the transaction ID in the hash table P, and registers the table information involved in the transaction and the starting LSN of the transaction in the hash table G.

[0081] In one embodiment, it is determined whether the current transaction is a rollback transaction. If it is, the worker thread needs to wait until all transactions with a commit LSN smaller than the current transaction are committed, and then the worker thread is awakened, and the registration information of the current transaction is removed from the hash table P and the hash table G, and the execution of the current transaction is completed; otherwise, the next step is performed. It should be noted that since the operations of the rollback transaction are discarded without being executed, the transaction information they depend on is also not clear, so when committing, it needs to wait until all transactions that are committed before the current transaction are committed to ensure the correctness of the transaction dependency order.

[0082] In one embodiment, the worker thread extracts an operation of the transaction, and determines whether the current operation is a commit operation. If it is, it is processed according to whether the transaction has an operation rollback flag; otherwise, the operation is a DML operation, and the next step is performed.

[0083] In one embodiment, the worker thread uses the conflict TRXID of the current operation to be executed to search for whether there is a transaction with the conflict TRXID in the hash table P. If there is, the current execution process is suspended, and the worker thread waits until the conflict transaction is executed and committed, and then the worker thread is awakened; otherwise, the next step is performed. It should be noted that the existence of a transaction with the conflict TRXID in the hash table P indicates that the data involved in the current operation and the transaction with the conflict TRXID are in conflict, and the current operation cannot be executed in parallel with the transaction with the conflict, and needs to wait until the conflict transaction is committed before execution.

[0084] In one embodiment, the working thread uses the current to-be-executed operation table information to search in the hash table G whether there is a transaction with the same table operation, and the transaction commit LSN is less than the current transaction, and the transaction start LSN is less than the conflict LSN of the current operation, if there is, suspend the current execution process, and wait for the conflict transaction to be executed and committed before waking up itself; otherwise, if the conflict detection of the current to-be-executed operation of the working thread is not detected, execute the current operation, and then continue to detect the next operation. It should be noted that this step is mainly to check whether there is a transaction in the current concurrent execution transaction that exceeds the conflict check LSN range, because the to-be-executed operation conflict transaction information is in the effective interval range, that is, the conflict transaction information is only valid within the conflict LSN to the current operation LSN range saved by the operation, and the operation does not perform conflict detection on the operation before the conflict LSN. That is, when there is a transaction with a commit LSN less than the commit LSN of the transaction to which the current to-be-executed operation belongs, and the start LSN of the transaction is less than the conflict LSN of the current to-be-executed operation, it is unknown whether the operation on the transaction and the current to-be-executed operation exist conflict, so the current to-be-executed operation cannot be executed recklessly, and needs to be executed after the transaction is committed.

[0085] Reference Figure 4 As shown in one embodiment, the transaction is processed according to whether there is an operation rollback flag, which specifically includes the following steps.

[0086] Step 301: If there is no operation rollback flag, the current transaction is committed, the working thread waiting for itself is woken up, and the registration information of the current transaction is removed from the hash table P and the hash table G, and the execution of the current transaction is completed. It should be noted that since the dependent transactions of each operation on the transaction have been determined in the conflict detection, the transaction can be directly committed after all operations are executed (that is, the dependent transactions have been committed), without waiting for the transactions submitted before itself to be committed, thereby effectively improving the synchronization performance.

[0087] Step 302: If there is a rollback flag, the working thread waiting for itself is woken up after all transactions with a commit LSN less than the current transaction are committed, and the registration information of the current transaction is removed from the hash table P and the hash table G, and the execution of the current transaction is completed. It should be noted that since there are operations rolled back on the transaction, these operations are not executed, and therefore the dependent transaction information is not clear, so when released, it needs to wait for all transactions submitted before itself to be committed to ensure the correctness of the transaction dependency sequence.

[0088] Based on the above process, the embodiment provides a transaction parallel execution method based on operation rollback marker, mainly for the current received operation, and the operation received before, conflict detection is performed within a set limited range, if the table ID and primary key or index or custom selected column combination KEY involved in the two operations are the same, it indicates that the current operation exists conflict transaction within the limited range, and the operation needs to wait for the conflict operation transaction to be executed first before execution.

[0089] Reference Figure 5 As shown in the figure, the embodiment also provides a transaction parallel execution system based on operation rollback marker, which applies the transaction parallel execution method based on operation rollback marker, and the system includes a hash table and work thread creation module, a transaction extraction module, a rollback judgment module, and an operation extraction and conflict detection module. Wherein: the hash table and work thread creation module is used to create hash table P, hash table G and a plurality of work threads through target end data synchronization service; the transaction extraction module is used to extract a transaction from the committed transaction linked list in transaction submission order through the work thread, and register the transaction ID to the hash table P, and register the table information involved in the transaction and the starting LSN of the transaction to the hash table G; the rollback judgment module is used to judge whether the current transaction is a rollback transaction, if yes, wait for all transactions with a smaller LSN than the current transaction to be submitted before waking up the work thread waiting for the current transaction; otherwise, call the operation extraction and conflict detection module; the operation extraction and conflict detection module is used to extract an operation of the transaction through the work thread, judge whether the current operation is a submission operation, if yes, process according to whether the transaction exists operation rollback marker; otherwise, judge the operation as DML operation, and the work thread searches whether there exists a transaction with conflict TRXID in the hash table P using the conflict TRXID of the current operation to be executed, if yes, wait for the conflict transaction to be executed and submitted before waking up itself; otherwise, the work thread searches whether there exists a transaction with the same table information in the hash table G using the table information of the current operation to be executed, and the transaction submission LSN is smaller than the current transaction, and the starting LSN of the transaction is smaller than the conflict LSN of the current operation, if yes, wait for the conflict transaction to be executed and submitted before waking up itself; otherwise, judge that no conflict is detected, and execute the current operation. The specific functions and principles of the function modules of the system correspond to the specific methods in the method part, and will not be repeated here.

[0090] In summary, the embodiment provides a transaction parallel execution method and system based on operation rollback marking, which has the following advantages. First, the target end data synchronization service receives the operations from the source end, and the operations have sequence and conflict. The operation received first is executed first, and the operation received later is executed later. The sequence is also the basis for conflict operation detection. The current received operation can be compared with the previously received operation. If there is a transaction operation with the same table and the same KEY, it means that the current operation and the previous transaction have conflict. The current operation must be set a range when compared with the previously received operation. The larger the range, the higher the accuracy, but the more resources consumed. The size of the detection range needs to be determined according to the current hardware condition. The current received operation is compared with the previously received operation within the set limited range for conflict detection. If the KEY involved in the two operations is the same, it means that the current operation has conflict transaction within the limited range. The operation needs to be executed after the conflict operation transaction is executed.

[0091] Second, using the file system to save the KEY information of the received operation can effectively ensure the continuity of the conflict detection of the received operation. After the target end data synchronization service fails and restarts, the KEY information of the previously received operation can be recovered. Thus, the conflict relationship of the newly received operation after the failure is continuous before the failure, which ensures that the failure does not affect the dependency relationship of the operation positioning, thereby ensuring that the parallel degree of the operation execution is not affected after the system failure recovery.

[0092] Finally, when the target end data synchronization service receives the rollback or partial rollback operation, the corresponding transaction needs to be marked to indicate that the operation of the transaction is rolled back. If the transaction is a rollback transaction, the transaction also needs to be added to the execution queue, and cannot be discarded directly, otherwise the transaction dependent on it will be executed in advance.

[0093] Embodiment 2

[0094] Based on the operation rollback marking based transaction parallel execution method provided in the above embodiment 1, the embodiment 2 of the present application provides a specific example for description.

[0095] For example, the source database has a table T (ID INT PRIMARY KEY, C1 INT).

[0096] The source end application has three transactions to perform the following operations on the table T in operation sequence.

[0097] TRX1: INSERT INTO T (ID, C1) VALUES (1, 1);

[0098] TRX1: COMMIT;

[0099] TRX2: INSERT INTO T(ID, C1) VALUES(2, 2);

[0100] TRX2: UPDATE T SET C1 = 1 WHERE ID = 1;

[0101] TRX2: ROLLBACK;

[0102] TRX3: UPDATE T SET C1 = 2 WHERE ID = 1;

[0103] TRX3: COMMIT;

[0104] The order of the above operations represents the order of these operations in the log stream, and the resulting log stream will form the following numbered table:

[0105]

[0106]

[0107] After the target data synchronization service is started, a hash table R with a size of N and two cache files with a size of M need to be initialized, and the cache files are numbered as 1 and 2 in sequence.

[0108] The receiving process is as follows:

[0109] 1. The log receiving thread receives the operation with LSN 1, extracts its table ID and primary key as KEY, and assumes that it is 1. The length of the combined TRXID and the space of the next record is 32. First, store its KEY and transaction ID into the first cache file, which is expressed as [file ID (K): file offset (F)]. Its offset in the file is [1:0].

[0110] KEY: 1 TRXID: 1 NEXT: empty

[0111] Since the KEY information is the first operation information of the current cache file, the LSN value of the operation needs to be set to the start LSN of the file. Set the start LSN of the file to 1.

[0112] 2. Insert the KEY into the hash table R, and return the TRXID as 0, which represents no conflict. Write the conflict-free TRXID 0 and the minimum start LSN 1 of the cache file back to the operation.

[0113] 3. The log receiving thread receives the operation with LSN 3, extracts its table ID and primary key as KEY, supposing it is 2, combines TRX ID and the space of next record to be 32, stores its KEY and transaction ID to the first cache file first, and its offset in the file is [1:32].

[0114] KEY: 1 TRXID: 1 NEXT: empty KEY: 2 TRXID: 2 NEXT: empty

[0115] 4. The KEY is inserted into the hash table R, and the returned TRX ID is 0, representing no conflict. The no-conflict TRX ID 0 and the minimum start LSN 1 of the cache file are written back to the operation.

[0116] 5. The log receiving thread receives the operation with LSN 4, extracts its table ID and primary key as KEY, supposing it is 1, combines TRX ID and the space of next record to be 32, stores its KEY and transaction ID to the first cache file first, and its offset in the file is [1:64].

[0117] KEY: 1 TRXID: 1 NEXT: empty KEY: 2 TRXID: 2 NEXT: empty

[0118] KEY: 1 TRXID: 2 NEXT: empty

[0119] 6. The KEY is inserted into the hash table R, and since the KEY 1 operation information already exists in the HASH, the returned conflict TRX ID is 1, representing that the current operation and the transaction ID 1 transaction exist in conflict. The conflict TRX ID 1 and the minimum start LSN 1 of the cache file are written back to the operation.

[0120] 7. The KEY information of the conflict KEY 1 is removed from the HASH table.

[0121] 8. The address of the KEY 1 in the hash table R is set to the next record of the current KEY information, obtaining the following table.

[0122] KEY: 1 TRXID: 1 NEXT: empty KEY: 2 TRXID: 2 NEXT: empty KEY: 1 TRXID: 2 NEXT: 1:0

[0123] 9. The target end data synchronization service is restarted, and the cache file 1 and the cache file 2 are loaded in sequence.

[0124] 10. The KEY information in the cache file 1 is loaded and merged into the hash table R to restore the HASH state before the failure.

[0125] 11. The log receiving thread receives the rollback operation with LSN 5, marks the corresponding TRX 2 of the transaction, indicating that there is a rolled-back operation on the transaction, and then adds TRX 2 to the execution queue.

[0126] 12. The log receiving thread receives the operation with LSN 6, extracts its table ID and primary key as KEY, supposing 3, combines TRXID and the space of next record to be 32, stores its KEY and transaction ID into the first cache file first, and its offset in the file is [1:96].

[0127] KEY: 1 TRXID: 1 NEXT: empty KEY: 2 TRXID: 2 NEXT: empty KEY: 1 TRXID: 2 NEXT: 1:0 KEY: 3 TRXID: 3 NEXT: empty

[0128] 13. The KEY is inserted into the hash table R, and the returned TRXID is 2, which means that the current operation and the transaction with transaction ID 2 exist conflict, and the conflict TRXID 2 and the minimum start LSN 1 of the cache file are written back to the operation.

[0129] After all operation conflict detection is completed, the following detection results are formed.

[0130]

[0131]

[0132] 14. Three worker threads are created, and TRX1, TRX2 and TRX3 are executed in the order of transaction submission.

[0133] 15. The worker thread 1 extracts the TRX1 transaction, registers the transaction information TRXID 1 into the hash table P, and registers the table information T associated with the start LSN 1 of the transaction into the hash table G.

[0134] 16. The worker thread 2 extracts the TRX2 transaction, registers the transaction information TRXID 2 into the hash table P, and registers the table information T associated with the start LSN 3 of the transaction into the hash table G.

[0135] 17. The worker thread 3 extracts the TRX3 transaction, registers the transaction information TRXID 3 into the hash table P, and registers the table information T associated with the start LSN 6 of the transaction into the hash table G.

[0136] 18. The worker thread 1 extracts the insert operation with LSN 1 of TRX1, and obtains the conflict TRXID and the conflict LSN on the operation to perform conflict detection.

[0137] (1) The operation conflict TRXID is 0, which means no conflict.

[0138] (2) The table information T and the conflict LSN 1 on the operation are obtained, and the conflict transaction whose transaction operation involves table T and whose submission LSN is less than 2 and whose minimum start LSN is less than 1 is searched in the hash table G, and no conflict transaction is searched, which can be directly executed.

[0139] 19. Worker thread 2 fetches TRX2, which is marked as a transaction that has a rollback operation and is also a rollback transaction, so it does not need to execute, but needs to wait for TRX1, which has a commit LSN smaller than it, to commit before removing its transaction information from the registered hash table P and hash table G, and then wakes up the transaction that waits for it to complete execution.

[0140] 20. Worker thread 3 fetches the update operation of TRX3 with LSN 6, obtains the conflict TRXID and conflict LSN of the operation for conflict detection.

[0141] (1) The operation conflict TRXID is 2, indicating that it has a conflict with TRX2, suspending the execution of the current worker thread and waiting for TRX2 to complete execution before waking itself up.

[0142] From this, it can be seen that TRX3 depends on TRX2, but TRX2 itself is rolled back and will not perform any operation. If TRX2 is discarded directly without execution, then the update of TRX3 will not find a conflict transaction and will execute in advance with the insert operation of TRX1, but the two transaction operations are on the same row. If the update operation of TRX3 is executed before the insert operation of TRX1, an error will be executed.

[0143] 21. After worker thread 1 executes the commit operation of TRX1 and deletes the current transaction information from hash table P and hash table G, it wakes up the execution of TRX2 transaction on worker thread 2.

[0144] 22. After the transaction TRX2 of worker thread 2 is woken up, it does not need to commit any operation because it is a rollback transaction, and only needs to delete the current transaction information from hash table P and hash table G before waking up the execution of the transaction TRX3 that waits for itself.

[0145] 23. After worker thread 3 is woken up, it executes the update operation of TRX3, and then executes the commit operation and deletes the current transaction information from hash table P and hash table G to complete the execution of the transaction.

[0146] From the above flow, it can be seen that the transaction TRX1 is the first received operation, and then the transaction TRX2, TRX2 finds that its operation depends on the transaction TRX1 in the operation conflict detection, although TRX2 receives the rollback operation last, which means that the transaction will not be executed, but the operation of TRX2 has covered the operation dependency in the HASH table when the detection is performed, so that the operation on the transaction TRX3 received last depends on TRX2 instead of TRX1 in the conflict detection, and therefore, the TRX2 is marked with the operation being rolled back and added to the execution chain table. The cascading relationship of TRX3 depending on TRX2 and TRX2 depending on TRX1 in the transaction dependency relationship is ensured not to be interrupted. Although TRX2 is a rollback transaction, it also needs to be registered when executed to ensure the continuity of the dependency relationship, and since there is no operation to be executed in the TRX2 transaction, the transaction information of TRX2 has no dependency when executed, and the release of TRX2 needs to wait until all transactions with a commit LSN smaller than that of TRX2 are completed.

[0147] Embodiment 3

[0148] It should be noted that for a rollback transaction, since the rollback transaction will not be executed, there are three ways to handle the operation of the rollback transaction in the HASH table R.

[0149] The first way is not to back off the influence of the operation of the rollback transaction on the operation dependency in the HASH table R, and no conflict TRXID of the operation dependency of the rollback transaction operation is collected, and the entire operation of the rollback transaction is directly released, the rollback transaction is also added to the execution chain table for execution to ensure the continuity of the operation dependency. When this scheme is adopted, the operation of the rollback transaction is directly released as a whole, which can improve the processing performance of the rollback transaction, but the operation depending on the rollback transaction needs to wait until the release of the rollback transaction in the execution chain table before being executed, and since the operation dependency conflict transaction is not collected when the operation is released, the release of the rollback transaction needs to wait until all transactions with a commit LSN or a rollback LSN smaller than the rollback LSN of the transaction are executed, which affects the synchronization efficiency. This way corresponds to the case that the target end data synchronization service receives a rollback transaction, and all operations on the transaction do not need to be executed, and are directly discarded, and only the transaction information is retained and added to the execution queue. For example, the target end data synchronization service receives a rollback or partial rollback operation, needs to locate the transaction according to the transaction ID on the operation, and then marks the transaction, indicating that the operation of the transaction is rolled back, if the operation is a rollback operation, then all operations on the transaction do not need to be executed, and are directly discarded, and only the transaction information is retained and added to the execution queue, and the transaction information of the rollback transaction cannot be discarded. The description of the rollback part in the embodiment 1 of the present application is an example description under this way.

[0150] The second solution: the operation of the rollback transaction does not affect the operation dependency in the hash table R, but collects the conflict TRXID of the operation dependency of the rollback transaction, for example, records it to the conflict TRXID dependency list L, the rollback transaction is also added to the execution list for execution, to ensure the continuity of the operation dependency. When this solution is used, the operation of the rollback transaction needs to collect the conflict TRXID in the operation before release, which reduces the processing performance of the rollback transaction, but improves the release performance of the rollback transaction during execution. The operation dependent on the rollback transaction needs to wait until the rollback transaction is released in the execution list before execution, and the release of the rollback transaction needs to wait until the conflict transaction dependent on the operation is executed, without waiting until all the transactions with the commit LSN or the rollback LSN smaller than the rollback LSN of the transaction are executed. Compared with the first solution, this can partially improve the synchronization efficiency. In this solution, when the target end data synchronization service receives the rollback or partial rollback operation, the conflict TRXID saved in the operation to be rolled back is collected, and for the rollback operation, the corresponding transaction cannot be discarded, but is added to the execution transaction queue.

[0151] The third solution: the operation of the rollback transaction affects the operation dependency in the hash table R, and then the rollback transaction is not dependent on other operations, and the rollback transaction does not need to be added to the execution list. When this solution is used, the transaction needs to roll back the operation dependency in the hash table R after receiving the rollback operation. Since the transaction size is uncontrollable, when a large transaction is rolled back, the rollback action will affect the receiving progress of the operation, thereby affecting the synchronization efficiency. In this solution, when the target end data synchronization service receives the rollback or partial rollback operation, the conflict TRXID saved in the operation to be rolled back is written back to the hash table R.

[0152] The above three solutions for processing the rollback transaction have advantages and disadvantages. Large-scale transaction rollback is suitable for the first solution, medium-scale transaction rollback is suitable for the second solution, and small-scale transaction rollback is suitable for the third solution. Therefore, two parameters W1 and W2 are set, W1 represents the minimum number of DML operations in a large-scale transaction, and W2 represents the minimum number of DML operations in a medium-scale transaction. Based on this, the target end data synchronization service needs to classify and manage the received DML operations by transaction after receiving the DML operation from the source end, and count the size S of the received DML operation in each transaction (S is the number of received DML operations in a single transaction). When the target end data synchronization service receives the rollback operation, the corresponding action is performed according to the size S of the DML operation in the transaction.

[0153] When the size S of the rollback transaction is greater than or equal to W1, it corresponds to the large-scale transaction rollback in the first scheme. In this case, the operation of the rollback transaction is not rolled back to affect the operation dependency in the hash table R, and the conflict TRXID of the operation dependency of the rollback transaction is not collected, and the entire operation of the rollback transaction is directly released. The rollback transaction is added to the execution list to ensure the continuity of the operation dependency. The description of the rollback part in Embodiment 1 of the present application is the same as the description of the instance when the size S of the rollback transaction is greater than or equal to W1.

[0154] When the size S of the rollback transaction is greater than or equal to W2 and less than W1, it corresponds to the medium-scale transaction rollback in the second scheme. In this case, the operation of the rollback transaction is not rolled back to affect the operation dependency in the hash table R, but the conflict TRXID of the operation dependency of the rollback transaction is collected, which can be recorded in the conflict TRXID dependency list L. The rollback transaction is added to the execution list to ensure the continuity of the operation dependency.

[0155] When the size S of the rollback transaction is less than W2, it corresponds to the small-scale transaction rollback in the third scheme. In this case, the operation of the rollback transaction is rolled back to affect the operation dependency in the hash table R, and the rollback transaction does not need to be added to the execution list and is directly released.

[0156] Further, for the partial rollback operation, due to the particularity of the partial rollback, the partial rollback itself needs to traverse the rollback operation, so that the process of collecting the conflict TRXID on the operation does not significantly reduce the performance of processing the partial rollback. Based on this, when the target end data synchronization service receives the partial rollback operation, the total number of rows S1 of the rollback operation needs to be counted, and the following processing is performed according to the size of S1: when the number of rows S1 of the rollback operation is less than W2, the operation of the rollback operation is rolled back to affect the operation dependency in the hash table R, that is, the conflict TRXID saved in the operation to be rolled back is written back to the hash table R. When the number of rows S1 of the rollback operation is greater than or equal to W2, the operation of the rollback transaction is not rolled back to affect the operation dependency in the hash table R, that is, the conflict TRXID saved in the operation to be rolled back is not written back to the hash table R, but the conflict TRXID of the operation dependency of the rollback transaction is collected, and the conflict TRXID can be recorded in the conflict TRXID dependency list L.

[0157] Based on the above settings, after the target end data synchronization service creates a worker thread, the worker thread extracts a transaction from the committed transaction list in the order of the transaction submission, and judges whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to judge whether the rollback transaction exists in the conflict TRXID list L.

[0158] If the rollback transaction does not exist the conflict TRXID chain table L, all the transactions with the LSN smaller than the current transaction need to be waited for completion before the work thread of the current transaction is woken up, and then the registration information of the current transaction is removed from the hash table P and the hash table G to complete the execution of the current transaction. Since the operations of the rollback transaction are discarded without being executed, the transaction information they depend on is not clear, so the correctness of the transaction dependency order can be ensured only after all the transactions that are committed before the current transaction are committed.

[0159] If the rollback transaction exists the conflict TRXID chain table L, all the conflict transactions in the conflict transaction chain table L of the transaction need to be waited for completion before the work thread of the current transaction is woken up, and then the registration information of the current transaction is removed from the hash table P and the hash table G to complete the execution of the current transaction. Specifically, a conflict TRXID is extracted from the conflict transaction chain table L of the current transaction, and it is checked whether it exists in the hash table P. If it exists, the current transaction needs to be waited for completion before the current transaction is woken up. After the current transaction is woken up, the next conflict TRXID is extracted from the conflict transaction chain table L for the same check, until all the conflict transactions in the conflict transaction chain table L are completed. Since the operations of the rollback transaction are discarded without being executed, the conflict TRXID of the operation is collected when the rollback operation is discarded, so the transaction information they depend on is clear, and the correctness of the transaction dependency order can be ensured only after the conflict transactions are committed.

[0160] In summary, since the rollback transaction will not be executed, three schemes are generated when the conflict is detected, each of which has its own advantages. Suitable values need to be set to exert the advantages of each scheme to improve the overall synchronization performance. In addition, for the partial rollback operation of the committed transaction, the conflict TRXID referenced by the operation is collected when the rollback operation is performed, which can effectively reduce the performance loss caused by the write-back of the conflict detection hash table R of the partial rollback operation to improve the overall synchronization performance. The present embodiment is targeted at the rollback transaction and the partial rollback operation according to the agreed operation scale, and the balance point between the operation rollback and the rollback transaction execution is found to improve the overall synchronization performance.

[0161] In some embodiments, the KEY in the hash table R can be stored by a file or by a memory slice. Specifically, when the KEY is stored by a file, the performance of storage is low, which affects the efficiency of conflict detection, but after a data synchronization failure, the KEY values in the storage file can be used to restore the state of the hash table R before the failure, so that the conflict detection of the received operation after the failure is continuous with the state before the failure, and the synchronization performance does not drop sharply after the failure. When the KEY is stored by a memory slice, the performance of conflict detection is greatly improved compared with the file storage scheme, but the data of the previous conflict detection is lost after the failure, and the detection can only start from the current received LSN, which causes the conflict detection after the failure to be discontinuous, and the synchronization performance drops sharply at the failure position. When the storage performance is high (for example, using an SSD fixed storage), the performance requirement of conflict detection can be met, and the efficiency of operation receiving can be balanced with the synchronization execution, so that the KEY in the hash table R is stored by a file to effectively avoid the loss of synchronization performance caused by the failure. When the storage performance affects the efficiency of conflict detection, the bottleneck of synchronization exists in operation receiving, and the KEY in the hash table R is stored by a memory slice to effectively improve the performance of operation receiving, thereby improving the overall synchronization performance. Embodiment 1 of the present application is an example of storing the KEY in the hash table R by a file.

[0162] In some embodiments, the MD5 value can be calculated at the source end or the target end as the KEY of the operation. The source data synchronization service can more accurately calculate the MD5 value of the conflict when detecting the conflict for each operation, and the MD5 value calculated by the target data synchronization service will have errors if the primary key structure or the column order of the target database table is inconsistent with that of the source end. Therefore, the MD5 value is generally calculated at the source end as the KEY of the operation, and the target data synchronization service directly extracts the MD5 value calculated by the source end as the KEY after receiving the DML operation of the source end.

[0163] In some embodiments, when calculating the KEY of an operation, since the column order in the operation parsed from the log is not fixed, the columns in the operation are sorted by column ID first, and then the KEY is calculated according to the fixed order of the primary key or the unique index combination column. After the source data synchronization service captures a DML log operation, the table ID in the log is obtained, and the primary key information and the unique index of the table are obtained in the source database through the table ID. After the source data synchronization service parses the current DML log operation, it is judged whether the table has a primary key or a unique index. If there is a primary key or a unique index, the table ID and the column value of the primary key or the unique index are combined to calculate an MD5 value as the KEY of the operation, which is stored on the current DML log operation and sent to the target synchronization service. It should be noted that when the table has both a primary key and a unique index, the table ID and the primary key are combined to calculate the MD5 value as the KEY; when the table has no primary key but has multiple unique indexes, the table ID and any unique index are combined to calculate the MD5 value. If there is no primary key or unique index, the source data synchronization service combines the table ID and the ROWID value or the custom selected column of the row data to calculate an MD5 value as the KEY of the operation after parsing the current DML log operation, which is stored on the current DML log operation and sent to the target synchronization service. It should be noted that when calculating the KEY of the table without a primary key or a unique constraint, if it is known that all columns of the table have no duplicate values, the ROWID of the table can be directly used to calculate the KEY, which can effectively reduce the cost of calculating the KEY; if the table has duplicate rows, the custom selected column method is used to calculate the KEY, and the distribution of column values can be used to select which columns or all column values to calculate the KEY, so that the target synchronization service can prevent the concurrent execution of the synchronization transaction from operating the duplicate value rows simultaneously to cause deadlock.

[0164] Embodiment 4

[0165] On the basis of the transaction parallel execution method based on operation rollback marking provided in Embodiment 1, the application further provides a transaction parallel execution device based on operation rollback marking, which can be used to implement the above method and system, as shown in Figure 6 The transaction parallel execution device based on operation rollback marking provided in the embodiment of the application includes one or more processors 21 and a memory 22. In the embodiment, the processor 21 is taken as an example. Figure 6

[0166] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 6 In the embodiment, the connection through the bus is taken as an example.

[0167] ​The memory 22, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the transaction parallel execution method based on operation rollback marking in Embodiment 1. The processor 21 performs various functional applications and data processing of the transaction parallel execution device based on operation rollback marking by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, implements the transaction parallel execution method based on operation rollback marking in Embodiment 1.

[0168] The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 can optionally include a memory disposed remotely with respect to the processor 21, and these remote memories can be connected to the processor 21 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The program instructions / modules are stored in the memory 22, and when executed by one or more processors 21, perform the transaction parallel execution method based on operation rollback marking in Embodiment 1 described above, for example, perform the above-described Figures 1-4 each step shown.

[0170] The above product can perform the method provided in the embodiments of the present application, and has the corresponding functional modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can refer to the method provided in the embodiments of the present application.

[0171] It should be noted that the device embodiments described above are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0173] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; under the idea of the present application, the technical features of the above examples or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the present application as described above. In order to be brief, they are not provided in detail; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for parallel execution of transactions based on operation rollback flags, characterized in that, The target data synchronization service includes: A hash table P, a hash table G, and a plurality of work threads; A work thread extracts a transaction from a submitted transaction list in transaction submission order, and registers the transaction ID to the hash table P, and registers the table information involved in the transaction and the starting LSN of the transaction to the hash table G; It is judged whether the current transaction is a rollback transaction, if yes, the work thread waiting for the current transaction is awakened after all transactions with a submission LSN smaller than the submission LSN of the current transaction are submitted; Otherwise, the operation extraction step is performed; The operation extraction step: the work thread extracts an operation of the transaction, judges whether the current operation is a submission operation, if yes, the operation is processed according to whether the transaction has an operation rollback mark; Otherwise, if the operation is a DML operation, the work thread searches for a transaction with a conflict TRXID in the hash table P using the conflict TRXID of the current operation to be executed, if the transaction exists, the work thread is awakened after the conflict transaction is executed and submitted; Otherwise, the work thread searches for a transaction with the same table information in the hash table G using the table information of the current operation to be executed, and the transaction has a submission LSN smaller than the submission LSN of the current transaction, and the starting LSN of the transaction is smaller than the conflict LSN of the current operation, if the transaction exists, the work thread is awakened after the conflict transaction is executed and submitted; Otherwise, it is judged that no conflict is detected, and the current operation is executed.

2. The method for transaction parallel execution based on operation rollback marking according to claim 1, wherein, The target data synchronization service initializes a hash table R when started, and initializes the record address in the hash table R to empty, which is used to store and locate the received operation; the target data synchronization service creates two cache files to store the KEY in the hash table R, and the file names of each cache file are named in ascending order according to the creation order, and the two cache files are added to the cache file list.

3. The method for transaction parallel execution based on operation rollback marking according to claim 2, wherein, The acquisition of the conflict TRXID and the conflict LSN includes: After the target data synchronization service receives the DML operation of the source end, the table ID and its primary key or index or custom selected column in the operation are extracted as the KEY, and the transaction ID in the operation is extracted as the TRXID, the KEY and the TRXID are combined into a record saved to the current cache file, if the record is the first record of the current cache file, the log LSN corresponding to the operation is marked on the cache file as the starting LSN of the cache file; A detection record X is constructed using the KEY of the current operation, the detection record X is merged into the hash table R, and conflict detection is performed in the merging process, if there is a conflict KEY, the TRXID value corresponding to the conflict KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict; The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first cache file corresponding to the current hash table R is written into the current operation as the conflict LSN.

4. The method for transaction parallel execution based on operation rollback marking according to claim 3, wherein, The conflict detection includes: Hashing the KEY of the current detection record X, locating to the hash item through the hash value; judging whether the record address in the hash item is empty, if yes, setting the record address in the hash item to the next address space of the current detection record X, backfilling the address of the detection record X into the hash item, and returning 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceeding to the next conflict detection judgment; judging whether the file ID in the record address in the hash item is less than the ID value of the first file in the cache file linked list, if yes, indicating that the record Y has exceeded the checking range, setting the record address in the hash item to the next address space of the current detection record X, backfilling the address of the detection record X into the hash item, and returning 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceeding to the next conflict detection judgment; loading the corresponding record Y from the file according to the record address in the hash item, judging whether the KEY of the record Y is equal to the KEY of the current detection record X and the TRXID of the record Y is not equal to the TRXID of the current operation, if yes, indicating that the operation exists conflict, removing the record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, setting the record address in the hash item to the next address space of the current detection record X, backfilling the address of the current detection record X into the hash item, and returning the TRXID stored in the record Y as the conflict TRXID; if not, proceeding to the next conflict detection judgment; sequentially traversing the next record of the record Y and performing conflict detection judgment until the end of the record linked list.

5. The method for transaction parallel execution based on operation rollback marking according to claim 2, wherein, The cache file linked list includes two cache files, when the first cache file is full, switching to the second cache file, when the second cache file is full, discarding the first cache file in the linked list, and then creating a new cache file with an incremental file ID and adding it to the cache file linked list.

6. The method for transaction parallel execution based on operation rollback marking according to claim 2, wherein, When the target end data synchronization service receives a rollback or partial rollback operation, the transaction ID on the operation is located to the transaction, and then the operation rollback flag is marked on the transaction, indicating that the transaction has partial operations rolled back, if the operation is a rollback operation, the whole operation on the transaction does not need to be executed, and is discarded directly, and only the reserved transaction information is added to the execution queue.

7. The method for parallel execution of transactions based on operation rollback markers according to any of claims 1-6, characterized in that, The processing according to whether the operation rollback flag exists specifically includes: if there is no operation rollback flag, submitting the current transaction, then waking up the waiting work thread, and then removing the registration information of the current transaction from the hash table P and the hash table G, completing the execution of the current transaction; if there is a rollback flag, waiting for all transactions with a submission LSN less than the submission LSN of the current transaction to be submitted, then waking up the waiting work thread, and then removing the registration information of the current transaction from the hash table P and the hash table G, completing the execution of the current transaction.

8. A transaction parallel execution apparatus based on operation rollback flags, characterized in that, The device comprises at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the transaction parallel execution method based on operation rollback mark as claimed in any one of claims 1-7.

9. A transaction parallel execution system based on operation rollback marker, applying the transaction parallel execution method based on operation rollback marker as claimed in any one of claims 1-7, characterized in that, The system comprises a hash table and work thread creation module, a transaction extraction module, a rollback judgment module and an operation extraction and conflict detection module, wherein: The hash table and work thread creation module is used to create hash table P, hash table G and a plurality of work threads through a target end data synchronization service; The transaction extraction module is used to extract a transaction from the committed transaction linked list in the transaction submission order through the work thread, and register the transaction ID to the hash table P, and register the table information involved in the transaction and the starting LSN of the transaction to the hash table G; The rollback judgment module is used to judge whether the current transaction is a rollback transaction, if yes, the work thread waiting for the current transaction is woken up after all transactions with a submission LSN smaller than the submission LSN of the current transaction are submitted; otherwise, the operation extraction and conflict detection module is called; The operation extraction and conflict detection module is used to extract an operation of the transaction through the work thread, judge whether the current operation is a submission operation, if yes, handle according to whether the transaction exists operation rollback mark; otherwise, judge the operation as a DML operation, the work thread searches whether there exists a transaction with the conflict TRXID in the hash table P using the conflict TRXID of the current operation to be executed, if yes, wait for the conflict transaction to be executed and submitted, and then wake up itself; otherwise, the work thread searches whether there exists a transaction with the same table information in the hash table G using the table information of the current operation to be executed, and the submission LSN of the transaction is smaller than the submission LSN of the current transaction, and the starting LSN of the transaction is smaller than the conflict LSN of the current operation, if yes, wait for the conflict transaction to be executed and submitted, and then wake up itself; otherwise, judge that no conflict is detected, and execute the current operation.

10. A non-transitory computer storage medium, comprising, The computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the transaction parallel execution method based on operation rollback mark as claimed in any one of claims 1-7.

Images