A method, device and system for setting transaction operation dependency based on log order

Abstract

The application relates to a method, device and system for setting transaction operation dependence based on log sequence. The method mainly comprises the following steps: after a target end data synchronization service receives a DML operation of a source end, extracting an MD5 value calculated by the source end in the operation as a KEY, and extracting a transaction ID in the operation as a TRXID; constructing a detection record X by using the KEY of the current operation, merging the detection record X into a hash table R, and performing conflict detection and judgment in the merging process; if there is a conflict KEY, returning a TRXID value corresponding to the conflict KEY; otherwise, returning 0 as the corresponding TRXID, indicating no conflict; setting the TRXID obtained through the conflict detection and judgment as a conflict TRXID, and writing a start LSN of a first memory piece in a memory linked list corresponding to the current hash table R into the current operation and setting the start LSN as a conflict LSN. The application can determine the dependence between transaction operations, can clearly indicate which transactions need to be waited for during synchronization operation, and can maximize the parallel degree of synchronization.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method, apparatus and system for setting transaction operation dependencies based on log order. Background Technology

[0002] In a real-time database synchronization system based on log parsing architecture (supporting heterogeneous databases), the source-side data synchronization service is responsible for capturing the operation logs of the source database, parsing the logs to restore the corresponding database operations, and then sending them to the target-side data synchronization service. The target-side data synchronization service is responsible for executing these operations in the target database, thereby completing the data synchronization process.

[0003] When a target-side data synchronization service performs synchronization operations on a target database, it often needs to execute according to the transaction commit order. This is because if the previous transaction hasn't completed synchronization, the next transaction's operation cannot be executed concurrently with previous transactions and its data entry cannot proceed until all transactions preceding the current operation have completed their commits. This significantly impacts the concurrency of transaction data entry. Currently, an effective method is to determine whether parallel execution is possible by comparing the log LSN of the current operation with the LSN of the commit operations of previously committed transactions. If the LSN of the currently executing transaction is smaller than the LSN of another currently executing transaction, then this operation can be executed concurrently with that transaction. This strategy helps improve concurrency. While this method of parallel execution based on transaction commit order improves performance, the granularity of conflict detection is too large and not fine enough, resulting in low parallelism. If the dependencies between transaction operations could be determined, and it could be clearly identified which transactions need to be waited for during synchronization, the parallelism of synchronization could be maximized. However, current technology has not yet achieved this.

[0004] In view of this, how to overcome the shortcomings of existing technologies and how to obtain the conflict relationship between each operation of the transaction to be synchronized and previous transactions have become important technical problems that the industry urgently needs to solve. Summary of the Invention

[0005] Addressing the shortcomings or improvement needs of existing technologies—specifically, how to obtain the conflict relationships between each operation of a transaction to be synchronized and previous transactions, and determine the dependencies between transaction operations—this invention provides a method, apparatus, and system for setting transaction operation dependencies based on log order. When the target-end data synchronization service receives synchronization transaction operations, it receives them according to the generation order of the source database logs. Therefore, the order of each received operation forms the basis for detecting potential conflicts. Based on the order of operation reception, this invention extracts the key from each received operation and checks whether this key, within a set range, is the same as a key from other previously received transactions. If so, the execution of this operation must wait for the conflicting transaction to commit before it can begin; otherwise, it can be executed concurrently with transactions that committed earlier, maximizing the parallelism of transaction execution and effectively improving synchronization performance.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for setting transaction operation dependencies based on log order, including:

[0008] After the target data synchronization service receives the DML operation from the source, it extracts the MD5 value calculated by the source as the KEY and the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice.

[0009] Construct a detection record X using the KEY of the current operation, merge the detection record X into the hash table R, and perform a collision detection during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no collision.

[0010] Set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN.

[0011] In some embodiments, the calculation of the MD5 value includes generating an MD5 value by combining the table ID and primary key, or the table ID and unique index information, or the table ID and a custom selection column based on the DML operation.

[0012] In some embodiments, for MD5 value calculation of DML operations, when the table has both a primary key and a unique index, the table ID and the primary key are selected together to calculate the MD5 value as the KEY; when the table has no primary key but has multiple unique indexes, the table ID and any one of the unique indexes are selected together to calculate the MD5 value; when there is no primary key and no unique index, the table ID and the ROWID value or a custom selection column of the data in that row are combined to calculate an MD5 value.

[0013] In some embodiments, when the target data synchronization service starts, it initializes a hash table R and initializes the record addresses in the hash entries of hash table R to be empty, which is used to store and locate the received operations; the target data synchronization service creates two memory slices to store the KEY in hash table R, and the memory of each memory slice is incremented by the version number in the order of creation, and the two memory slices are added to the memory linked list.

[0014] In some embodiments, the memory list includes two memory blocks. When the first memory block is full, the system switches to the second memory block. When the second memory block is full, the first memory block in the list is first removed, and then a new memory block is created and added to the memory list using an incrementing version number.

[0015] In some embodiments, the conflict detection determination includes:

[0016] Calculate the hash value of the KEY of the current detection record X, and locate the hash entry using the hash value; determine whether the record address in the hash entry is empty. If so, set the record address in the hash entry to the next address space of the current detection record X, fill the hash entry with the address of the detection record X, and return 0 as the conflict TRXID, indicating that no conflict was detected; otherwise, proceed to the next conflict detection and judgment step.

[0017] Determine if the memory slice version number K at the address recorded in the hash entry is less than the version number K of the first memory slice in the memory linked list. If so, it indicates that the record Y has exceeded the inspection range. Set the record address in the hash entry to the next address space of the current detection record X, fill the hash entry with the address of the detection record X, and return 0 as the conflict TRXID, indicating that no conflict was detected. If not, proceed to the next conflict detection and judgment step.

[0018] Based on the address recorded in the hash entry, load the corresponding record Y from the memory slice. Determine if the KEY of record Y is equal to the KEY of the currently detected record X and if the TRXID of record Y is not equal to the TRXID of the current operation. If so, it indicates a conflict. Remove record Y from the record list of the hash entry to shorten the length of the linked list in the hash entry. Set the address of the record in the hash entry to the next address space of the currently detected record X. Fill the address of the currently detected record X back into the hash entry and return it using the TRXID stored in record Y as the conflict TRXID. If not, proceed to the next conflict detection step.

[0019] Iterate through the next record of record Y and perform conflict detection and judgment until the record linked list ends.

[0020] In some embodiments, when the target data synchronization service receives a rollback or partial rollback operation, it writes the conflict TRXIDs saved in the rolled-back operation back to the hash table R used for conflict detection; specifically:

[0021] Locate the transaction that needs to be rolled back by the transaction ID of the rollback or partial rollback operation, and extract the number of rows contained in the transaction in the rollback operation or the number of rows that need to be rolled back in the partial rollback operation.

[0022] Extract rollback operations from the tail of the transaction operation list backwards, and extract the KEY value, conflicting TRXID, and conflicting LSN saved in the operation;

[0023] The conflicting LSN of the operation is compared with the starting LSN of the first memory slice. If it is less than the starting LSN, the rollback operation is terminated and the write-back to hash table R is stopped. Otherwise, the conflicting operation record with the same key is searched in hash table R. If no record is found, the rollback operation is terminated and the write-back to hash table R is stopped. Otherwise, the conflicting TRXID is written back to the record in hash table R, and the previous operation to be rolled back is extracted and the same write-back is performed until the specified number of rows is reached.

[0024] Secondly, the present invention also provides an apparatus for setting transaction operation dependencies based on log order, the apparatus comprising:

[0025] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method of setting transaction operation dependencies based on log order as described in the first aspect.

[0026] Thirdly, the present invention also provides a system for setting transaction operation dependencies based on log order, using the method for setting transaction operation dependencies based on log order as described in the first aspect. The system includes a key acquisition module, a conflict detection module, and a conflict setting module, wherein:

[0027] The KEY acquisition module is used to extract the MD5 value calculated by the source end as the KEY after the target end data synchronization service receives the DML operation from the source end, and then extract the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record of the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice.

[0028] The conflict detection module constructs a detection record X using the KEY of the current operation, merges the detection record X into the hash table R, and performs conflict detection during the merging process. If there is a conflicting KEY, it returns the TRXID value corresponding to the conflicting KEY; otherwise, it returns 0 as the corresponding TRXID, indicating that there is no conflict.

[0029] The conflict setting module is used to set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN.

[0030] Fourthly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method for setting transaction operation dependencies based on log order as described in the first aspect.

[0031] Compared with the prior art, the present invention provides a method, apparatus and system for setting transaction operation dependencies based on log order, the advantages of which are:

[0032] First, conflict detection is performed on operations within a fixed LSN interval, and the detection results and the LSN interval are saved to the corresponding operation. Based on this, the dependencies between transaction operations can be determined, and it can be clearly identified which transactions need to be waited for during synchronization operations, thereby maximizing the parallelism of synchronization.

[0033] Secondly, when a worker thread executes an operation, it checks the currently concurrently executing transactions for conflicting transactions based on the conflicting transaction information stored in the operation. If a conflicting transaction exists, the operation cannot proceed. Otherwise, it checks the conflicting LSNs stored in the operation to see if there is a transaction involving the same table as the current operation, and if the transaction's initial LSN is less than the conflicting LSN and its commit LSN is less than the commit LSN of the current operation's transaction. If such a transaction exists, the operation cannot proceed until these transactions have completed. Otherwise, the operation can proceed immediately without waiting for transactions committed earlier to complete, effectively increasing the concurrency of operation execution and improving synchronization performance.

[0034] Finally, when a transaction is committed, since all operations of the transaction have undergone conflict detection during execution, transactions that conflict with the current transaction have certainly already been committed. The current transaction does not need to wait for transactions that committed before it to commit before it can commit synchronously, which further improves synchronization performance. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0036] Figure 1 This is a flowchart of a method for setting transaction operation dependencies based on log order, provided in Embodiment 1 of the present invention.

[0037] Figure 2 This is a flowchart of the conflict detection and judgment process provided in Embodiment 1 of the present invention;

[0038] Figure 3 This is a flowchart of the receive rollback operation provided in Embodiment 1 of the present invention;

[0039] Figure 4 This is a schematic diagram of a system for setting transaction operation dependencies based on log order, as provided in Embodiment 1 of the present invention.

[0040] Figure 5 This is a schematic diagram of a device structure for setting transaction operation dependencies based on log order, as provided in Embodiment 3 of the present invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention. It should be noted that, unless otherwise specified, the various features in the embodiments of the present invention can be combined with each other, all within the protection scope of this application. Furthermore, although functional modules may be divided in the device schematic diagram, and a logical order may be shown in the flowchart, in some cases, the steps shown or described may be executed differently from the module division in the device or the order in the flowchart.

[0042] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of the invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] Example 1:

[0046] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for setting transaction operation dependencies based on log order, the method including the following steps.

[0047] Step 100: After receiving the DML operation from the source end, the target end data synchronization service extracts the MD5 value calculated by the source end as the KEY, and then extracts the transaction ID from the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If this record is the first record in the current memory slice, the corresponding log LSN is marked on the memory slice as the starting LSN of the memory slice. In this embodiment, LSN is the log sequence number; TRXID is the ID that identifies a transaction; DML refers to data insertion, update, and deletion operations; MD5 is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value to ensure the integrity and consistency of information transmission; KEY is the key used to identify a specific row in the table; ROWID is a pseudo-column used to uniquely identify rows in the table.

[0048] Step 200: Construct a detection record X using the KEY of the current operation, merge the detection record X into the hash table R, and perform a collision detection judgment during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no collision.

[0049] Step 300: Set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN.

[0050] The target data synchronization service of this invention marks all received operations with transaction information that the operation depends on for conflict and the conflict LSN value for dependency detection. When the operation is executed, conflict detection is performed based on the conflict transaction information attached to the operation. If no conflicting transaction is detected, and no transaction with a commit LSN less than the commit LSN of the transaction to which the current operation belongs or a transaction start LSN less than the conflict LSN of the current operation is detected in the running transaction, then the operation can be executed in parallel with the operation that was committed earlier than itself, without waiting for the transaction that was committed earlier to complete synchronization, thereby improving the synchronization performance.

[0051] The following embodiment will provide a more detailed description of the above method.

[0052] In one embodiment, a synchronization system needs to be deployed in both the source and destination databases. The source database synchronization system reads logs from the source database and parses them into operations. It then generates an MD5 value based on the table ID and primary key, or the table ID and unique index information, or the table ID and a custom selection column in the DML operation. This MD5 value serves as the key for the operation. The destination database synchronization system is responsible for classifying and collecting the synchronization operations sent from the source database by transaction and using the key of each received operation to set its dependency relationship with any conflicting transactions received before it.

[0053] When calculating the key for an operation, since the order of columns in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the key is calculated according to the fixed order of the primary key or unique index combination columns.

[0054] The source-side data synchronization service can more accurately calculate the MD5 value of the conflict for each operation by detecting the conflict. However, if the calculation is performed by the target-side data synchronization service, the MD5 value calculated based on the primary key information of the target database table will be inaccurate if the primary key structure or column order of the target database table is inconsistent with that of the source.

[0055] In one embodiment, when the target data synchronization service starts, it initializes a hash table R and initializes the record addresses in the hash entries of hash table R to be empty, which is used to store and locate received operations. The target data synchronization service creates two memory slices to store the keys in hash table R. The memory of each memory slice is incremented by a version number in the order of creation, and these two memory slices are added to a memory linked list. The size of hash table R can be set to N, and the size of each memory slice can be set to M, meaning each memory slice is a memory space of size M.

[0056] For hash table R, the structure of a hash entry in its hash table consists of a 4-byte version number, denoted as K; and a 4-byte memory offset, denoted as F; K and F together form a hash entry in the hash table, as shown in the table below:

[0057]

[0058] This indicates that the first key in the HASH entry is stored in the memory slice with version number K, and its offset in that memory slice is F.

[0059] In one embodiment, after the source data synchronization service captures a DML log operation, it obtains the table ID from the log and uses the table ID to retrieve the primary key information and unique index of the table in the source database.

[0060] In one embodiment, after parsing the current DML log operation, the source data synchronization service determines whether the table has a primary key or a unique index. If so, it calculates an MD5 value by combining the table ID and the column values ​​of the primary key or unique index, and uses this as the key for the operation, storing it in the current DML log operation and sending it to the target synchronization service. Otherwise, it proceeds to the next step. It should be noted that when the table has both a primary key and a unique index, the MD5 value calculated by combining the table ID and the primary key is preferentially selected as the key; when the table has no primary key but has multiple unique indexes, the MD5 value is calculated by combining the table ID and any one of the unique indexes.

[0061] In one embodiment, when there is no primary key and unique index, after parsing the current DML log operation, the source data synchronization service calculates an MD5 value by combining the table ID and the ROWID value or a custom selection column of the row data. This MD5 value is then used as the key for the operation and stored in the current DML log operation before being sent to the target synchronization service. It should be noted that when calculating the key for a table without a primary key or unique constraint, if it is known that there are no duplicate values ​​when combining all columns of the table, the ROWID of the table can be used directly to calculate the key, effectively reducing the cost of key calculation. If the table contains duplicate rows, then a custom selection column approach must be used to calculate the key. The selection column can be chosen based on the distribution of column values, allowing the target synchronization service to prevent deadlocks caused by concurrent synchronization transactions operating on rows with duplicate values.

[0062] In one embodiment, after the target data synchronization service receives the DML operation from the source end, it extracts the MD5 value calculated by the source end in the operation as the KEY, and then extracts the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice, which is also the conflict LSN of the operation in the memory slice.

[0063] In the above steps, the target data synchronization service combines the KEY and TRXID into a record and saves it to the offset F of the memory slice with the current version number K. Since the hash record needs to save the address of the next record, the record saved in the memory slice also needs to reserve the address to store the next record, as shown in the table below.

[0064]

[0065] In one embodiment, a detection record X is constructed using the KEY of the current operation. This detection record X is then merged into a hash table R. During the merging process, conflict detection is performed. If a conflicting KEY is found, the TRXID value corresponding to the conflicting KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict. For details, refer to [link to relevant documentation]. Figure 2 As shown, the conflict detection and judgment process includes the following steps.

[0066] Step 101: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, fill the hash item with the address of the detection record X, and return 0 as the conflict TRXID, indicating that no conflict was detected; if not, proceed to the next conflict detection judgment.

[0067] Step 102: Determine whether the memory slice version number K at the address recorded in the hash entry is less than the version number K of the first memory slice in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the record address in the hash entry to the next address space of the current detection record X, fill the hash entry with the address of the detection record X, and return 0 as the conflict TRXID, indicating that no conflict was detected. If not, proceed to the next conflict detection and judgment step.

[0068] Step 103: Load the corresponding record Y from the memory slice based on the address recorded in the hash item. Determine whether the KEY of record Y is equal to the KEY of the currently detected record X and whether the TRXID of record Y is not equal to the TRXID of the current operation. If so, it indicates that there is a conflict in the operation. Remove record Y from the record list of the hash item to shorten the length of the linked list in the hash item. Set the address of the record in the hash item to the next address space of the currently detected record X. Fill the address of the currently detected record X back into the hash item and return it using the TRXID stored in record Y as the conflict TRXID. If not, proceed to the next conflict detection and judgment step.

[0069] Step 104: Iterate through the next record after record Y and perform conflict detection until the record list ends. The condition for the list to end is: the version number at the address of the next record is less than the version number of the first memory slice in the current cache list, or the version number at the address of the next record is null.

[0070] In one embodiment, the TRXID obtained from the conflict check is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN, thereby obtaining the conflict information of this operation.

[0071] In one embodiment, the target data synchronization service receives the next message and saves it to memory slice K as described above. The memory list includes two memory slices. When the first memory slice is full (the written data exceeds the set size M), the system switches to the second memory slice. When the second memory slice is full, the first memory slice in the list is first evicted, and then a new memory slice is created and added to the list using an incrementing version number K. When a record is saved to a new memory slice, if it is the first record in that memory slice, the operation LSN corresponding to that record is marked as the starting LSN of that memory slice.

[0072] In one embodiment, when the target-end data synchronization service receives a rollback or partial rollback operation, it writes the conflicting TRXIDs saved in the rolled-back operation back to the hash table R used for conflict detection. It should be noted that since the transaction dependency information of the rolled-back transactions and partial rollback operations is restored, the dependency relationships of each transaction received in chronological order are clear. These transactions can be committed immediately upon commit, without waiting for transactions with LSNs smaller than their own to commit, thereby further improving synchronization performance. (Reference) Figure 3 The specific steps for receiving a rollback operation are as follows.

[0073] Step 201: Locate the transaction that needs to be rolled back by the transaction ID of the rollback or partial rollback operation, and extract the number of rows contained in the transaction in the rollback operation or the number of rows that need to be rolled back in the partial rollback operation.

[0074] Step 202: Extract the rollback operations from the tail of the transaction operation list backwards, and extract the KEY value, conflicting TRXID and conflicting LSN saved in the operation.

[0075] Step 203: Compare the conflicting LSN of the operation with the starting LSN of the first memory slice. If it is less than the starting LSN, terminate the rollback operation and write back to hash table R. Otherwise, search for a conflicting operation record with the same KEY in hash table R based on the operation KEY. If no record is found, terminate the rollback operation and write back to hash table R. Otherwise, write the conflicting TRXID back to the record in hash table R, and continue to extract the previous operation to be rolled back and perform the same write-back until the specified number of rows is reached.

[0076] By writing back the conflicting TRXID information in the rollback operation as described above, the continuity problem of operation conflict detection can be solved. These rolled-back operations can be directly discarded. Otherwise, when a transaction is executed, although the partially rolled-back operations do not need to be executed, conflict detection still needs to be performed during execution. Otherwise, other transaction operations that depend on it will not be able to find the conflicting transactions they depend on and will be executed prematurely, resulting in data inconsistency.

[0077] In one embodiment, when the system restarts after a failure, the conflict detection mechanism will be reset, and newly received operations will not be able to be executed in parallel with operations performed before the failure. It should be noted that after the target data synchronization service restarts, the LSN of newly received operations will be greater than the LSN of operations performed before the failure. Therefore, the detection range of subsequent operations will be limited to after that LSN. In other words, transactions received after the failure cannot be executed in parallel with transactions performed before the failure.

[0078] The above describes the method for setting transaction operation dependencies based on log order. The following describes the workflow of the target data synchronization service after setting transaction operation dependencies.

[0079] In one embodiment, the target data synchronization service creates H worker threads to execute transactions in the order of their committed LSNs received, and then stores the transaction operations in the database. It should be noted that the target data synchronization service also needs to create a hash table P to register the currently executing transaction information (transaction ID) and a hash table G to register the table information and the starting LSN of the current transaction.

[0080] In one embodiment, a worker thread retrieves a transaction from the committed transaction list in the order of transaction commits, registers the transaction ID in hash table P, and registers the table information involved in the transaction and the starting LSN of the transaction in hash table G.

[0081] In one embodiment, a worker thread extracts an operation from a transaction, determines whether the current operation is a commit operation, and if so, commits the current transaction, wakes up its waiting worker threads, removes the current transaction's registration information from hash tables P and G, and completes the execution of the current transaction; otherwise, it retrieves the conflict TRXID and conflict LSN stored in the operation. It should be noted that since the transactions that each operation in a transaction depends on have already been determined during conflict detection, the transaction can commit directly after executing all operations (meaning that all its dependent transactions have committed), without waiting for transactions that committed before it, thus effectively improving synchronization performance.

[0082] In one embodiment, the worker thread uses the conflicting TRXID of the currently pending operation to search in hash table P for a transaction with the conflicting TRXID. If a conflicting TRXID exists, the current execution process is suspended, and the worker thread waits for the conflicting transaction to complete and commit before waking up; otherwise, it proceeds to the next step. It should be noted that the existence of a conflicting TRXID transaction in hash table P indicates that the data involved in the current operation conflicts with the conflicting TRXID transaction. The current operation cannot be executed concurrently with the conflicting transaction; it must wait for the conflicting transaction to commit before execution can begin.

[0083] In one embodiment, the worker thread uses the information from the current pending operation table to search in hash table G for a transaction that operates on the same table, where the commit LSN of this transaction is less than the current transaction's, and the starting LSN of the transaction is less than the conflict LSN of the current operation. If such a transaction exists, the current execution process is suspended, waiting for the conflicting transaction to complete and commit before waking up. Otherwise, it is determined that the conflict detection for the worker thread's current pending operation has not detected a conflict, so the current operation is executed, and then the next operation continues. It should be noted that this step mainly checks whether there are any concurrently executing transactions that exceed the conflict check LSN range. This is because the conflicting transaction information for the pending operation has a valid range: the conflicting transaction information is only valid within the range from the conflicting LSN saved in the operation to the current operation's LSN. The operation does not perform conflict detection on operations before the conflicting LSN. In other words, when there is a transaction whose commit LSN is less than the commit LSN of the transaction to which the current pending operation belongs, and whose starting LSN is less than the conflicting LSN saved in the current pending operation, it is unknown whether there is a conflict between the operations in this transaction and the current pending operation. Therefore, the current pending operation cannot be executed rashly and must wait for the transaction to commit before execution.

[0084] Based on the above process, this embodiment proposes a method for setting transaction operation dependencies based on log order. The main method is to perform conflict detection on the operation within a fixed LSN interval, and save the detection result and LSN interval to the corresponding operation. When the worker thread executes the operation, it can use the conflict transaction information saved on the operation and its detection LSN interval to determine whether there is a conflict in the currently concurrently executed transactions, thereby improving the parallel execution capability.

[0085] refer to Figure 4As shown, this embodiment also provides a system for setting transaction operation dependencies based on log order. Applying the aforementioned method for setting transaction operation dependencies based on log order, the system includes a KEY acquisition module, a conflict detection module, and a conflict setting module. Specifically: the KEY acquisition module is used to extract the MD5 value calculated by the source end as the KEY after the target-end data synchronization service receives a DML operation from the source end, and then extract the transaction ID from the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If this record is the first record in the current memory slice, then the record corresponding to the operation is saved. The log LSN is marked on the memory slice as its starting LSN. The conflict detection module constructs a detection record X using the KEY of the current operation, merges the detection record X into the hash table R, and performs conflict detection during the merging process. If a conflicting KEY is found, the TRXID value corresponding to the conflicting KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict. The conflict setting module is used to set the TRXID obtained from the conflict detection as the conflicting TRXID, and writes the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation, setting it as the conflicting LSN. The specific functions and principles of the system's functional modules correspond to the specific methods in the method section, and will not be elaborated here.

[0086] In summary, this embodiment proposes a method and system for setting transaction operation dependencies based on log order, which includes the following advantages: First, conflict detection is performed on operations within a fixed LSN interval, and the detection results and the LSN interval are saved to the corresponding operations. Based on this, the dependencies between transaction operations can be determined, and it can be clearly identified which transactions need to be waited for during synchronization operations, thereby maximizing the parallelism of synchronization.

[0087] Secondly, when a worker thread executes an operation, it checks the currently concurrently executing transactions for conflicting transactions based on the conflicting transaction information stored in the operation. If a conflicting transaction exists, the operation cannot proceed. Otherwise, it checks the conflicting LSNs stored in the operation to see if there is a transaction involving the same table as the current operation, and if the transaction's initial LSN is less than the conflicting LSN and its commit LSN is less than the commit LSN of the current operation's transaction. If such a transaction exists, the operation cannot proceed until these transactions have completed. Otherwise, the operation can proceed immediately without waiting for transactions committed earlier to complete, effectively increasing the concurrency of operation execution and improving synchronization performance.

[0088] Finally, when a transaction is committed, since all operations of the transaction have undergone conflict detection during execution, transactions that conflict with the current transaction have certainly already been committed. The current transaction does not need to wait for transactions that committed before it to commit before it can commit synchronously, which further improves synchronization performance.

[0089] Example 2:

[0090] Based on the method for setting transaction operation dependencies based on log order provided in Embodiment 1 above, Embodiment 2 of the present invention provides a specific example for illustration.

[0091] For example: The source database has a table T(ID INT PRIMARY KEY, C1 INT);

[0092] The source application has three transactions performing the following operations on table T in the following order:

[0093] TRX1:INSERT INTO T(ID, C1) VALUES(1, 1);

[0094] TRX1:COMMIT;

[0095] TRX2:INSERT INTO T(ID, C1) VALUES(2, 2);

[0096] TRX2:UPDATE T SET C1=1 WHERE ID=1;

[0097] TRX2:COMMIT;

[0098] TRX3:INSERT INTO T(ID, C1) VALUES(3, 3);

[0099] TRX3:COMMIT;

[0100] The order of the operations described above represents their order within the log stream, resulting in the following numbered table of the generated log stream:

[0101]

[0102] After the target data synchronization service starts, it needs to initialize a hash table R of size N and two memory slices of size M, with version numbers 1 and 2.

[0103] The process is as follows:

[0104] 1. When parsing logs, the source data synchronization service needs to first obtain the primary key information column C of the table through the table ID. Then, based on the combination of the table ID and the value in column C, the key of each operation is calculated after parsing the log operations. For ease of explanation, the primary key value is used to represent the MD5 value of the key in the table below.

[0105]

[0106] 2. The log receiving thread receives operations with LSN 1, extracts its KEY as 1, combines the TRXID and the space of the next record, and the length is 32. First, it stores its KEY and transaction ID in the first memory slice, expressed as [version number (K): memory offset (F)], and its offset in the memory slice is [1:0].

[0107]

[0108] Since this KEY information is the first operation information of the current memory chip, the LSN value of the operation should be set to the starting LSN of the memory chip, which is set to 1.

[0109] 3. Insert the KEY into hash table R. The returned TRXID is 0, which means there is no conflict. Write back the conflict-free TRXID of 0 and the minimum starting LSN of the memory slice of 1 to the operation.

[0110] 4. The log receiving thread receives the operation with LSN 3, extracts its KEY as 2, combines the TRXID and the space of the next record, and the length is 32. First, it stores its KEY and transaction ID in the first memory slice, and its offset in the memory slice is [1:32].

[0111]

[0112] 5. Insert the KEY into hash table R. The returned TRXID is 0, which means there is no conflict. Write back the conflict-free TRXID of 0 and the minimum starting LSN of the memory slice of 1 to the operation.

[0113] 6. The log receiving thread receives the operation with LSN 4, extracts its KEY as 1, combines the TRXID and the space of the next record, and the length is 32. First, it stores its KEY and transaction ID in the first memory slice, and its offset in the memory slice is [1:64].

[0114]

[0115] 7. Insert the KEY into the hash table R. Since the operation information with KEY 1 already exists in the HASH, the returned conflicting TRXID is 1, which means that the current operation and the transaction with transaction ID 1 have a conflict. Write the conflicting TRXID 1 and the minimum starting LSN of the memory slice of 1 back into the operation.

[0116] 8. Remove conflicting key information with a key value of 1 from the HASH table.

[0117] 9. Set the address in hash table R with KEY 1 to the next record of the current KEY information, and get the following table.

[0118]

[0119] 10. Restart the target data synchronization service. The information in the memory linked list of the target data synchronization service will be reset, and a new round of reception will begin.

[0120] 11. The log receiving thread receives an operation with LSN 6, extracts its KEY (3), combines it with the TRXID and the space of the next record, resulting in a length of 32. It first stores its KEY and transaction ID in the first memory slice, with an offset of [1:0] within the memory slice. Since it is the first KEY information in the memory slice, its LSN 6 is set as the starting LSN of that memory slice.

[0121]

[0122] Insert the KEY into hash table R. The returned TRXID is 0, which means there is no collision. Write back the collision-free TRXID 0 and the minimum starting LSN of the memory slice is 6 to the operation.

[0123] After all operation conflict detections are completed, the following detection results are generated.

[0124]

[0125] 12. Create three worker threads and execute TRX1, TRX2 and TRX3 respectively in the order of transaction commit.

[0126] 13. Worker thread 1 extracts the TRX1 transaction, registers the transaction information TRXID as 1 in hash table P, and registers the starting LSN of the transaction associated with table information T as 1 in hash table G.

[0127] 14. Worker thread 2 extracts the TRX2 transaction, registers the transaction information TRXID 2 in hash table P, and registers the starting LSN of the transaction associated with table information T, which is 3, in hash table G.

[0128] 15. Worker thread 3 extracts the TRX3 transaction, registers the transaction information TRXID 3 in hash table P, and registers the starting LSN of the transaction associated with table information T, which is 6, in hash table G.

[0129] 16. Worker thread 1 extracts the insertion operations with LSN 1 on TRX1, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0130] (1) The operation conflict TRXID is 0, indicating no conflict.

[0131] (2) Obtain the table information T and the conflict LSN of the operation is 1. Search in the hash table G for conflicting transactions that involve table T and have a commit LSN less than 2 and a minimum starting LSN of the transaction less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0132] 17. Worker thread 2 extracts the insertion operation with LSN 3 on TRX2, obtains the conflict TRXID and conflict LSN of the operation, and performs conflict detection.

[0133] (1) The operation conflict TRXID is 0, indicating no conflict.

[0134] (2) Obtain the table information T and the conflict LSN of the operation is 1. Search in the hash table G for conflicting transactions that involve table T and have a commit LSN less than 5 and a minimum starting LSN of the transaction less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0135] 18. Worker thread 2 extracts the update operation with LSN 4 on TRX2, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0136] (1) The operation conflict TRXID is 1, which means there is a conflict with transaction TRX1. The current working thread needs to be suspended and woken up after the working thread 1 completes the synchronization of transaction TRX1.

[0137] 19. Worker thread 3 extracts the insertion operation with LSN 6 on TRX3, obtains the conflict TRXID and conflict LSN of the operation, and performs conflict detection.

[0138] (1) The operation conflict TRXID is 0, indicating no conflict.

[0139] (2) Obtain the table information T and the conflict LSN of the operation is 6. Search in the hash table G for conflicting transactions that involve table T and have a commit LSN less than 7 and a minimum starting LSN less than 6. TRX1 and TRX2 transactions are found to meet the conflict conditions. Since the commit LSN of TRX2 is larger than that of TRX1, the current working thread needs to be suspended and waited for working thread 2 to complete the synchronization of transaction TRX2 before being woken up.

[0140] 20. After worker thread 1 executes the commit operation of TRX1, it deletes the current transaction information from hash table P and hash table G respectively, and then wakes up worker thread 2 to execute the TRX2 transaction.

[0141] 21. After worker thread 2 executes the commit operation of TRX2, it deletes the current transaction information from hash table P and hash table G respectively, and then wakes up worker thread 3 to execute the TRX3 transaction.

[0142] 22. After worker thread 3 executes the commit operation of TRX3, it deletes the current transaction information from hash table P and hash table G respectively, and then completes the execution of all transactions.

[0143] As can be seen from the above process, transaction TRX1, being the first to be committed, will not conflict with subsequent committed transactions and can therefore execute smoothly. Transaction TRX2's update operation has a conflict with TRX1's insert operation during conflict detection, so TRX2 must wait for TRX1 to complete before it can execute. Transaction TRX3's insert operation, from a data perspective, does not conflict with TRX1 or TRX2. However, because it is a re-performed conflict detection after a failure, its conflict detection LSN has been reset to 6. Therefore, although it has no conflicting transactions, when using the conflict LSN to determine whether execution is possible, it can find transactions that meet the conditions. Thus, it must wait for the conflicting transactions to complete before it can begin execution.

[0144] Example 3:

[0145] It should be noted that for rollback transactions, since the rollback transaction will not be executed, there are three ways to handle the rollback transaction operation in the hash table R.

[0146] The first approach: This method does not roll back the impact of rollback transactions on operations dependent on them in hash table R, nor does it collect conflicting TRXIDs of rollback transaction dependencies. It directly releases all operations of the rollback transaction. This rollback transaction will be depended upon by other operations, and it will still be added to the pending execution list for execution to ensure the continuity of operation dependencies. When using this approach, the entire rollback transaction is released, which can improve the performance of handling rollback transactions. However, operations dependent on the rollback transaction can only execute after the rollback transaction is released in the pending execution list. Since conflicting transactions were not collected during the release operation, the rollback transaction can only be released after all transactions with commit LSNs or rollback LSNs less than its own rollback LSN have been executed, which affects synchronization efficiency. The corresponding approach is that when the target data synchronization service receives a rollback transaction, all operations on this transaction do not need to be executed; they are discarded directly, and only the retained transaction information is added to the pending execution queue. For example, when the target data synchronization service receives a rollback or partial rollback operation, it needs to locate the transaction based on the transaction ID of the operation, and then mark the transaction to indicate that some operations of the transaction have been rolled back. If the operation is a rollback operation, then all operations on this transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed. The transaction information of the rollback transaction cannot be discarded.

[0147] The second approach: While not reversing the impact of rollback transactions on operation dependencies in hash table R, it collects conflicting TRXIDs of rollback transaction dependencies, for example, by recording them in a conflicting TRXID dependency list L. This rollback transaction will be depended upon by other operations, and it will also be added to the pending execution list for execution to ensure the continuity of operation dependencies. When using this approach, conflicting TRXIDs in the operation must be collected before the rollback transaction is released. Although this reduces the processing performance of rollback transactions, it improves the release performance of rollback transactions during execution. Operations dependent on rollback transactions can only be executed after the rollback transaction is released in the pending execution list. However, since conflicting transactions that depend on the rollback transaction were collected during the release operation, the rollback transaction can be released only after these conflicting transactions have completed, without waiting for all transactions with commit LSNs or rollback LSNs less than its own rollback LSN to complete. Compared to the first approach, this partially improves synchronization efficiency. This approach corresponds to the following: when the target data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation. Furthermore, for the rollback operation, the corresponding transaction cannot be discarded, but is added to the queue of pending transactions.

[0148] The third approach involves rolling back the impact of a transaction's operations on the dependencies of operations in hash table R. This rolled-back transaction will not be depended upon by other operations, and it does not need to be added to the pending execution list. When using this approach, after receiving a rollback operation, the transaction needs to roll back the impact of all operations within the transaction on the dependencies of operations in hash table R. Since the transaction size is uncontrollable, rolling back a large transaction will affect the progress of receiving operations, thus impacting synchronization efficiency. In this method, when the target data synchronization service receives a rollback or partial rollback operation, it writes the conflicting TRXIDs saved in the rolled-back operations back to hash table R. The description of the rollback portion in Embodiment 1 of this invention is an example description of this approach.

[0149] The three schemes for handling transaction rollback each have their advantages and disadvantages. The first scheme is suitable for large-scale transaction rollbacks, the second scheme for medium-scale transaction rollbacks, and the third scheme for small-scale transaction rollbacks. Therefore, two parameters, W1 and W2, are set: W1 represents the minimum number of DML operations in a large-scale transaction, and W2 represents the minimum number of DML operations in a medium-scale transaction. Based on this, after receiving DML operations from the source end, the target-side data synchronization service needs to classify and manage the received DML operations by transaction and calculate the scale S of the DML operations received in each transaction (S is the number of DML operations received in a single transaction). When the target-side data synchronization service receives a rollback operation, it takes corresponding actions based on the scale S of the DML operations in the transaction.

[0150] When the size S of the rollback transaction is greater than or equal to W1, it corresponds to a large-scale transaction rollback in the first scheme. In this case, the impact of the rollback transaction's operations on the operation dependencies in the hash table R is not rolled back, and conflicting TRXIDs of the rollback transaction's operation dependencies are not collected. All operations of the rollback transaction are directly released. This rollback transaction will be depended on by other operations, and the rollback transaction will also be added to the pending execution list for execution to ensure the continuity of operation dependencies.

[0151] When the size S of the rollback transaction is greater than or equal to W2 and less than W1, it corresponds to a medium-sized transaction rollback in the second scheme. In this case, the operation of not rolling back the rollback transaction does not affect the operation dependencies in the hash table R. However, the conflicting TRXIDs of the rollback transaction operation dependencies are collected, and they can be recorded in the conflicting TRXID dependency list L. This rollback transaction will be depended on by other operations, and the rollback transaction should also be added to the pending execution list for execution to ensure the continuity of operation dependencies.

[0152] When the size S of the rollback transaction is less than W2, it corresponds to the small-scale transaction rollback in the third scheme. In this case, the rollback operation affects the operation dependencies in the hash table R, and the rollback transaction does not need to be added to the pending execution list; it is released directly. The description of the rollback part in Embodiment 1 of this invention is an example description when the size S of the rollback transaction is less than W2.

[0153] Furthermore, for partial rollback operations, due to the special nature of partial rollback, which inherently requires traversing all rollback operations, adding a process to collect conflicting TRXIDs from these operations does not significantly degrade the performance of handling partial rollbacks. Based on this, when the target data synchronization service receives a partial rollback operation, it needs to count the total number of rolled-back rows S1. Depending on the size of S1, the following processing is performed: When the number of rollback rows S1 is less than W2, the impact of the rollback operation on the operation dependencies in hash table R is rolled back, i.e., the conflicting TRXIDs saved in the rolled-back operations are written back to hash table R. When the number of rollback rows S1 is greater than or equal to W2, the impact of the rollback operation on the operation dependencies in hash table R is not rolled back, i.e., the conflicting TRXIDs saved in the rolled-back operations are not written back to hash table R, but the conflicting TRXIDs dependent on the rollback operation are collected and recorded in the conflicting TRXID dependency linked list L.

[0154] Based on the above settings, after the target data synchronization service creates a worker thread, the worker thread extracts a transaction from the committed transaction list in the order of transaction commits, and then determines whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to determine whether the rollback transaction has a conflict with the TRXID list L.

[0155] If the rollback transaction does not conflict with the TRXID chain L, it needs to wait for all transactions with a commit LSN less than the current transaction to complete before waking up its waiting worker thread. Then, it removes the current transaction's registration information from hash tables P and G, completing the execution of the current transaction. This is because the operations on the rollback transaction do not need to be executed and are discarded, so the transaction information they depend on is unclear. Therefore, when committing, it must wait for all transactions that committed before it to commit to ensure the correctness of the transaction dependency order.

[0156] If a rollback transaction has a conflicting TRXID chain L, it needs to wait for all conflicting transactions in chain L to be committed before waking up its waiting worker thread. Then, it removes the current transaction's registration information from hash tables P and G, completing the execution of the current transaction. Specifically, it sequentially extracts a conflicting TRXID from chain L, checks if it exists in hash table P, and if it does, waits for that transaction to complete before waking up its own thread. After waking up, it extracts the next conflicting TRXID from chain L and performs the same check until all conflicting transactions in chain L are completed. This is because the operations on the rollback transaction do not need to be executed and are discarded. When discarding the rollback operations, the conflicting TRXIDs of the operations are collected, so their dependent transaction information is clear. Therefore, during release, it is only necessary to wait for these conflicting transactions to be committed to ensure the correctness of the transaction dependency order.

[0157] In summary, since rollback transactions are not executed, three conflict detection schemes arise, each with its own advantages. Appropriate values ​​need to be set to leverage the strengths of each scheme to improve overall synchronization performance. Furthermore, for partial rollback operations of committed transactions, specifically collecting the conflicting TRXIDs referenced during the rollback operation can effectively mitigate the performance loss caused by writing back to the conflict detection hash table R during partial rollback operations, thus improving overall synchronization performance. This embodiment handles rollback transactions and partial rollback operations according to a predetermined operational scale, seeking a balance between the costs of operation rollback and rollback transaction execution, thereby improving overall synchronization performance.

[0158] In some embodiments, the keys in hash table R can be stored either in files or in memory slices. Specifically, the difference lies in the storage method. While file storage reduces performance and impacts collision detection efficiency, the state of hash table R before the failure can be restored using the key values ​​in the stored files after a data synchronization failure. This ensures continuity in collision detection for received operations after the failure, preventing a precipitous drop in synchronization performance. Conversely, while memory slice storage significantly improves collision detection performance compared to file storage, it loses previously detected collision data after a failure. Recovery requires starting from the currently received LSN, leading to a discontinuity in subsequent collision detection and a sharp drop in synchronization performance at the failure point. When storage performance is high (e.g., using SSD fixed storage), the performance requirements for conflict detection can be met, and the efficiency of operation reception and synchronization execution can be balanced. In this case, storing the keys in the hash table R through files can effectively avoid the loss of synchronization performance caused by failures. When storage performance affects the efficiency of conflict detection, the bottleneck of synchronization lies in operation reception. In this case, storing the keys in the hash table R through memory slices can effectively improve the operation reception performance, thereby improving the overall synchronization performance. Embodiment 1 of this application is an example of storing the keys in the hash table R through memory.

[0159] In some embodiments, the MD5 value can be calculated at either the source or the target end as the key for the operation. Calculating the MD5 value for each operation by detecting conflicts in the source-end data synchronization service allows for more accurate conflict MD5 value calculation. However, if the calculation is performed by the target-end data synchronization service, the MD5 value calculated using the primary key information of the target database table will be inaccurate if the primary key structure or column order is inconsistent with the source table. Therefore, it is generally preferred to calculate the MD5 value at the source end as the key for the operation. After receiving the DML operation from the source end, the target-end data synchronization service directly extracts the MD5 value calculated at the source end as the key.

[0160] In some embodiments, when calculating the key for an operation, since the column order in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the key is calculated according to the fixed order of the primary key or unique index combination columns. After the source data synchronization service captures a DML log operation, it obtains the table ID in the log and uses the table ID to retrieve the primary key information and unique index of the table in the source database. After parsing the current DML log operation, the source data synchronization service determines whether the table has a primary key or unique index. If a primary key or unique index exists, it calculates an MD5 value by combining the table ID and the column values ​​of the primary key or unique index as the key for this operation, stores it in the current DML log operation, and sends it to the target synchronization service. It should be noted that when the table has both a primary key and a unique index, the MD5 value calculated by combining the table ID and the primary key is preferred as the key; when the table has no primary key but has multiple unique indexes, the MD5 value calculated by combining the table ID and any one of the unique indexes is selected. If there is no primary key or unique index, after parsing the current DML log operation, the source data synchronization service calculates an MD5 value by combining the table ID and the ROWID value or a custom selection column of the corresponding row. This MD5 value is then stored in the current DML log operation and sent to the target synchronization service. It should be noted that when calculating the key for a table without a primary key or unique constraint, if it is known that there are no duplicate values ​​in any combination of columns, the table's ROWID can be used directly to calculate the key, effectively reducing the cost of key calculation. If the table contains duplicate rows, then a custom selection column approach must be used to calculate the key. The selection column can be chosen based on its distribution, allowing the target synchronization service to prevent deadlocks caused by concurrent synchronization transactions operating on rows with duplicate values.

[0161] Example 4:

[0162] Based on the method for setting transaction operation dependencies based on log order provided in Embodiment 1 above, the present invention also provides an apparatus for setting transaction operation dependencies based on log order that can be used to implement the above method and system, such as... Figure 5 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The device for setting transaction operation dependencies based on log order in this embodiment includes one or more processors 21 and a memory 22. Figure 5 Take a processor 21 as an example.

[0163] Processor 21 and memory 22 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0164] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the method for setting transaction operation dependencies based on log order in Embodiment 1. The processor 21 executes various functional applications and data processing of the device for setting transaction operation dependencies based on log order by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the method for setting transaction operation dependencies based on log order in Embodiment 1.

[0165] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] The program instructions / modules are stored in memory 22. When executed by one or more processors 21, they execute the method for setting transaction operation dependencies based on log order as described in Embodiment 1 above. For example, they execute the method described above. Figures 1-4 The steps shown.

[0167] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0168] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above. For the sake of brevity, they are not provided in detail; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for setting transaction operation dependencies based on log order, characterized in that, include: After the target data synchronization service receives the DML operation from the source, it extracts the MD5 value calculated by the source as the KEY and the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice. Construct a detection record X using the KEY of the current operation, merge the detection record X into the hash table R, and perform a collision detection during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no collision. Set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN; The memory list includes two memory slices, which are used to store the keys in the hash table R. When the first memory slice is full, the process switches to the second memory slice. When the second memory slice is full, the first memory slice in the list is first removed, and then a new memory slice is created and added to the memory list with an incrementing version number. The memory of each memory slice is set with an incrementing version number according to the creation order. The two memory slices are then added to the memory list. The conflict detection judgment includes: Calculate the hash value of the KEY of the current detection record X, locate the hash entry through the hash value; determine whether the address of the record in the hash entry is empty, if so, set the address of the record in the hash entry to the next address space of the current detection record X, fill the hash entry with the address of the detection record X, and return 0 as the collision TRXID, indicating that no collision was detected; If not, check if the memory slice version number K at the address recorded in the hash entry is less than the version number K of the first memory slice in the memory linked list. If so, it indicates that this record Y has exceeded the check range.

2. The method for setting transaction operation dependencies based on log order according to claim 1, characterized in that, After the source data synchronization service captures a DML log operation, it obtains the table ID from the log and retrieves the primary key information and unique index of the table in the source database using the table ID. The calculation of the MD5 value includes generating an MD5 value by combining the table ID and primary key, or the table ID and unique index information, or the table ID and a custom selected column from the DML operation.

3. The method for setting transaction operation dependencies based on log order according to claim 2, characterized in that, For MD5 value calculation in DML operations, when a table has both a primary key and a unique index, the table ID and the primary key are used together to calculate the MD5 value as the key; when a table has no primary key but has multiple unique indexes, the table ID and any one of the unique indexes are used to calculate the MD5 value; when there is no primary key and no unique index, the table ID and the ROWID value of the row data in the table or a custom selection column are used to calculate an MD5 value.

4. The method for setting transaction operation dependencies based on log order according to claim 1, characterized in that, When the target data synchronization service starts, it initializes a hash table R and initializes the record addresses in the hash entries of hash table R to be empty, which is used to store and locate the received operations.

5. The method for setting transaction operation dependencies based on log order according to any one of claims 1-4, characterized in that, The conflict detection judgment includes: When record Y has exceeded the inspection range, the address of the record in the hash field is set to the next address space of the current detection record X, and the address of the detection record X is used to fill the hash field. 0 is used as the collision TRXID to return, indicating that no collision was detected; otherwise, the next collision detection judgment is performed. Based on the address recorded in the hash entry, load the corresponding record Y from the memory slice. Determine if the KEY of record Y is equal to the KEY of the currently detected record X, and if the TRXID of record Y is not equal to the TRXID of the current operation. If so, it indicates a conflict. Remove record Y from the record list of the hash entry to shorten the length of the linked list in the hash entry. Set the address of the record in the hash entry to the next address space of the currently detected record X. Fill the address of the currently detected record X back into the hash entry, and return the TRXID stored in record Y as the conflict TRXID. If not, proceed to the next conflict detection step. Iterate through the next record of record Y and perform conflict detection and judgment until the record linked list ends.

6. The method for setting transaction operation dependencies based on log order according to any one of claims 1-4, characterized in that, When the target data synchronization service receives a rollback or partial rollback operation, it writes the conflict TRXIDs saved in the rolled-back operation back to the hash table R used for conflict detection. Specifically: Locate the transaction that needs to be rolled back by the transaction ID of the rollback or partial rollback operation, and extract the number of rows contained in the transaction in the rollback operation or the number of rows that need to be rolled back in the partial rollback operation. Extract rollback operations from the tail of the transaction operation list backwards, and extract the KEY value, conflicting TRXID, and conflicting LSN saved in the operation; The conflicting LSN of the operation is compared with the starting LSN of the first memory slice. If it is less than the starting LSN, the rollback operation is terminated and the write-back to hash table R is stopped. Otherwise, the conflicting operation record with the same key is searched in hash table R. If no record is found, the rollback operation is terminated and the write-back to hash table R is stopped. Otherwise, the conflicting TRXID is written back to the record in hash table R, and the previous operation to be rolled back is extracted and the same write-back is performed until the specified number of rows is reached.

7. An apparatus for setting transaction operation dependencies based on log order, characterized in that, The apparatus includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the method for setting transaction operation dependencies based on log order as described in any one of claims 1-6.

8. A system for setting transaction operation dependencies based on log order, using the method for setting transaction operation dependencies based on log order as described in any one of claims 1-6, characterized in that, The system includes a KEY acquisition module, a conflict detection module, and a conflict setting module, wherein: The KEY acquisition module is used to extract the MD5 value calculated by the source end as the KEY after the target end data synchronization service receives the DML operation from the source end, and then extract the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record of the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice. The conflict detection module constructs a detection record X using the KEY of the current operation, merges the detection record X into the hash table R, and performs conflict detection during the merging process. If there is a conflicting KEY, it returns the TRXID value corresponding to the conflicting KEY; otherwise, it returns 0 as the corresponding TRXID, indicating that there is no conflict. The conflict setting module is used to set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN.

9. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions that are executed by one or more processors to perform the method for setting transaction operation dependencies based on log order as described in any one of claims 1-6.

Images