A fault recovery method, device and system for setting transaction operation dependency

Abstract

The present invention relates to a fault recovery method, device, and system for setting transaction operation dependencies. The method portion mainly includes: after the target-side data synchronization service receives the operation of the source side, it determines whether the difference between the LSN of the current operation and the LSN value of the last checkpoint is greater than or equal to a preset value N. If so, the operations within the range of the last checkpoint LSN to the current LSN 1 are classified as transactions and saved in a checkpoint file; after the target-side data synchronization service receives the commit operation of the transaction, it asynchronously executes and stores the transaction; when the system restarts after the failure, the last checkpoint file is used to restore the KEY in the hash table R. The present invention can achieve seamless connection between the operations received after the failure and the conflict detection of the operations before the failure, thereby ensuring that the failure will not affect the parallel execution capability of transactions before and after the failure.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a fault recovery method, device and system for setting transaction operation dependencies. Background Art

[0002] In a database real-time synchronization system (supporting heterogeneous architecture) based on a log parsing architecture, the source-side data synchronization service is responsible for capturing the operation logs of the source database, parsing the logs to restore the corresponding database operations, and then sending them to the target-side data synchronization service. The target-side data synchronization service is responsible for executing these operations in the target database, thereby completing the data synchronization process.

[0003] When the target-side data synchronization service synchronizes operations on the target database, it often needs to execute them in the order in which transactions were submitted. This is because if the previous transaction hasn't completed synchronization, the next transaction can't be executed concurrently with the previous transaction without first confirming whether it conflicts with the previous one. Execution can only begin after all transactions preceding the current operation have completed their submissions, significantly impacting the concurrency of transactional operations. Currently, conflict detection is performed on each operation, and dependent transaction information is set for each operation. This allows for concurrent execution based on this dependency information, thereby improving synchronization performance. However, conflict detection relies on a continuous conflict detection mechanism. If a failure occurs and a restart is required, conflict detection is interrupted and reset after the restart, reducing synchronization performance after failure recovery.

[0004] In view of this, how to overcome the defects of existing technologies, how to implement a fault recovery mechanism for conflict detection, and how to reduce the impact of faults on synchronization performance have become important technical issues that need to be urgently addressed in the industry. Summary of the Invention

[0005] In response to the defects or improvement needs in the existing technology: how to implement a fault recovery mechanism for conflict detection and reduce the impact of faults on synchronization performance. The present invention provides a fault recovery method, device, and system for setting transaction operation dependencies. To ensure the continuity of conflict detection during a fault, the target-side data synchronization service needs to periodically save the received operations in the form of checkpoints. After a fault occurs, the conflict detection environment is restored through the operations saved in the checkpoints to ensure the continuity of conflict detection before and after the fault, thereby reducing the impact of the fault on synchronization performance.

[0006] The present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a fault recovery method for setting transaction operation dependency, comprising:

[0008] After receiving the operation from the source, the target-side data synchronization service determines whether the difference between the current operation's LSN and the last checkpoint's LSN is greater than or equal to the preset value N. If so, the operations between the last checkpoint's LSN and the current LSN-1 are grouped as transactions and saved in the checkpoint file. The starting LSN of the checkpoint is set to the last checkpoint's LSN+1, and the ending LSN is set to the current operation's LSN-1.

[0009] After receiving the transaction commit operation, the target-side data synchronization service asynchronously executes the transaction into the database. After the storage is completed, it determines whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The checkpoint file's start LSN <= commit LSN <= end LSN is used for the check. If the checkpoint file's start LSN <= commit LSN <= end LSN is satisfied, the transaction ID is appended to the checkpoint file.

[0010] When the system fails and restarts, the last checkpoint file is used to restore the KEY in the hash table R. After the restart is complete, the source end operations are continued to be received. The operations whose LSN is less than or equal to the end LSN of the checkpoint file are discarded. If the LSN is greater than the end LSN, the KEY in the operation is extracted for conflict detection.

[0011] In some embodiments, after the target-side data synchronization service is started, a hash table Q for registering transaction information is initialized to save the status of all transactions within the received conflict detection LSN range, including active status, committed status, and rollback status; after the target-side data synchronization service receives the DML operation from the source, the transaction ID of the extraction operation is registered in the hash table Q, and the transaction status is initially active.

[0012] In some embodiments, when the target-side data synchronization service receives a commit operation, it adds the transaction to the pending transaction queue and modifies the transaction status in the hash table Q from the active status to the committed status; when the target-side data synchronization service receives a rollback operation, it discards the transaction and modifies the transaction status in the hash table Q from the active status to the rollback status, and writes the conflicting TRXID saved in the rolled back operation back to the hash table R.

[0013] In some embodiments, after the target-side data synchronization service is started, a hash table R is initialized for storing and locating received operations, and two memory slices are created to store the keys in the hash table R. The version number of each memory slice is incremented in the order of creation, and the two memory slices are added to the memory linked list;

[0014] After receiving the DML operation from the source, the target-side data synchronization service extracts the MD5 value calculated by the source as the key, then extracts the transaction ID from the operation as the TRXID, combines the key and TRXID into a record and saves it to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice.

[0015] Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict.

[0016] The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

[0017] In some embodiments, the conflict detection judgment specifically includes:

[0018] Calculate the hash value of the KEY of the current detection record X and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, backfill the hash item with the address of the detection record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection and judgment;

[0019] Determine whether the memory slice ID in the address recorded in the hash item is less than the ID value of the first memory slice in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and backfill the hash item with the address of the detection record X. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment;

[0020] According to the address recorded in the hash item, the corresponding record Y is loaded from the memory chip, and it is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, and record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item. The record address in the hash item is set to the next address space of the current detection record X, and the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflict TRXID for return; if not, the detected conflict TRXID is searched in the hash table Q to obtain the status of the conflicting transaction, and the conflicting TRXID is returned according to the conflicting transaction status;

[0021] Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record list.

[0022] In some embodiments, searching the hash table Q for the detected conflicting TRXID to obtain the conflicting transaction status, and returning the conflicting TRXID according to the conflicting transaction status specifically includes:

[0023] If the conflicting transaction is active: -1 is returned as the conflict TRXID, indicating that it may conflict with transactions whose commit LSN is smaller than that of the transaction to which the current operation belongs;

[0024] If the conflicting transaction is in the rollback state: ignore the transaction and continue searching for other conflicting transactions in the current hash item. If a conflicting transaction is found, continue to classify and process it according to the conflicting transaction status; otherwise, return 0 as the conflict TRXID, indicating that no conflict was detected.

[0025] If the conflicting transaction is in the committed state: return the conflicting transaction ID as the conflicting TRXID.

[0026] In some embodiments, when the system fails and restarts, restoring the KEY in the hash table R using the last checkpoint file specifically includes:

[0027] Get the most recent checkpoint file F, the starting LSN of the checkpoint file F is LSN1, and the ending LSN is LSN2; create an array A of size N;

[0028] Read the transaction ID information of the executed transaction in the checkpoint file F for filtering during recovery; read the transaction operation OP saved in the checkpoint file F, extract the transaction ID of the operation, and determine whether the transaction ID has been executed. If so, abandon the operation; otherwise, extract the LSN value of the operation OP and store it in array A according to the algorithm A[LSN-LSN1]=OP;

[0029] Extract the operation OP and determine whether it is a DML operation. If so, mark the transaction as active in the hash table Q based on the TRXID on the operation. Otherwise, determine whether it is a commit operation. If so, mark the transaction as committed in the hash table Q based on the TRXID on the operation. Otherwise, mark the transaction as rolled back in the hash table Q based on the TRXID on the operation.

[0030] Read the operations in the order of array A, extract the KEY and transaction ID in the operation to restore the state of hash table R before the failure.

[0031] In a second aspect, the present invention further provides a fault recovery device for setting transaction operation dependency, the device comprising:

[0032] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor to execute the fault recovery method for setting transaction operations based on laziness as described in the first aspect.

[0033] In a third aspect, the present invention further provides a fault recovery system for setting transaction operation dependencies, which applies the fault recovery method for setting transaction operation dependencies as described in the first aspect. The system includes a checkpoint setting module, a commit LSN determination module, and a fault restart module, wherein:

[0034] The checkpoint setting module is used to determine whether the difference between the LSN of the current operation and the LSN of the last checkpoint is greater than or equal to a preset value N after the target-side data synchronization service receives the operation from the source side. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1 and the ending LSN is set to the current operation LSN-1.

[0035] The commit LSN determination module is used to asynchronously execute and store the transaction after the target-side data synchronization service receives the commit operation of the transaction. After the storage is completed, it is used to determine whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The determination is made based on the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If the condition is met, the transaction ID is appended to the checkpoint file.

[0036] The fault restart module is used to restore the KEY in the hash table R using the last checkpoint file after the system fault restarts; after the restart is completed, it continues to receive operations from the source end, discards operations whose operation LSN is less than or equal to the end LSN of the checkpoint file, and extracts the KEY in the operation for conflict detection and judgment if the operation LSN is greater than the end LSN.

[0037] In a fourth aspect, the present invention further provides a non-volatile computer storage medium storing computer executable instructions, which are executed by one or more processors to complete the fault recovery method for setting transaction operation dependency as described in the first aspect.

[0038] Compared with the prior art, the present invention provides a fault recovery method, device and system for setting transaction operation dependency, which has the following beneficial effects:

[0039] First, the target-side data synchronization service sets an interval LSN value of N and stores received operations in shards up to N. After a failure restart, it only needs to use the operations in the most recently saved shard to restore the key information in hash table R and the transaction status in hash table Q before the failure. This allows for seamless conflict detection between operations received after the failure and those before the failure, ensuring that the failure does not affect the parallel execution of transactions before and after the failure.

[0040] Secondly, after a transaction is completed, if the transaction's commit LSN falls within the start and end LSN range of the most recent checkpoint file, the transaction ID of that transaction is appended to the most recent checkpoint file. When the target-side data synchronization service uses the most recent checkpoint file to restore the pre-failure hash table R information, it can filter out completed transactions, speeding up recovery. Filtering out completed transactions also helps reduce the size of the key information in the restored hash table R, thereby improving the performance of conflict detection in the first shard received. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0042] Figure 1 A flowchart of a fault recovery method for setting transaction operation dependency provided in Example 1 of the present invention;

[0043] Figure 2 This is a flowchart of conflict detection and judgment provided in Example 1 of the present invention;

[0044] Figure 3 Conflicting TRXID write-back flow chart for the rollback operation provided in Example 1 of the present invention;

[0045] Figure 4 Flowchart for recovering the KEY of the hash table R provided in Example 1 of the present invention;

[0046] Figure 5 A schematic diagram of a module of a fault recovery system for setting transaction operation dependencies provided in Example 1 of the present invention;

[0047] Figure 6 This is a schematic structural diagram of a fault recovery device for setting transaction operation dependency provided by Example 3 of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention. These all fall within the scope of protection of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other and are all within the scope of protection of this application. In addition, although the functional modules may be divided in the device schematic diagram and the logical order may be shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow chart.

[0049] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this invention belongs. The terms used in this specification and in the description of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. Furthermore, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0050] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as meaning open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that the specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0051] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate this application and are not intended to limit this application. The present invention will be described in detail below with reference to the accompanying drawings and examples.

[0052] Example 1:

[0053] like Figure 1 As shown, an embodiment of the present invention provides a fault recovery method for setting transaction operation dependency, which includes the following steps.

[0054] Step 100: After receiving the operation from the source, the target-side data synchronization service determines whether the difference between the LSN of the current operation and the LSN of the last checkpoint is greater than or equal to the preset value N. If so, the operations within the range from the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1 and the ending LSN is set to the current operation LSN-1. It should be noted that in this embodiment, LSN is the log sequence number; TRXID is the id that identifies a transaction; DML is the insert, update, and delete operation of data; MD5 is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value to ensure the integrity and consistency of information transmission; KEY is a key that can be used to identify a specific row of records in a table; ROWID is a pseudo-column used to uniquely mark rows in a table.

[0055] Step 200: After the target-side data synchronization service receives the transaction commit operation, it asynchronously executes and stores the transaction. After the storage is completed, it determines whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The judgment is made according to the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If it is satisfied, the transaction ID is appended to the checkpoint file.

[0056] Step 300: When the system fails and restarts, the last checkpoint file is used to restore the KEY in the hash table R; after the restart is completed, the source end operation is continued to be received, and the operations whose received operation LSN is less than or equal to the end LSN of the checkpoint file are discarded. If the operation LSN is greater than the end LSN, the KEY in the operation is extracted for conflict detection and judgment.

[0057] The target-end data synchronization service of the embodiment of the present invention needs to periodically save the received operations in the form of checkpoints. When a failure occurs, the conflict detection environment is restored through the operations saved in the checkpoints to ensure the continuity of conflict detection before and after the failure, thereby reducing the impact of the failure on synchronization performance.

[0058] This embodiment will now explain the above method in more detail.

[0059] In one embodiment, a synchronization system needs to be deployed on both the source and target databases. The source database synchronization system reads logs from the source database and parses them into operations. It then generates an MD5 value based on the table ID and primary key, or the table ID and unique index information, or the table ID and custom selected columns in the DML operation, as the key for the operation. The target database synchronization system is responsible for classifying and collecting the synchronization operations sent from the source by transaction, and using the key of each received operation to set its dependency on conflicting transactions received before it.

[0060] It should be noted that when calculating the operation key, since the column order in the operation is not fixed when log parsing, the columns in the operation must first be sorted by column ID, and then the key is calculated according to the fixed order of the primary key or unique index combination columns. In addition, the source-side data synchronization service can detect conflicts for each operation and more accurately calculate the conflicting MD5 value. If the target-side data synchronization service calculates the MD5 value based on the primary key information of the target database table, if the primary key structure or column order of the target database table is inconsistent with the source, the MD5 value calculated based on the primary key information of the target table will be inaccurate.

[0061] In one embodiment, after the target-side data synchronization service is started, it is necessary to initialize a hash table R of size N and initialize the record address in the hash item to empty to store and locate the received operations; and create two memory slices of memory space size M to store the KEY in the hash table R. The version number of each memory slice is incremented in the order of creation, and the two memory slices are added to the memory linked list.

[0062] For hash table R, the structure of the HASH item in its HASH table is a 4-byte version number, set as K; a 4-byte memory offset, set as F. K and F constitute a HASH item in the HASH table, as shown in the following table:

[0063] Version number (K) Memory offset (F)

[0064] It indicates that the first KEY in the HASH item is stored in the memory slice with version number K, and its offset in the memory slice is F.

[0065] In one embodiment, after the target-side data synchronization service is started, a hash table Q for registering transaction information needs to be initialized to store the commit status of all transactions within the LSN range of the received conflict detection. The hash table Q stores transaction states in the following three categories: Active state: The transaction has not received a commit or rollback operation at the time of the current detection. Committed state: The transaction has received a commit operation at the time of the current detection. Rollback state: The transaction has received a rollback operation at the time of the current detection.

[0066] In one embodiment, after the target-side data synchronization service receives the DML operation from the source side, the transaction ID of the extraction operation is registered in the hash table Q, and the transaction status is active.

[0067] In one embodiment, after receiving an operation from the source, the target-side data synchronization service first determines whether the difference between the current operation's LSN and the last checkpoint's LSN is greater than or equal to N. If so, operations between the last checkpoint's LSN and the current LSN - 1 are categorized as transactions and saved in a checkpoint file. The checkpoint's starting LSN is set to the last checkpoint's LSN + 1, and its ending LSN is set to the current operation's LSN - 1. It should be noted that categorizing operations saved in the checkpoint file by transaction facilitates rapid retrieval of transaction operation data during transaction execution.

[0068] In one embodiment, after capturing a DML log operation, the source-side data synchronization service retrieves the table ID from the log and uses it to retrieve the table's primary key and unique index from the source database. After parsing the current DML log operation, the source-side data synchronization service determines whether the table has a primary key or unique index. If so, it combines the table ID with the column values ​​of the primary key or unique index to calculate an MD5 value, which serves as the key for the operation. This value is stored with the current DML log operation and sent to the target-side synchronization service. It should be noted that when a table has both a primary key and a unique index, the combined MD5 value of the table ID and the primary key is preferred as the key. When a table has no primary key but multiple unique indexes, the combined MD5 value of the table ID and any unique index is preferred. If neither a primary key nor a unique index exists, after parsing the current DML log operation, the source-side data synchronization service combines the table ID with the row's ROWID value or a custom selected column to calculate an MD5 value as the key for the operation. This value is stored with the current DML log operation and sent to the target-side synchronization service. It should be noted that when calculating the KEY for a table without a primary key or unique constraint, if it is known that there are no duplicate values ​​in the combination of all the table's columns, the table's ROWID can be used directly to calculate the KEY, which can effectively reduce the cost of calculating the KEY. If the table has duplicate rows, it is necessary to use the custom column selection method to calculate the KEY. You can choose which columns or all column values ​​to use for calculating the KEY based on the distribution of column values. In this way, the target-side synchronization service can use the KEY to prevent deadlocks caused by concurrent synchronization transactions operating on rows with duplicate values.

[0069] In one embodiment, after the target-side data synchronization service receives the DML operation from the source, it extracts the MD5 value calculated by the source in the operation as the KEY, then extracts the transaction ID in the operation as the TRXID, combines the KEY and TRXID into a record and saves it to the offset F of the memory slice with the current version number K. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice, which is also the conflicting LSN of the operation in the memory slice.

[0070] Since the hash record needs to save the address of the next record, the record saved in the memory chip also needs to reserve the address of the next record, as shown in the following table.

[0071] KEY TRXID Next record memory chip version (K) Next record memory offset (F)

[0072] In one embodiment, the detection record X is constructed using the current operation's KEY, and the above detection record X is merged into the hash table R. During the merging process, conflict detection is required. If there is a conflicting KEY, the TRXID value corresponding to the conflicting KEY is returned; otherwise, 0 is returned as the conflicting TRXID, indicating no conflict. For details, refer to Figure 2 As shown, the conflict detection and judgment specifically includes the following steps.

[0073] Step 101: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value.

[0074] Step 102: Determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, backfill the hash item with the address of the detection record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection.

[0075] Step 103: Determine whether the memory slice version number K in the address recorded in the hash item is less than the version number K value of the first memory slice in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the record address in the hash item to the next address space of the current detection record X, and backfill the hash item with the address of the detection record X. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection.

[0076] Step 104: Load the corresponding record Y from the memory slice according to the address recorded in the hash item (find the memory slice with the corresponding version number through the record version number, and then locate the specific memory address through the offset), and determine whether the KEY of record Y is equal to the KEY of detection record X and the TRXID is not equal to the TRXID of the current operation. If so, it indicates that there is a conflict in the operation, remove record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, set the record address in the hash item to the next address space of the current detection record X, backfill the address of the detection record X into the hash item, and use the TRXID stored in record Y as the conflicting TRXID for return; if not, proceed to the next step of conflict detection.

[0077] Step 105: Search the detected conflicting TRXID in hash table Q to obtain the status of the conflicting transaction, and return the conflicting TRXID based on the conflicting transaction status. If the conflicting transaction is in the active state, -1 is returned, indicating that it may conflict with transactions whose commit LSN is smaller than the transaction to which the current operation belongs. It should be noted that when the conflicting transaction status detected by the operation conflict is active, this indicates that duplicate data exists in the source database row. The final status of the dependent active transaction is unknown, potentially being a rollback or a commit. Therefore, the execution order of the current operation cannot be determined. Therefore, the current operation must wait until all transactions with a commit LSN smaller than the current operation have completed before it can begin. If the conflicting transaction is in the rollback state, the transaction is ignored and the current hash entry is searched for other conflicting transactions. If a conflicting transaction is found, it is processed according to its status. Otherwise, 0 is returned as the conflicting TRXID, indicating that no conflict has been detected. The next record after record Y is traversed, and conflict detection is performed in the same manner as described above until the end of the record list. When the conflicting transaction is in the committed state: return the conflicting transaction ID as the conflicting TRXID.

[0078] Based on the above process, the next record after record Y is traversed in sequence, and conflict detection is performed in the same way as above until the end of the record list. The end condition of the list is: the version number in the next record address is less than the version number of the first memory slice in the current memory list, or the version number in the next record address is empty.

[0079] In one embodiment, the TRXID obtained by the conflict check is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN, thereby obtaining the conflict information of this operation.

[0080] In one embodiment, the target-side data synchronization service receives the next message and saves it to memory slice K as described above. There are two memory slices in the memory linked list. When the first memory slice is full (the written data exceeds the set size M), it switches to the second memory slice. When the second memory slice is full, the first memory slice in the linked list must be eliminated first, and then a new memory slice is created with an incremented version number K and added to the linked list. It should be noted that when saving a record to a new memory slice, if it is the first record in that memory slice, the operation LSN corresponding to the record must be marked as the starting LSN of the memory slice.

[0081] In one embodiment, upon receiving a commit operation, the target-side data synchronization service adds the transaction to the pending transaction queue and modifies the transaction status in hash table Q from active to committed. It should be noted that since only committed transactions are executed and stored, dependent transactions must also be committed during conflict detection. Therefore, upon receiving a transaction commit operation, the transaction status must be clearly marked in hash table Q as committed.

[0082] In one embodiment, the target-side data synchronization service receives a rollback operation, discards the transaction, and modifies the transaction status in the hash table Q from the active state to the rolled-back state. It should be noted that when a received transaction is rolled back, since the rolled-back transaction operation will not be executed when the data is stored, no committed transaction operation is allowed to depend on a rolled-back transaction. Otherwise, the dependent conflicting transaction will not be found during execution and it will be considered that there is no conflict, causing the subsequent transactions to be executed in advance, causing an error. For example, there are three transactions A, B, and C that all modify the same row of the same table, but transaction B is rolled back. When the target-side data synchronization service receives the operation, the operation conflict detection process will form a dependency order relationship in which B depends on A and C depends on B. Transaction B is directly discarded and not executed because it is rolled back. Then, when C is executed, it will not find the dependent transaction B and will be executed in advance. Finally, transaction A and transaction C will be executed concurrently, causing an error. Therefore, after receiving a rollback operation, the transaction needs to clearly mark the transaction status as rollback in the hash table Q. When the subsequent operations are found to conflict with the rollback transaction during conflict detection, the rollback transaction must be ignored and the search for conflicting transactions with a committed status must continue.

[0083] In one embodiment, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to write the conflicting TRXIDs saved in the rolled-back operation back to the conflict detection HASH table, i.e., the hash table R. It should be noted that since the transaction dependency information of the rolled-back operation in the rollback transaction and the partial rollback operation is restored, the dependency relationship of each transaction received in sequence is clear. These transactions can be committed immediately when they are submitted, without having to wait for the transaction with a smaller LSN than their own to be committed, thereby further improving the synchronization performance. For the write-back of the conflicting TRXIDs of the rollback operation, refer to Figure 3 As shown, the specific steps include the following.

[0084] Step 201: Locate the transaction that needs to be rolled back using the transaction ID of the rollback or partial rollback operation, extract the number of rows that need to be rolled back in the partial rollback operation, or extract all rows involved in the transaction in the rollback operation.

[0085] Step 202: Extract the rollback operation from the tail of the transaction operation linked list, and extract the KEY value, conflicting TRXID, and conflicting LSN saved in the operation.

[0086] Step 203: Compare the conflicting LSN of the operation with the starting LSN of the first memory slice. If the conflicting LSN is smaller than the starting LSN, terminate the write-back of the partial rollback operation to the hash table R; otherwise, jump to the next step.

[0087] Step 204: Search for conflicting operation records with the same KEY in the HASH table based on the operation KEY. If not found, terminate the writeback of this part of the rollback operation to the hash table R; otherwise, write back the conflicting TRXID to the record in the hash table R, and continue to extract the previous record to be rolled back and perform the same writeback until the specified number of rows is rolled back.

[0088] By writing back the conflicting TRXID information in the rollback operation, the continuity problem of conflict detection can be solved. These rolled-back operations can be discarded directly. Otherwise, when the transaction is executed, although the partially rolled-back operation does not need to be executed, conflict detection is still required during execution. Otherwise, other transaction operations that depend on it will not be able to find the dependent conflicting transaction and will be executed prematurely, resulting in data inconsistency.

[0089] In one embodiment, after receiving a transaction commit operation, the target-side data synchronization service asynchronously stores the transaction. After storage is complete, the target-side data synchronization service determines whether the transaction commit LSN is in the most recent checkpoint file, based on the condition that the checkpoint file's start LSN <= commit LSN <= checkpoint file's end LSN. If so, the transaction ID is appended to the checkpoint file. It should be noted that during failure recovery, the target-side data synchronization service only restores the most recent checkpoint file, recording transactions that have been stored within the checkpoint file's range. This can speed up failure recovery and reduce the size of the conflict detection key after failure recovery.

[0090] In one embodiment, when the system fails and restarts, it is necessary to use the last checkpoint file to restore the KEY in the conflict detection container, that is, to restore the KEY in the hash table R. For details, refer to Figure 4 As shown, the process of recovering the KEY of the hash table R specifically includes the following steps.

[0091] Step 301: Get the most recent checkpoint file F, where the starting LSN of the checkpoint file F is LSN1 and the ending LSN is LSN2; create an array A of size N.

[0092] Step 302: Read the ID information of executed transactions stored in checkpoint file F for filtering during recovery (completed transactions are no longer relevant for recovery after a failure); read the transaction operation OP stored in checkpoint file F, extract the operation's transaction ID, and determine whether the transaction ID has been completed. If so, abandon the operation; otherwise, extract the LSN value of operation OP and store it in array A according to the algorithm A[LSN-LSN1]=OP. It should be noted that OP refers to the operator used to perform a specific operation. It should also be noted that since the operations stored in checkpoint file F are categorized by transaction, the operations read from checkpoint file F are also categorized by transaction. However, the operation LSNs between transactions are discontinuous, so transactions recovered from the checkpoint need to be sorted by transaction operation LSN. Since the transaction operations stored in checkpoint file F are confined to a limited LSN range (size N), an array can be used to sort the operations in the checkpoint file.

[0093] Step 303: Extract the operation OP and determine whether it is a DML operation. If so, mark the transaction as active in the hash table Q based on the TRXID of the operation. Otherwise, determine whether it is a commit operation. If so, mark the transaction as committed in the hash table Q based on the TRXID of the operation. Otherwise, mark the transaction as rollback in the hash table Q based on the TRXID of the operation.

[0094] Step 304: Read the operations in the order of array A, extract the KEY and transaction ID in the operations to restore the state of the hash table R before the failure.

[0095] In one embodiment, after the restart is completed, operations from the source end are continued to be received, and operations whose received operation LSN is less than or equal to the recovery checkpoint file end LSN are discarded. For operations whose received operation LSN is greater than the recovery checkpoint file end LSN, the KEY in the operation is extracted according to the above method for conflict detection.

[0096] The above describes the fault recovery method for setting transaction operation dependencies. The following describes the workflow of the target-side data synchronization service after setting transaction operation dependencies.

[0097] In one embodiment, the target-side data synchronization service creates H worker threads, executes transactions in order of their commit LSNs, and stores the transaction operations. It should be noted that the target-side data synchronization service also creates a hash table P to record the currently executing transaction information (transaction ID) and a hash table G to record the current transaction's table information and the transaction's start LSN.

[0098] In one embodiment, the worker thread extracts a transaction from the committed transaction list in the order in which the transactions were committed, registers the transaction ID in the hash table P, and registers the table information involved in the transaction and the transaction start LSN in the hash table G.

[0099] In one embodiment, a worker thread extracts an operation from a transaction and determines whether the current operation is a commit operation. If so, it commits the current transaction, wakes up its own waiting worker thread, removes the current transaction's registration information from hash tables P and G, and completes the current transaction. Otherwise, it retrieves the conflicting TRXID and conflicting LSN stored in the operation. It should be noted that since the transactions that each operation in a transaction depends on have been determined during conflict detection, a transaction can be committed directly after executing all operations (indicating that all dependent transactions have been committed), without having to wait for transactions that submitted earlier than it, thereby effectively improving synchronization performance.

[0100] In one embodiment, the worker thread uses the conflicting TRXID of the currently pending operation to search hash table P for a transaction with the conflicting TRXID. If so, the worker thread suspends the current execution process and waits for the conflicting transaction to complete and commit before waking up again. Otherwise, the worker thread jumps to the next step. It should be noted that the presence of a transaction with the conflicting TRXID in hash table P indicates that the data involved in the current operation conflicts with the transaction with the conflicting TRXID. Therefore, the current operation cannot be executed in parallel with the conflicting transaction and must wait until the conflicting transaction commits before it can begin execution.

[0101] In one embodiment, a worker thread uses the information in the current pending operation table to search hash table G for a transaction operating on the same table, whose commit LSN is smaller than the current transaction and whose start LSN is smaller than the conflict LSN of the current operation. If so, the worker thread suspends the current execution process and waits for the conflicting transaction to complete and commit before waking up. Otherwise, if the conflict detection for the worker thread's current pending operation does not detect a conflict, the worker thread executes the current operation and then proceeds to the next operation. It should be noted that this step primarily checks whether there are transactions in the currently executing concurrent transactions that exceed the conflict check LSN range. This is because the conflict transaction information for pending operations is valid only within a certain range, from the conflict LSN stored in the operation to the current operation LSN. The operation does not perform conflict detection on operations prior to the conflict LSN. In other words, if a transaction exists with a commit LSN smaller than the commit LSN of the transaction to which the current pending operation belongs and a start LSN smaller than the conflict LSN stored in the current pending operation, it is unknown whether the operations in this transaction conflict with the current pending operation. Therefore, the current pending operation cannot be executed rashly and must wait until the transaction commits.

[0102] Based on the above process, this embodiment mainly performs conflict detection on operations within a fixed LSN interval, and saves the detection results and LSN interval to the corresponding operation. When the working thread executes the operation, it can use the conflict transaction information saved on the operation and its detection LSN interval to determine whether there is a conflict in the currently concurrently executed transaction, thereby improving the parallel execution capability.

[0103] refer to Figure 5 As shown, this embodiment also provides a fault recovery system for setting transaction operation dependency, which applies the aforementioned fault recovery method for setting transaction operation dependency. The system includes a checkpoint setting module, a commit LSN judgment module, and a fault restart module, wherein: the checkpoint setting module is used to judge whether the difference between the LSN of the current operation and the LSN value of the last checkpoint is greater than or equal to a preset value N after the target-side data synchronization service receives the operation of the source side. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1 and the ending LSN is set to the LSN-1 of the current operation; the commit LSN judgment module is used to judge whether the difference between the LSN of the current operation and the LSN value of the last checkpoint is greater than or equal to a preset value N after the target-side data synchronization service receives the operation of the source side. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1 and the ending LSN is set to the LSN-1 of the current operation; The fault module is used to asynchronously execute and store transactions after the target-side data synchronization service receives a transaction commit operation. After storage is complete, it determines whether the commit LSN of the transaction commit operation is in the most recent checkpoint file, based on the checkpoint file's start LSN <= commit LSN <= end LSN. If so, the transaction ID is appended to the checkpoint file. The fault restart module is used to restore the keys in hash table R using the last checkpoint file after a system fault restart. After the restart is complete, it continues to receive operations from the source side, discarding operations with an LSN less than or equal to the checkpoint file's end LSN. For operations with a LSN greater than the checkpoint file's end LSN, the key in the operation is extracted for conflict detection. The specific functions and principles of this system functional module correspond to the specific methods in the method section and will not be elaborated here.

[0104] In summary, this embodiment provides a fault recovery method and system for setting transaction operation dependencies, which has the following advantages: First, the target-side data synchronization service sets an interval LSN value of N and stores received operations in shards with a maximum of N. When the fault is restarted, only the operations in the most recently saved shard need to be used to restore the key information of hash table R and the transaction status in hash table Q before the fault. This allows for seamless connection between post-failure operations and pre-failure operations conflict detection, thereby ensuring that the fault does not affect the parallel execution capability of transactions before and after the fault.

[0105] Secondly, after a transaction is completed, if the transaction's commit LSN falls within the start and end LSN range of the most recent checkpoint file, the transaction ID of that transaction is appended to the most recent checkpoint file. When the target-side data synchronization service uses the most recent checkpoint file to restore the pre-failure hash table R information, it can filter out completed transactions, speeding up recovery. Filtering out completed transactions also helps reduce the size of the key information in the restored hash table R, thereby improving the performance of conflict detection in the first shard received.

[0106] Example 2:

[0107] Based on the fault recovery method for setting transaction operation dependency provided in the above embodiment 1, embodiment 2 of the present invention provides a specific example for illustration.

[0108] For example: The source database has table T(ID INT PRIMARY KEY, C1 INT);

[0109] The source application has three transactions that perform the following operations on table T in the following order:

[0110] TRX1:INSERT INTO T(ID,C1)VALUES(1,1);

[0111] TRX1:COMMIT;

[0112] TRX2:INSERT INTO T(ID,C1)VALUES(2,2);

[0113] TRX2:UPDATE T SET C1=1WHERE ID=1;

[0114] TRX2:COMMIT;

[0115] TRX3:INSERT INTO T(ID, C1)VALUES(3, 3);

[0116] TRX3:COMMIT;

[0117] The order of the above operations represents the order of these operations in the log stream, and the resulting log stream will form the following numbered table:

[0118]

[0119]

[0120] The receiving process is as follows:

[0121] 1. The target-side data synchronization service sets the checkpoint LSN interval preset value N to 5 and creates a hash table R that can accommodate 5*2=10 KEY information. Since this is the first startup, the LSN of the previous checkpoint is set to 0.

[0122] 2. The log receiving thread receives an operation with LSN 1. The difference between LSN 1 and the previous checkpoint LSN 0 is less than 5. Its KEY and TRXID are extracted and stored in the hash table R for conflict detection. The conflict detection result is no conflict.

[0123] 3. The log receiving thread receives an operation with LSN 3. The difference between LSN 3 and the previous checkpoint LSN 0 is less than 5. Its KEY and TRXID are extracted and stored in hash table R for conflict detection. The conflict detection result is no conflict.

[0124] 4. The log receiving thread receives an operation with LSN 4. The difference between LSN 4 and the previous checkpoint LSN 0 is less than 5. Its KEY and TRXID are extracted and stored in hash table R for conflict detection. The conflict detection result shows that there is a conflict with the transaction with TRXID 1. The conflicting TRXID is written back to the operation.

[0125] 5. The log receiving thread receives a commit operation with LSN 5. If the difference between LSN 5 and the previous checkpoint LSN 0 is greater than or equal to 5, the currently received sharding operation needs to be saved in the checkpoint file F, and the starting LSN of the checkpoint file is marked as 1 (the ending LSN 0+1 of the previous checkpoint) and the ending LSN is 4 (LSN 5-1 of the current operation).

[0126] 6. When the target-side data synchronization service receives a commit operation with LSN 2 and TRXID 1, it executes the transaction asynchronously. After the transaction is executed, since the commit LSN of the transaction falls within the start and end LSN range of the most recent checkpoint file, the TRXID of the transaction is appended to the most recent checkpoint file F.

[0127] 7. Restart the target data synchronization service and use the last checkpoint file to restore the key in the conflict detection container.

[0128] 8. Get the most recent checkpoint file F, whose starting LSN is 1 and ending LSN is 4.

[0129] 9. Create an array A of size 5.

[0130] 10. Read the completed transaction information in the checkpoint file F, including the transaction with transaction ID 1.

[0131] 11. Read the operations in the checkpoint file F by transaction classification. The transaction with TRXID 1 is discarded. The transaction with TRXID 2 and the two operations with LSNs 3 and 4 are stored in array A at positions A[2] and A[3] respectively after subtracting the checkpoint file starting LSN 1 from the operation LSN.

[0132] 12. Sequentially read the KEY information of the operations in array A and store it in hash table R to restore the state of hash table R before the failure.

[0133] 13. After the fault recovery is completed, the recovery checkpoint file is received again to terminate the operations after the LSN.

[0134] 14. The log receiving thread receives an operation with LSN 6. The difference between LSN 6 and the previous checkpoint LSN 5 is less than 5. Its KEY and TRXID are extracted and stored in hash table R for conflict detection. The conflict detection result is no conflict. In this way, this transaction can be executed in parallel with the transaction TRX2 before the failure. Otherwise, it must wait until the execution of TRX2 before the failure is completed before it can start execution, because it cannot confirm whether this operation conflicts with the transaction operation before the failure.

[0135] From the conflict detection and fault recovery process above, we can see that the operations of TRX1 and TRX2 are saved in the checkpoint file F of the first shard. Since TRX1 has received the commit operation, it will be executed asynchronously. When it is completed, its commit LSN will fall within the start LSN and end LSN range of the most recent checkpoint file F, and TRX1's transaction information will be appended to the checkpoint file F. When the fault is recovered, the operation of TRX1 will be abandoned, and the transaction operation of TRX2 will be sorted on array A using LSN. Restoring the order of receipt of the operations in the checkpoint file can ensure the correctness and completeness of conflict detection. Due to the filtering of the executed transaction TRX1, the hash table R after the fault recovery will have one less key information than the hash table R before the fault. The hash table R with reduced key information will have better performance in conflict detection. At the same time, TRX3 received after the fault can also be executed concurrently with the transaction TRX2 before the fault, without reducing the degree of parallelism due to the fault.

[0136] Example 3:

[0137] It should be noted that for rollback transactions, since the rollback transaction will not be executed, there are three ways to handle the rollback transaction operation in the hash table R.

[0138] The first approach is to not roll back the impact of the rolled-back transaction's operations on the operation dependencies in hash table R, nor collect the conflicting TRXIDs of the rolled-back transaction's dependencies. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies. When this approach is adopted, the rolled-back transaction's operations are directly released as a whole, which can improve the processing performance of rolled-back transactions. However, operations that depend on the rolled-back transaction must wait until the rolled-back transaction is released from the pending execution list before they can be executed. Since conflicting transactions with operation dependencies were not collected before the rollback transaction was released, the rollback transaction must wait until all transactions with commit LSNs or rollback LSNs smaller than its own rollback LSN have completed execution before it can be released, which affects synchronization efficiency. This approach corresponds to the following: when the target-side data synchronization service receives a rolled-back transaction, all operations in the transaction are discarded without execution, and only the retained transaction information is added to the pending execution queue. For example, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to locate the transaction based on the transaction ID in the operation, and then mark the transaction to indicate that some operations in the transaction are rolled back. If the operation is a rollback operation, then all operations in the transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed. The transaction information of the rolled back transaction cannot be discarded.

[0139] The second approach does not roll back the impact of the rolled-back transaction on the dependencies in hash table R. Instead, the conflicting TRXIDs of the rolled-back transaction's dependencies are collected, for example, recorded in the conflicting TRXID dependency list L. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending list for execution to ensure the continuity of the dependencies. When this approach is adopted, the conflicting TRXIDs in the rolled-back transaction are collected before the operation is released. Although this reduces the processing performance of the rolled-back transaction, it improves the release performance of the rolled-back transaction during execution. Operations that depend on the rolled-back transaction must wait until the rolled-back transaction is released from the pending list before they can be executed. Since the conflicting transactions of the rolled-back transaction were collected during the release operation, the rolled-back transaction can be released only after the execution of these conflicting transactions, without having to wait for the completion of all transactions with a commit LSN or a rollback LSN smaller than the rollback LSN. This partially improves synchronization efficiency compared to the first approach. This method corresponds to that when the target-side data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation. For the rollback operation, the corresponding transaction cannot be discarded, but is added to the queue of pending transactions.

[0140] The third method is to roll back the impact of the rolled-back transaction on the dependencies of the operations in hash table R. In this way, the rolled-back transaction will not be dependent on other operations, and the rolled-back transaction does not need to be added to the pending execution list. When this solution is adopted, after receiving the rollback operation, the transaction needs to roll back the impact of all operations in the transaction on the dependencies of the operations in hash table R. Because the transaction scale is uncontrollable, when a large transaction is rolled back, the rollback action will affect the progress of receiving the operation, thereby affecting the efficiency of synchronization. Corresponding to this method, when the target-side data synchronization service receives a rollback or partial rollback operation, it writes the conflicting TRXID saved in the rolled-back operation back to hash table R.

[0141] The three aforementioned transaction rollback solutions each have their advantages and disadvantages. The first solution is suitable for large-scale transaction rollbacks, the second for medium-scale transactions, and the third for small-scale transactions. Therefore, two parameters, W1 and W2, are set: W1 represents the minimum number of DML operations in large-scale transactions, and W2 represents the minimum number of DML operations in medium-scale transactions. Based on this, after receiving DML operations from the source, the target-side data synchronization service must manage these operations by transaction classification and calculate the number of DML operations received per transaction, S (S being the number of DML operations received in a single transaction). When the target-side data synchronization service receives a rollback operation, it takes appropriate action based on the number of DML operations in the transaction, S.

[0142] When the size S of the rolled-back transaction is greater than or equal to W1, it corresponds to the large-scale transaction rollback in the first solution. In this case, the impact of the rolled-back transaction's operations on the operation dependencies in hash table R is not rolled back, and the conflicting TRXIDs of the rolled-back transaction's operation dependencies are not collected. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations and must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies.

[0143] When the size S of the rolled-back transaction is greater than or equal to W2 and less than W1, it corresponds to the medium-sized transaction rollback in the second solution. In this case, the impact of the rolled-back transaction's operations on the operation dependencies in hash table R is not rolled back. However, the conflicting TRXIDs of the rolled-back transaction's operation dependencies are collected and recorded in the conflicting TRXID dependency list L. This rolled-back transaction will be dependent on other operations and must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies.

[0144] When the size S of the rolled-back transaction is smaller than W2, it corresponds to the small-scale transaction rollback in the third solution. In this case, the impact of the rolled-back transaction operation on the operation dependencies in the hash table R is rolled back, and the rolled-back transaction does not need to be added to the pending execution list and is directly released.

[0145] Furthermore, for partial rollback operations, due to their special nature, they inherently require traversing the rolled-back operations. Therefore, adding the process of collecting conflicting TRXIDs for these operations does not significantly reduce the performance of handling partial rollbacks. Therefore, when the target-side data synchronization service receives a partial rollback operation, it counts the total number of rolled-back rows S1 and, based on the size of S1, performs the following processing: When the number of rolled-back rows S1 is less than W2, the impact of the rollback operation on the operation dependencies in hash table R is rolled back, that is, the conflicting TRXIDs stored in the rolled-back operation are written back to hash table R. When the number of rolled-back rows S1 is greater than W2, the impact of the rolled-back transaction on the operation dependencies in hash table R is not rolled back, that is, the conflicting TRXIDs stored in the rolled-back operation are not written back to hash table R. However, the conflicting TRXIDs of the rolled-back transaction operation dependencies are collected and recorded in the conflicting TRXID dependency list L.

[0146] Based on the above settings, after the target-side data synchronization service creates a worker thread, the worker thread extracts a transaction from the committed transaction list in the order in which the transactions were committed, and determines whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to determine whether the rollback transaction has a conflicting TRXID list L.

[0147] If the rolled-back transaction does not conflict with the TRXID linked list L, it must wait for all transactions with a commit LSN lower than the current transaction to complete before waking up the waiting worker thread. It then removes the current transaction's registration information from hash tables P and G, completing the current transaction. This is because the operations on the rolled-back transaction are no longer needed and are discarded, so the information on the transactions they depend on is unclear. Therefore, when committing, it must wait until all transactions submitted earlier than the current transaction have completed their commit to ensure the correct order of transaction dependencies.

[0148] If the rolled back transaction has a conflicting TRXID linked list L, it is necessary to wait for all conflicting transactions on the conflicting transaction linked list L to commit before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and hash table G to complete the execution of the current transaction. Specifically, extract a conflicting TRXID from the current transaction conflicting transaction linked list L in turn, and check whether it exists in the hash table P. If it exists, it is necessary to wait for the transaction to be completed before waking itself up. After waking up, it extracts the next conflicting TRXID from the conflicting transaction linked list L and performs the same check until all conflicting transactions in the conflicting transaction linked list L are completed. This is because the operations on the rolled back transaction do not need to be executed and are discarded. When the rollback operation is discarded, the conflicting TRXIDs on the operations are collected, so the transaction information they depend on is very clear. Then, when releasing, it only needs to wait for these conflicting transactions to be committed to ensure the correctness of the transaction dependency order.

[0149] In summary, since the rollback transaction will not be executed, it will generate three schemes during conflict detection. Each scheme has its own advantages. It is necessary to set appropriate values ​​to give full play to the advantages of the three schemes in order to achieve the purpose of improving the overall synchronization performance. In addition, for the partial rollback operations of the committed transactions, the conflicting TRXIDs referenced by them are collected in a targeted manner during the rollback operation, which can effectively reduce the performance loss caused by the partial rollback operation writing back the conflict detection hash table R, so as to achieve the purpose of improving the overall synchronization performance. This embodiment processes the rollback transaction and partial rollback operations in a targeted manner according to the agreed operation scale, and finds the balance point between the cost of the operation rollback and the execution of the rollback transaction, thereby improving the overall synchronization performance.

[0150] In addition, in some embodiments, the KEY in the hash table R can be stored in a file or in a memory chip. Specifically, the difference between the two is that when using files for storage, although the storage performance is low and affects the efficiency of conflict detection, after a data synchronization failure, the state of the hash table R before the failure can be restored by storing the KEY value in the file. In this way, when performing conflict detection on the received operations after the failure, there is continuity with the failure, and the synchronization performance will not drop drastically due to the failure. When using memory chips for storage, the conflict detection performance will be greatly improved compared to the file storage solution, but the data of the previous conflict detection will be lost after the failure. When restoring the detection, it can only start from the currently received LSN, which will cause a discontinuity between the subsequent conflict detection and the failure. During synchronous execution, the synchronization performance will drop drastically at the failure location. When the storage performance is high (for example, using SSD fixed storage), it can meet the performance requirements of conflict detection, and the efficiency of operation reception can be balanced with synchronous execution. Then, the method of storing the KEY in the hash table R through a file can effectively avoid the loss of synchronization performance caused by failures. When the storage performance affects the efficiency of conflict detection, the bottleneck of synchronization exists in the operation reception. At this time, the method of storing the KEY in the hash table R through a memory chip can effectively improve the reception performance of the operation, thereby improving the overall synchronization performance. Example 1 of the present application is an example of a method of storing the KEY in the hash table R through memory.

[0151] In some embodiments, you can choose to calculate the MD5 value as the operation key on either the source or target side. The source-side data synchronization service detects conflicts for each operation, allowing for a more accurate calculation of the conflicting MD5 value. However, if the target-side data synchronization service calculates the MD5 value based on the target database table's primary key information, errors may occur if the target database table's primary key structure or column order differs from that on the source side. For this reason, the MD5 value is typically calculated on the source side as the operation key. Upon receiving the DML operation from the source side, the target-side data synchronization service directly extracts the MD5 value calculated by the source side as the key.

[0152] In some embodiments, when calculating the KEY of an operation, since the order of columns in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the KEY is calculated according to the fixed order of the primary key or unique index combination columns. After the source-side data synchronization service captures a DML log operation, it obtains the table ID in the log and uses the table ID to obtain the primary key information and unique index of the table in the source-side database. After parsing the current DML log operation, the source-side data synchronization service determines whether the table has a primary key or unique index. If there is a primary key or unique index, the table ID and the column values ​​of the primary key or unique index are combined to calculate an MD5 value as the KEY for this operation. This value is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when a table has both a primary key and a unique index, the table ID and the primary key are preferably combined to calculate the MD5 value as the KEY; when a table has no primary key but has multiple unique indexes, the table ID and any unique index are combined to calculate the MD5 value. If there is no primary key or unique index, after parsing the current DML log operation, the source-side data synchronization service combines the table ID and the row's ROWID value or custom selected columns to calculate an MD5 value as the key for this operation. This value is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when calculating the key for a table without a primary key or unique constraint, if it is clear that all columns in the table have no duplicate values, the table's ROWID can be used directly to calculate the key, effectively reducing the cost of calculating the key. If the table has duplicate rows, custom selected columns are required to calculate the key. The key can be calculated based on the distribution of column values, which columns or all column values ​​are used to calculate the key. This allows the target-side synchronization service to use the key to prevent deadlocks caused by concurrent synchronization transactions operating on rows with duplicate values.

[0153] Example 4:

[0154] Based on the fault recovery method for setting transaction operation dependency provided in the above embodiment 1, the present invention also provides a fault recovery device for setting transaction operation dependency that can be used to implement the above method and system, such as Figure 6 FIG. 1 is a schematic diagram of the device architecture of an embodiment of the present invention. The fault recovery device for setting transaction operations in this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0155] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0156] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the method for recovering from a fault in which transaction operations are dependent on each other in Example 1. The processor 21 executes the non-volatile software programs, instructions, and modules stored in the memory 22 to execute various functional applications and data processing of the apparatus for recovering from a fault in which transaction operations are dependent on each other, thereby implementing the method for recovering from a fault in which transaction operations are dependent on each other in Example 1.

[0157] The memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21, and such remote memory may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the program instructions / modules execute the above-described method for recovering from a failure in the embodiment 1. For example, the above-described method is executed. Figure 1-Figure 4 The steps shown.

[0159] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0160] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0161] Through the description of the above embodiments, it can be clearly understood by those skilled in the art that each embodiment can be implemented by means of software plus a general hardware platform, or of course by hardware. It can be understood by those skilled in the art that all or part of the processes in the above embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above. For the sake of simplicity, they are not provided in detail. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fault recovery method for setting transaction operation dependency, characterized in that: include: After receiving the operation from the source, the target-side data synchronization service determines whether the difference between the current operation's LSN and the last checkpoint's LSN is greater than or equal to the preset value N. If so, the operations between the last checkpoint's LSN and the current LSN-1 are grouped as transactions and saved in the checkpoint file. The starting LSN of the checkpoint is set to the last checkpoint's LSN+1, and the ending LSN is set to the current operation's LSN-1. After receiving the transaction commit operation, the target-side data synchronization service asynchronously executes the transaction into the database. After the storage is completed, it determines whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The checkpoint file's start LSN <= commit LSN <= end LSN is used for the check. If the checkpoint file's start LSN <= commit LSN <= end LSN is satisfied, the transaction ID of the transaction is appended to the checkpoint file. When the system fails and restarts, the last checkpoint file is used to restore the key in the hash table R. After the restart is complete, the source end's operations are continued to be received. Operations with an LSN less than or equal to the end LSN of the checkpoint file are discarded. Operations with an LSN greater than the end LSN of the checkpoint file are extracted and the key in the operation is detected and judged for conflicts. After the target-side data synchronization service is started, a hash table R is initialized to store and locate received operations. Two memory slices are created to store the keys in the hash table R. The version number of each memory slice is incremented in the order of creation. The two memory slices are added to the memory linked list. After the target-side data synchronization service receives the DML operation from the source, it extracts the MD5 value calculated by the source in the operation as the KEY, and then extracts the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved in the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice. The detection record X is constructed using the KEY of the current operation, and the detection record X is merged into the hash table R. During the merging process, conflict detection is performed. If there is a conflicting KEY, the TRXID value corresponding to the conflicting KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict. The TRXID obtained by the conflict detection is set as the conflicting TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation as the conflicting LSN.

2. The fault recovery method for setting transaction operation dependency according to claim 1, characterized in that: After the target-side data synchronization service is started, a hash table Q for registering transaction information is initialized to save the status of all transactions within the received conflict detection LSN range, including active status, committed status, and rollback status. After the target-side data synchronization service receives the DML operation from the source, the transaction ID of the extracted operation is registered in the hash table Q, and the transaction status is initially active.

3. The fault recovery method for setting transaction operation dependency according to claim 2, characterized in that: When the target-side data synchronization service receives a commit operation, it adds the transaction to the pending transaction queue and modifies the transaction status in the hash table Q from the active status to the committed status. When the target-side data synchronization service receives a rollback operation, it discards the transaction and modifies the transaction status in the hash table Q from the active status to the rollback status. It also writes the conflicting TRXID saved in the rolled-back operation back to the hash table R.

4. The fault recovery method for setting transaction operation dependency according to claim 1, characterized in that: The conflict detection and judgment specifically includes: Calculate the hash value of the KEY of the current detection record X and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, backfill the hash item with the address of the detection record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection and judgment; Determine whether the memory slice ID in the address recorded in the hash item is less than the ID value of the first memory slice in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and backfill the hash item with the address of the detection record X. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment; According to the address recorded in the hash item, the corresponding record Y is loaded from the memory chip, and it is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, and record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item. The record address in the hash item is set to the next address space of the current detection record X, and the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflict TRXID for return; if not, the detected conflict TRXID is searched in the hash table Q to obtain the status of the conflicting transaction, and the conflicting TRXID is returned according to the conflicting transaction status; Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record list.

5. The fault recovery method for setting transaction operation dependency according to claim 4, characterized in that: The step of searching the hash table Q for the detected conflicting TRXID to obtain the conflicting transaction status and returning the conflicting TRXID according to the conflicting transaction status specifically includes: If the conflicting transaction is active: -1 is returned as the conflict TRXID, indicating that it may conflict with transactions whose commit LSN is smaller than that of the transaction to which the current operation belongs; If the conflicting transaction is in the rollback state: ignore the transaction and continue searching for other conflicting transactions in the current hash item. If a conflicting transaction is found, continue to classify and process it according to the conflicting transaction status; otherwise, return 0 as the conflict TRXID, indicating that no conflict was detected. If the conflicting transaction is in the committed state: return the conflicting transaction ID as the conflicting TRXID.

6. The method for failure recovery according to any one of claims 1 to 5, wherein: When the system fails and restarts, the key in the hash table R is restored using the last checkpoint file, specifically including: Get the most recent checkpoint file F, the starting LSN of the checkpoint file F is LSN1, and the ending LSN is LSN2; create an array A of size N; Read the transaction ID information of the executed transaction in the checkpoint file F for filtering during recovery; read the transaction operation OP saved in the checkpoint file F, extract the transaction ID of the operation, and determine whether the transaction ID has been executed. If so, abandon the operation; otherwise, extract the LSN value of the operation OP and store it in array A according to the algorithm A[LSN-LSN1]=OP; Extract the operation OP and determine whether it is a DML operation. If so, mark the transaction as active in the hash table Q based on the TRXID on the operation. Otherwise, determine whether it is a commit operation. If so, mark the transaction as committed in the hash table Q based on the TRXID on the operation. Otherwise, mark the transaction as rolled back in the hash table Q based on the TRXID on the operation. Read the operations in the order of array A, extract the KEY and transaction ID in the operation to restore the state of hash table R before the failure.

7. A fault recovery device for setting transaction operation dependency, characterized in that: The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the fault recovery method for setting transaction operation dependency as described in any one of claims 1-6.

8. A fault recovery system for setting transaction operation dependency, used to execute the fault recovery method for setting transaction operation dependency according to any one of claims 1 to 6, characterized in that: The system includes a checkpoint setting module, a submission LSN determination module, and a fault restart module, wherein: The checkpoint setting module is used to determine whether the difference between the LSN of the current operation and the LSN of the last checkpoint is greater than or equal to a preset value N after the target-side data synchronization service receives the operation from the source side. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1 and the ending LSN is set to the current operation LSN-1. The commit LSN determination module is used to asynchronously execute and store the transaction after the target-side data synchronization service receives the commit operation of the transaction. After the storage is completed, it is used to determine whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The determination is made based on the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If the condition is met, the transaction ID is appended to the checkpoint file. The fault restart module is used to restore the KEY in the hash table R using the last checkpoint file after the system fault restarts; after the restart is completed, it continues to receive operations from the source end, discards operations whose operation LSN is less than or equal to the end LSN of the checkpoint file, and extracts the KEY in the operation for conflict detection and judgment if the operation LSN is greater than the end LSN.

9. A non-volatile computer storage medium, characterized in that The computer storage medium stores computer executable instructions, which are executed by one or more processors to implement the failure recovery method for setting transaction operation dependency according to any one of claims 1-6.

Images