Transaction operation parallel execution method, device and system based on mapping function container

Abstract

The invention relates to a transaction operation parallel execution method, device and system based on a mapping function container. The method mainly comprises the steps that after a target end data synchronization service is started, a container P with a mapping function is initialized and used for storing the mapping relation of a source end database ROWID in a target end database corresponding to the ROWID; determining a conflict TRXID and a conflict LSN of the operation according to the KEY, the TRXID and the initial LSN; and the target end data synchronization service creates a plurality of working threads, processes the operation according to the conflict TRXID and the conflict LSN of the operation, judges the type of the current operation when executing the current operation through the working threads, and performs classification processing based on the container P according to different types. According to the method, the concurrency of operation execution can be effectively increased, and the purpose of improving the synchronization performance is achieved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method, device and system for parallel execution of transaction operations based on a mapping function container. Background Art

[0002] In a database real-time synchronization system (supporting heterogeneous architecture) based on a log parsing architecture, the source-side data synchronization service is responsible for capturing the operation logs of the source database, parsing the logs to restore the corresponding database operations, and then sending them to the target-side data synchronization service. The target-side data synchronization service is responsible for executing these operations in the target database, thereby completing the data synchronization process.

[0003] When the target-side data synchronization service synchronizes operations on the target-side database, it often needs to execute them in the order in which the transactions were committed. This is because when the previous transaction has not yet completed synchronization, the operation of the next transaction cannot be executed and stored concurrently with the previous transaction without clarifying whether it conflicts with the previous transaction. Execution can only begin after all transactions preceding the current operation have been committed. This greatly affects the concurrency of transaction storage. The current effective method is to determine whether parallel execution is possible by judging the size of the log LSN of the operation and the LSN of the previously submitted transaction commit operation. If the LSN of the currently executing transaction operation is smaller than the LSN of the commit operation of another transaction currently being executed, then this operation can be executed concurrently with that transaction. Although this method of parallel execution based on the order in which transactions are committed improves performance, the granularity of conflict detection is large and not detailed enough, resulting in low parallelism and a small performance improvement.

[0004] In view of this, how to overcome the defects of existing technologies and how to improve the parallel execution of target-side data synchronization services to improve data synchronization performance have become important technical issues that need to be urgently addressed in the industry. Summary of the Invention

[0005] In response to the defects or improvement needs in the existing technology: How to improve the parallel execution strength of the target-side data synchronization service to improve the performance of data synchronization. The present invention believes that if the dependency between transaction operations can be determined, it can be clear which transactions need to be waited for during the synchronization operation, and the parallelism of synchronization can be maximized. Based on this, the present invention provides a method, device and system for parallel execution of transaction operations based on a mapping function container. When receiving synchronization transaction operations, the target-side data synchronization service of this method receives them in the order in which the source-side database log is generated. Then the order of each received operation is the basis for them to detect whether there is a conflict. The embodiment of the present invention extracts the ROWID of each received operation based on the order in which the operation is received, and finds whether this ROWID is the same as the ROWID of the operation previously received from other transactions within the set range. If so, the execution of this operation needs to wait for the conflicting transaction to be committed before it can start execution; otherwise, it can be executed concurrently with the transaction submitted first, thereby maximizing the parallelism of transaction execution, thereby effectively improving the performance of synchronization.

[0006] The present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for parallel execution of transaction operations based on a mapping function container, comprising:

[0008] After the target-side data synchronization service is started, a mapping container P is initialized to store the mapping relationship between the source-side database ROWID and the corresponding ROWID in the target-side database.

[0009] After receiving the DML operation from the source, the target-side data synchronization service extracts the key calculated from the table ID and row ID in the operation, then extracts the transaction ID in the operation as the TRXID, combines the key and TRXID into a record and saves it to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice.

[0010] Determine the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID and starting LSN;

[0011] The target-side data synchronization service creates several worker threads to process operations based on their conflicting TRXIDs and conflicting LSNs. When executing the current operation through the worker thread, the type of the current operation is determined and classified based on the container P according to the type.

[0012] In some embodiments, after the target-side data synchronization service is started, a hash table R is initialized, and the record address in the hash item of the hash table R is initialized to empty, which is used to store and locate the received operations; the target-side data synchronization service creates two memory slices to store the KEY in the hash table R, and each memory slice is incremented in the order of creation to set the version number, and the two memory slices are added to the memory linked list.

[0013] In some embodiments, the generation of the KEY specifically includes: generating an MD5 value as the KEY of the operation based on the table ID and ROWID in the DML operation.

[0014] In some embodiments, determining the conflicting TRXID and the conflicting LSN of the operation based on the KEY, the TRXID, and the starting LSN specifically includes:

[0015] Use the KEY of the current operation to construct the detection record X and merge the detection record X into the hash table R;

[0016] During the merging process, conflict detection is performed. If there is a conflicting key, the TRXID value corresponding to the conflicting key is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict.

[0017] The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

[0018] In some embodiments, the conflict detection judgment specifically includes:

[0019] Calculate the hash value of the KEY of the current detection record X and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, backfill the hash item with the address of the detection record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection and judgment;

[0020] Determine whether the memory chip version number in the address recorded in the hash item is less than the version number value of the first memory chip in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and backfill the hash item with the address of the detection record X. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment;

[0021] According to the address recorded in the hash item, the corresponding record Y is loaded from the memory chip. It is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation. Record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item. The record address in the hash item is set to the next address space of the current detection record X, the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflicting TRXID for return; if not, the next step of conflict detection is carried out;

[0022] Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record list.

[0023] In some embodiments, the memory linked list includes two memory slices. When the first memory slice is full, it switches to the second memory slice. When the second memory slice is full, the first memory slice in the memory linked list is eliminated first, and then a new memory slice is created with an incremented version number and added to the memory linked list.

[0024] In some embodiments, when executing the current operation through the worker thread, determining the type of the current operation and performing classification processing based on the container P according to the type specifically include:

[0025] If the current operation is an INSERT operation, a ROWID return expression is constructed when constructing the INSERT statement. After execution is completed, the ROWID of the current operation and the ROWID returned by the database are inserted into the mapping container P;

[0026] If the current operation is an UPDATE operation, the ROWID of the row in the target database is queried in the mapping container P using the ROWID of the current operation. The update statement is then executed using the ROWID of the target database as the update condition.

[0027] If the current operation is a DELETE operation, the ROWID of the row of data in the target database is queried in the mapping container P through the ROWID of the current operation. Then, the delete statement is executed using the ROWID of the target database as the deletion condition, and the mapping relationship is cleared from the container P.

[0028] In a second aspect, the present invention further provides a transaction operation parallel execution device based on a mapping function container, the device comprising:

[0029] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the transaction operation parallel execution method based on the mapping function container described in the first aspect.

[0030] In a third aspect, the present invention further provides a transaction operation parallel execution system based on a mapping function container, applying the transaction operation parallel execution method based on a mapping function container as described in the first aspect, wherein the system includes a ROWID mapping module, a KEY acquisition module, a conflict information acquisition module, and a classification processing module, wherein:

[0031] The ROWID mapping module is used to initialize a mapping function container P after the target-side data synchronization service is started, which is used to store the mapping relationship between the source-side database ROWID and the corresponding ROWID in the target-side database;

[0032] The KEY acquisition module is used to extract the KEY calculated based on the table ID and ROWID in the operation after the target-side data synchronization service receives the DML operation from the source side, then extract the transaction ID in the operation as the TRXID, combine the KEY and TRXID into a record and save it to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice;

[0033] The conflict information acquisition module is used to determine the conflicting TRXID and conflicting LSN of the operation according to the KEY, TRXID and starting LSN;

[0034] The classification processing module is used to process operations according to the conflicting TRXID and conflicting LSN of the operations after creating several working threads in the target end data synchronization service, and to judge the type of the current operation when executing the current operation through the working thread, and to perform classification processing based on the container P according to the different types.

[0035] In a fourth aspect, the present invention further provides a non-volatile computer storage medium, which stores computer-executable instructions, which are executed by one or more processors to complete the transaction operation parallel execution method based on the mapping function container described in the first aspect.

[0036] Compared with the prior art, the present invention provides a method, device and system for parallel execution of transaction operations based on mapping function containers, which have the following beneficial effects:

[0037] First, use the ROWID from the source operation to calculate the conflicting key. The biggest advantage is that the ROWID length is fixed, generally not exceeding 20 bytes. The calculation cost is stable and smaller than the calculation cost of user-defined columns, which helps reduce resource consumption.

[0038] Secondly, when executing operations, the worker thread needs to execute according to the operation type. When executing an INSERT operation, it is necessary to construct a ROWID mapping relationship between the source database and the target database. When executing an update or delete operation, it is necessary to first determine the ROWID of the operation data in the target database through the mapping relationship, and then use the ROWID of the target database as the positioning condition to achieve the purpose of improving synchronization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0040] Figure 1 A flowchart of a method for parallel execution of transaction operations based on a mapping function container provided in Example 1 of the present invention;

[0041] Figure 2 Flowchart of determining the conflicting TRXID and conflicting LSN of an operation based on KEY, TRXID and starting LSN provided in Example 1 of the present invention;

[0042] Figure 3 This is a flowchart of conflict detection and judgment provided in Example 1 of the present invention;

[0043] Figure 4 This is a flowchart of classifying and processing containers P according to different operation types, provided in Example 1 of the present invention;

[0044] Figure 5 A module diagram of a transaction operation parallel execution system based on a mapping function container provided in Example 1 of the present invention;

[0045] Figure 6 A schematic diagram of the structure of a transaction operation parallel execution device based on a mapping function container provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention. These all fall within the scope of protection of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other and are all within the scope of protection of this application. In addition, although the functional modules may be divided in the device schematic diagram and the logical order may be shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow chart.

[0047] Unless otherwise defined, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this invention belongs. The terms used in this specification and in the description of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. Furthermore, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0048] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as meaning open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that the specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0049] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate this application and are not intended to limit this application. The present invention will be described in detail below with reference to the accompanying drawings and examples.

[0050] Example 1:

[0051] like Figure 1 As shown, an embodiment of the present invention provides a method for parallel execution of transaction operations based on mapping function containers, which includes the following steps.

[0052] Step 100: After the target-side data synchronization service is started, a mapping container P is initialized to store the mapping relationship between the source-side database ROWID and the corresponding ROWID in the target-side database.

[0053] It should be noted that in this embodiment, LSN is the log sequence number; TRXID is the ID that identifies a transaction; DML refers to data insert, update, and delete operations; and KEY is a key that can be used to identify a specific row in a table. ROWID is a pseudo-column used to uniquely identify rows in a table. MD5 is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value to ensure the integrity and consistency of information transmission.

[0054] Step 200: After the target-side data synchronization service receives the DML operation from the source, it extracts the KEY calculated based on the table ID and ROWID in the operation, then extracts the transaction ID in the operation as the TRXID, combines the KEY and TRXID into a record and saves it to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice.

[0055] Step 300: Determine the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID, and starting LSN. Specifically, construct a detection record X using the KEY of the current operation and merge the detection record X into the hash table R. During the merging process, perform a conflict detection and judgment. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict. Set the TRXID obtained from the conflict detection as the conflicting TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation as the conflicting LSN.

[0056] Step 400: The target-side data synchronization service creates several worker threads to process operations based on their conflicting TRXIDs and conflicting LSNs. When executing the current operation through a worker thread, the type of operation is determined and, based on the type, the operation is classified and processed based on the container P. Specifically, when executing an INSERT operation, a ROWID mapping relationship between the source and target databases must be constructed. When executing an update or delete operation, the ROWID of the target database must be determined through the mapping relationship. The target database's ROWID is then used as the positioning condition to improve synchronization performance.

[0057] The target-side data synchronization service of the embodiment of the present invention marks the received operations with the transaction information on which the operation conflicts and the conflict LSN value detected. When the operation is executed, conflict detection is performed based on the conflict transaction information attached to the operation. If no conflicting transaction is detected to be in execution, and no transaction is detected in the running transaction whose commit LSN is smaller than the commit LSN of the transaction to which the current operation belongs and whose transaction start LSN is smaller than the conflict LSN of the current operation, then the operation can be executed in parallel with the operation submitted earlier than itself, without waiting for the synchronization of the transaction submitted earlier than itself, thereby improving the synchronization performance.

[0058] This embodiment will now explain the above method in more detail.

[0059] In one embodiment, a synchronization system needs to be deployed on both the source and target databases. The source database synchronization system reads logs from the source database and parses them into operations. It then generates an MD5 value based on the table ID and row ID of the DML operation, which serves as the key for the operation. The target database synchronization system is responsible for classifying and collecting synchronization operations sent from the source by transaction, and setting a dependency relationship between each received operation and any conflicting transactions received before it.

[0060] When calculating the operation key, using the ROWID is more efficient than using the primary key, index, or custom column. This is because the ROWID length of the operation row remains fixed, generally within 20 bytes, while the column values ​​of other user columns in the operation row are uncontrollable. Excessively long column values ​​increase the cost of calculating the operation key.

[0061] In addition, since calculating the KEY of the operation will consume computer performance, the method of calculating the KEY of the operation can also be placed in the target-side data synchronization service, which can be selected according to actual needs.

[0062] In one embodiment, after the target-side data synchronization service is started, it is necessary to initialize a hash table R of size N and initialize the record address in the hash item to empty to store and locate the received operations; and create two memory slices of memory space size M to store the KEY in the hash table R. The memory slice names are set with version numbers in the order of creation, and the two memory slices are added to the memory linked list.

[0063] For hash table R, the structure of the HASH item in its HASH table is a 4-byte memory chip version number, set as K; a 4-byte memory offset, set as F. K and F constitute a HASH item in the HASH table, as shown in the following table:

[0064] Memory chip version number (K) Memory offset (F)

[0065] It indicates that the first KEY in the HASH item is stored in the memory slice with version number K, and its offset in the memory slice is F.

[0066] In one embodiment, the target-side data synchronization service also needs to initialize a container P with a mapping function to store the mapping relationship between the source-side database ROWID and the corresponding ROWID in the target-side database.

[0067] In one embodiment, after the source-side data synchronization service captures a DML log operation, it obtains the table ID and the ROWID value of the operation in the table in the log, and jointly calculates the MD5 value as the KEY of the operation.

[0068] In one embodiment, after the target-side data synchronization service receives the DML operation from the source, it extracts the KEY calculated by the source in the operation, then extracts the transaction ID in the operation as the TRXID, combines the KEY and TRXID into a record and saves it at offset F of memory slice K. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice, which is also the conflicting LSN of the operation in the memory slice.

[0069] Since the hash record needs to save the address of the next record, the record saved in the memory chip also needs to reserve the address of the next record, as shown in the following table.

[0070] KEY TRXID Next record memory chip version number (K) Next record memory offset (F)

[0071] refer to Figure 2 As shown, in one embodiment, the step of determining the conflicting TRXID and conflicting LSN of the operation according to the KEY, TRXID and starting LSN specifically includes the following steps.

[0072] Step 101: Use the KEY of the current operation to construct a detection record X, and merge the detection record X into the hash table R.

[0073] Step 102: Conflict detection and judgment are performed during the merging process. If there is a conflicting KEY, the TRXID value corresponding to the conflicting KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict.

[0074] Step 103: Set the TRXID obtained by conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN, thereby obtaining the conflict information of this operation.

[0075] Based on the above steps, the target-side data synchronization service receives the next message and saves it to memory slice K in the same manner as described above. The memory linked list includes two memory slices. When the first memory slice is full (the written data exceeds the set size M), it switches to the second memory slice. When the second memory slice is full, the first memory slice in the linked list is eliminated first, and then a new memory slice is created with an incremented version number and added to the linked list. When a record is saved to a new memory slice, if it is the first record in the memory slice, the operation LSN corresponding to the record is marked as the starting LSN of the memory slice.

[0076] refer to Figure 3 As shown, in one embodiment, the conflict detection and judgment in the above step 102 specifically includes the following steps.

[0077] Step 201: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, backfill the hash item with the address of the detection record X, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection judgment.

[0078] Step 202: Determine whether the memory chip version number K in the address recorded in the hash item is less than the version number K value of the first memory chip in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the record address in the hash item to the next address space of the current detection record X, and backfill the hash item with the address of the detection record X. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment.

[0079] Step 203: Load the corresponding record Y from the memory chip according to the address recorded in the hash item, and determine whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, remove record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, set the record address in the hash item to the next address space of the current detection record X, backfill the address of the current detection record X into the hash item, and use the TRXID stored in record Y as the conflicting TRXID for return; if not, proceed to the next step of conflict detection judgment.

[0080] Step 204: Traverse the next record of record Y and perform conflict detection until the end of the record list. The end condition of the list is: the version number in the next record address is less than the version number of the first memory slice in the current memory list or the version number in the next record address is empty.

[0081] In one embodiment, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to write the conflicting TRXIDs saved in the rolled-back operation back to the conflict detection HASH table, i.e., hash table R. It should be noted that since the transaction dependency information of the rolled-back operation in the rollback and partial rollback operations is restored, the dependency relationship of each transaction received in sequence is clear. These transactions can be committed immediately at the time of submission, without having to wait for transactions with smaller LSNs to commit, thereby further improving synchronization performance.

[0082] Specifically, locate the transaction that needs to be rolled back through the transaction ID of the rollback or partial rollback operation, and extract the number of rows that need to be rolled back in the partial rollback operation or all rows involved in the transaction in the rollback operation. Extract the rolled back operation from the tail of the transaction operation linked list, and extract the KEY value, conflict TRXID and conflict LSN saved in the operation. Compare the conflict LSN of the operation with the starting LSN of the first memory slice. If it is smaller than it, terminate the write back of the partial rollback operation to the HASH table; otherwise, jump to the next step. Search for the conflict operation record with the same KEY in the HASH table according to the KEY of the operation. If it is not found, terminate the write back of the partial rollback operation to the HASH table; otherwise, write back the conflict TRXID to the record in the HASH table, and continue to extract the previous record to be rolled back and perform the same write back until the specified number of rows is rolled back.

[0083] By writing back the conflicting TRXID information in the rollback operation, the continuity problem of operation conflict detection can be solved. These rolled-back operations can be discarded directly. Otherwise, when the transaction is executed, although the partially rolled-back operations do not need to be executed, conflict detection is still required during execution. Otherwise, other transaction operations that depend on it will not be able to find the dependent conflicting transactions and will be executed prematurely, resulting in data inconsistency.

[0084] The above describes the process of setting transaction operation dependencies based on log sequence. The following describes the execution process of the target-side data synchronization service after setting transaction operation dependencies.

[0085] In one embodiment, the target-side data synchronization service creates H worker threads, executes transactions in order of their commit LSNs, and stores the transaction operations. It should be noted that the target-side data synchronization service also creates a hash table P to record the currently executing transaction information (transaction ID) and a hash table G to record the current transaction's table information and the transaction's start LSN.

[0086] In one embodiment, the worker thread extracts a transaction from the committed transaction list in the order in which the transactions were committed, registers the transaction ID in the hash table P, and registers the table information involved in the transaction and the transaction start LSN in the hash table G.

[0087] In one embodiment, a worker thread extracts an operation from a transaction and determines whether the current operation is a commit operation. If so, it commits the current transaction and wakes up the waiting worker thread. It then removes the current transaction's registration information from hash tables P and G, completing the current transaction. Otherwise, it retrieves the conflicting TRXID and conflicting LSN stored in the operation. This is because the transactions that each operation in a transaction depends on have already been determined during conflict detection. Therefore, after executing all operations (indicating that all dependent transactions have already committed), the transaction can commit directly without waiting for transactions that submitted earlier than it, thereby effectively improving synchronization performance.

[0088] In one embodiment, the worker thread uses the conflicting TRXID of the currently pending operation to search hash table P for a transaction with the conflicting TRXID. If so, the worker thread suspends the current execution process and waits for the conflicting transaction to complete and commit before waking up again. Otherwise, the worker thread jumps to the next step. It should be noted that the presence of a transaction with the conflicting TRXID in hash table P indicates that the data involved in the current operation conflicts with the transaction with the conflicting TRXID. Therefore, the current operation cannot be executed in parallel with the conflicting transaction and must wait until the conflicting transaction commits before it can begin execution.

[0089] In one embodiment, a worker thread uses the information in the current pending operation table to search hash table G for a transaction operating on the same table, whose commit LSN is smaller than the current transaction and whose start LSN is smaller than the conflict LSN of the current operation. If so, the worker thread suspends the current execution process and waits for the conflicting transaction to complete and commit before waking up. Otherwise, if the conflict detection for the worker thread's current pending operation does not detect a conflict, the worker thread executes the current operation and then proceeds to the next operation. It should be noted that this step primarily checks whether there are transactions in the currently executing concurrent transactions that exceed the conflict check LSN range. This is because the conflict transaction information for pending operations is valid only within a certain range, from the conflict LSN stored in the operation to the current operation LSN. The operation does not perform conflict detection on operations prior to the conflict LSN. In other words, if a transaction exists with a commit LSN smaller than the commit LSN of the transaction to which the current pending operation belongs and a start LSN smaller than the conflict LSN stored in the current pending operation, it is unknown whether the operations in this transaction conflict with the current pending operation. Therefore, the current pending operation cannot be executed rashly and must wait until the transaction commits.

[0090] In one embodiment, reference Figure 4 , it is also necessary to determine the type of the current operation and then classify it for processing. The specific steps are as follows.

[0091] Step 301: If the current operation is an INSERT operation, a ROWID return expression needs to be constructed when constructing the INSERT statement. After execution is completed, the ROWID of the current operation and the ROWID returned by the database need to be inserted into the mapping container P.

[0092] Step 302: If the current operation is an UPDATE operation, the ROWID of the row of data in the target database is queried in the mapping container P using the ROWID of the current operation, and then the update statement is executed using the ROWID of the target database as the update condition.

[0093] Step 303: If the current operation is a DELETE operation, the ROWID of the row of data in the target database is queried in the mapping container P using the ROWID of the current operation. The delete statement is then executed using the ROWID of the target database as the deletion condition, and the mapping relationship is cleared from the container P.

[0094] It should be noted that when using the ROWID of the source database as the key for conflict detection, the target-side data synchronization service must use the mapped ROWID of the target as the positioning condition when performing update and delete operations. Otherwise, when updating or deleting duplicate rows, although their ROWIDs do not conflict, using columns with duplicate values ​​to perform updates and deletes will cause conflicts, causing the synchronization service to deadlock.

[0095] Based on the above process, this embodiment proposes a method for parallel execution of transaction operations based on a mapping function container, which mainly performs conflict detection on operations within a fixed LSN interval, and saves the detection results and the LSN interval on the corresponding operation. When the working thread executes the operation, it can use the conflict transaction information saved on the operation and its detection LSN interval to determine whether there is a conflict in the currently concurrently executed transaction, thereby improving the parallel execution capability.

[0096] refer to Figure 5As shown, this embodiment also provides a transaction operation parallel execution system based on mapping function container, which applies the aforementioned transaction operation parallel execution method based on mapping function container. The system includes a ROWID mapping module, a KEY acquisition module, a conflict information acquisition module and a classification processing module, wherein: the ROWID mapping module is used to initialize a mapping function container P after the target-side data synchronization service is started, which is used to store the mapping relationship between the source-side database ROWID and the corresponding ROWID in the target-side database; the KEY acquisition module is used to extract the KEY calculated according to the table ID and ROWID in the operation after the target-side data synchronization service receives the DML operation of the source side, and then extract the KEY of the operation. The transaction ID in the operation is used as the TRXID, and the KEY and TRXID are combined into a record and saved in the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice; the conflict information acquisition module is used to determine the conflict TRXID and conflict LSN of the operation based on the KEY, TRXID, and starting LSN; the classification processing module is used to process the operation based on the conflict TRXID and conflict LSN of the operation after creating several work threads in the target-side data synchronization service, and when the current operation is executed by the work thread, the type of the current operation is determined, and classification processing is performed based on the different types based on the container P. The specific functions and principles of this system functional module correspond to the specific methods in the method section and will not be repeated here.

[0097] In summary, this embodiment proposes a method and system for parallel execution of transaction operations based on a mapping function container, which has the following advantages: First, the ROWID on the source-side operation is used to calculate the conflicting key. The biggest advantage is that the ROWID length is fixed, generally not exceeding 20 bytes, and the calculation cost is stable and less than the calculation cost of user-defined columns, which helps to reduce resource consumption.

[0098] Secondly, when executing operations, the worker thread needs to execute according to the operation type. When executing an INSERT operation, it is necessary to construct a ROWID mapping relationship between the source database and the target database. When executing an update or delete operation, it is necessary to first determine the ROWID of the operation data in the target database through the mapping relationship, and then use the ROWID of the target database as the positioning condition to achieve the purpose of improving synchronization performance.

[0099] Example 2:

[0100] Based on the transaction operation parallel execution method based on mapping function container provided in the above embodiment 1, embodiment 2 of the present invention provides a specific example for illustration.

[0101] For example: The source database has table T(ID INT, C1 INT);

[0102] The source application has three transactions that perform the following operations on table T in the following order:

[0103] TRX1: INSERT INTO T(ID, C1)VALUES(1, 1);

[0104] Assume that the ROWID of the last INSERT row is SROWID1.

[0105] TRX1: INSERT INTO T(ID, C1)VALUES(1, 1);

[0106] Assume that the ROWID of the last INSERT operation row is SROWID2;

[0107] TRX1:COMMIT;

[0108] TRX2: UPDATE T SET C1=1 WHERE ROWID=SROWID1; --Updates the first row inserted in TRX1;

[0109] TRX2:COMMIT;

[0110] TRX3: DELETE FROM T WHERE ROWID = SROWID2; -- delete the second row inserted by TRX1;

[0111] TRX3:COMMIT;

[0112] The order of the above operations represents the order of these operations in the log stream, and the resulting log stream will form the following numbered table:

[0113] Transaction ID operate LSN number TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 1 TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 2 TRX1 COMMIT; 3 TRX2 UPDATE T SET C1=1WHERE ROWID=SROWID1; 4 TRX2 COMMIT; 5 TRX3 DELETE FROM T WHERE ROWID=SROWID2; 6 TRX3 COMMIT; 7

[0114] After the target-side data synchronization service is started, it needs to initialize a hash table R of size N and two memory slices of size M. The memory slices are numbered 1 and 2 in sequence.

[0115] The target-side data synchronization service creates a ROWID mapping container P to store the ROWID mapping relationship between the source-side database and the target-side database data.

[0116] The process is as follows:

[0117] 1. When parsing logs, the source-side data synchronization service first combines the table ID and the row's ROWID value. After parsing the log operations, it calculates the KEY for each operation. For ease of presentation, the following table uses the ROWID value to represent the MD5 value of the KEY.

[0118] Transaction ID operate LSN number KEY TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 1 SROWID1 TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 2 SROWID2 TRX1 COMMIT; 3 TRX2 UPDATE T SET C1=1WHERE ROWID=SROWID1; 4 SROWID1 TRX2 COMMIT; 5 TRX3 DELETE FROM T WHERE ROWID=SROWID2; 6 SROWID2 TRX3 COMMIT; 7

[0119] The source-side data synchronization service calculates the operation key by combining the table ID and the operation's ROWID.

[0120] 2. The log receiving thread receives the operation with LSN 1, extracts its key as SROWID1, combines the TRXID and the space for the next record to make the length 32, and first stores its key and transaction ID in the first memory slice, expressed as [memory slice version number (K): memory offset (F)]. Its offset in the memory slice is [1:0].

[0121] KEY: SROWID1 TRXID: 1NEXT: empty

[0122] Since the KEY information is the first operation information of the current memory slice, the LSN value of the operation should be set to the starting LSN of the memory slice, and the starting LSN of the memory slice should be set to 1.

[0123] 3. Insert the KEY into the hash table R. The returned TRXID is 0, indicating no conflict. The TRXID 0 indicating no conflict and the minimum starting LSN of the memory slice 1 are written back into the operation.

[0124] 4. The log receiving thread receives the operation with LSN 2, extracts its key as SROWID2, combines the TRXID and the space for the next record to make the length 32, and first stores its key and transaction ID in the first memory slice. Its offset in the memory slice is [1:32].

[0125] KEY: SROWID1 TRXID: 1NEXT: empty KEY: SROWID2 TRXID: 1NEXT: empty

[0126] 5. Insert the KEY into the hash table R. The returned TRXID is 0, indicating no conflict. The TRXID 0 indicating no conflict and the minimum starting LSN of the memory slice 1 are written back into the operation.

[0127] 6. The log receiving thread receives the operation with LSN 4, extracts its key as SROWID1, combines the TRXID and the space for the next record to make the length 32, and first stores its key and transaction ID in the first memory slice. Its offset in the memory slice is [1:64].

[0128] KEY: SROWID1 TRXID: 1NEXT: empty KEY: SROWID2 TRXID: 1NEXT: empty KEY: SROWID1 TRXID: 2NEXT: empty

[0129] 7. Insert the key into the hash table R. Since the operation information with the key SROWID1 already exists in the HASH, the conflicting TRXID returned is 1, indicating that there is a conflict between the current operation and the transaction with transaction ID 1. The conflicting TRXID 1 and the minimum starting LSN of the memory slice 1 are written back to the operation.

[0130] 8. Remove the conflicting KEY information with SROWID1 from the HASH table, and set the address in the hash table R with KEY SROWID1 to the next record of the current KEY information.

[0131] 9. The log receiving thread receives the operation with LSN 6, extracts its key as SROWID2, combines the TRXID and the space for the next record to make the length 32, and first stores its key and transaction ID in the first memory slice. Its offset in the memory slice is [1:96].

[0132] KEY: SROWID1 TRXID: 1NEXT: empty KEY: SROWID2 TRXID: 1NEXT: empty KEY: SROWID1 TRXID: 2NEXT: 1:0 KEY: SROWID2 TRXID: 3NEXT: empty

[0133] 10. Insert the KEY into the hash table R. The returned TRXID is 1, indicating that there is a conflict between the current operation and the transaction with transaction ID 1. The conflicting TRXID 1 and the minimum starting LSN of the memory slice 1 are written back to the operation.

[0134] After all operation conflicts are detected, the following detection results are formed:

[0135]

[0136]

[0137] 11. Create three worker threads and execute TRX1, TRX2, and TRX3 respectively in the order in which the transactions were submitted.

[0138] 12. Worker thread 1 extracts the TRX1 transaction, registers the transaction information TRXID as 1 in hash table P, and registers the start LSN of the transaction associated with table information T as 1 in hash table G.

[0139] 13. Worker thread 2 extracts the TRX2 transaction, registers the transaction information TRXID 2 into hash table P, and registers the starting LSN 4 of the transaction associated with table information T into hash table G.

[0140] 14. Worker thread 3 extracts the TRX3 transaction, registers the transaction information TRXID 3 into hash table P, and registers the starting LSN 6 of the transaction associated with table information T into hash table G.

[0141] 15. Worker thread 1 extracts the insert operation on TRX1, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0142] (1) The first operation is an insert operation. The conflict TRXID is 0, indicating no conflict.

[0143] (2) Obtain the table information T and the conflict LSN of the operation as 1. Search the hash table G for conflicting transactions involving table T, whose commit LSN is less than 3 and whose minimum transaction start LSN is less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0144] (3) When constructing an insert statement, you need to add a ROWID return expression. After execution, you need to store the ROWID mapping relationship [SROWID1, DROWID1] in the ROWID mapping container P.

[0145] (4) The second operation is an insert operation. The conflict TRXID is 0, indicating no conflict.

[0146] (5) Obtain the table information T and the conflict LSN 2 for the operation. Search the hash table G for conflicting transactions involving table T, with a commit LSN less than 3 and a minimum transaction start LSN less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0147] (6) When constructing an insert statement, you need to add a ROWID return expression. After execution, you need to store the ROWID mapping relationship [SROWID2, DROWID2] in the ROWID mapping container P.

[0148] 16. Worker thread 2 extracts the update operation with LSN 4 on TRX2, obtains the conflicting TRXID and conflicting LSN on the operation, and performs conflict detection.

[0149] The conflicting TRXID of this operation is 1, indicating that there is a conflict with transaction TRX1. The current working thread needs to be suspended and woken up after the working thread 1 completes the synchronization of transaction TRX1.

[0150] 17. Work thread 3 extracts the delete operation with LSN 6 on TRX3, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0151] The conflicting TRXID of this operation is 1, indicating that there is a conflict with transaction TRX1. The current working thread needs to be suspended and woken up after the working thread 1 completes the synchronization of transaction TRX1.

[0152] 18. After worker thread 1 executes the commit operation of TRX1, it deletes the current transaction information from hash table P and hash table G respectively, and then wakes up the TRX2 transaction on worker thread 2 and the TRX3 transaction on worker thread 3 for execution.

[0153] 19. Worker thread 2 extracts the update operation with LSN 4 on TRX2, obtains the conflicting TRXID and conflicting LSN on the operation, and performs conflict detection.

[0154] (1) The conflicting operation TRXID is 1, indicating that there is a conflict in TRX1. Since TRX1 has been submitted, the transaction TRX1 cannot be found in the hash table G, and the judgment proceeds to the next step.

[0155] (2) Obtain the table information T and the conflict LSN of the operation as 1. Search the hash table G for conflicting transactions involving table T, whose commit LSN is less than 5 and whose minimum transaction start LSN is less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0156] (3) Since it is an update operation, it is necessary to extract the ROWID value SROWID1 in the operation and obtain its corresponding target database ROWID value DROWID1 in the mapping container P. The synchronous update statement is constructed with ROWID as the condition: UPDATE T SETC1=1 WHERE ROWID=DROWID1; to execute the update.

[0157] (4) After executing the commit operation of TRX2, the current transaction information is deleted from hash table P and hash table G respectively to complete the execution of TRX2 transaction.

[0158] 20. Work thread 3 extracts the delete operation with LSN 6 on TRX3, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0159] (1) The conflicting operation TRXID is 1, indicating that there is a conflict in TRX1. Since TRX1 has been submitted, the transaction TRX1 cannot be found in the hash table G, and the judgment proceeds to the next step.

[0160] (2) Obtain the table information T and the conflict LSN of the operation as 1. Search the hash table G for conflicting transactions involving table T, whose commit LSN is less than 7 and whose minimum transaction start LSN is less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0161] (3) Since it is a delete operation, it is necessary to extract the ROWID value SROWID2 in the operation and obtain its corresponding target database ROWID value DROWID2 in the mapping container P. The synchronous delete statement is constructed with ROWID as the condition: DELETE FROM TWHERE ROWID=SROWID2; to execute the deletion.

[0162] (4) Delete the ROWID mapping relationship [SROWID2, DROWID2] from the mapping container P.

[0163] (5) After executing the commit operation of TRX3, the current transaction information is deleted from hash table P and hash table G respectively to complete the execution of TRX3 transaction.

[0164] As shown in the preceding flow, transaction TRX1 inserts two rows with identical values, transaction TRX2 updates the first row, and TRX3 deletes the second row. If the user column is used to calculate the conflicting keys, the conflicting keys for TRX1's two inserts, TRX2's update, and TRX3's delete will all be the same. This will cause the execution of these three transactions to form a dependency relationship based on the order in which the transactions were submitted: TRX2 will depend on TRX1's execution, and TRX3 will depend on TRX2's execution. When the target synchronization service executes TRX2 and TRX3 concurrently, these two transactions cannot execute concurrently. If only the ROWID is used as the conflict key for conflict detection, but the target does not use ROWID mapping to locate the rows for TRX2 and TRX3's operations using the mapped ROWID, and instead uses the user column as the update and delete criteria, the concurrent update and delete operations of TRX2 and TRX3 will target the same row, leading to a conflict deadlock. The present invention combines the use of ROWID as the key for conflict detection and ROWID mapping. Transactions TRX2 and TRX3, which can only be executed serially in traditional solutions, can be executed concurrently in this solution, effectively solving the data synchronization performance problem of the source database when there are duplicate rows of data.

[0165] Example 3:

[0166] It should be noted that for rollback transactions, since the rollback transaction will not be executed, there are three ways to handle the rollback transaction operation in the hash table R.

[0167] The first approach is to not roll back the impact of the rolled-back transaction's operations on the operation dependencies in hash table R, nor collect the conflicting TRXIDs of the rolled-back transaction's dependencies. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies. When this approach is adopted, the rolled-back transaction's operations are directly released as a whole, which can improve the processing performance of rolled-back transactions. However, operations that depend on the rolled-back transaction must wait until the rolled-back transaction is released from the pending execution list before they can be executed. Since conflicting transactions with operation dependencies were not collected before the rollback transaction was released, the rollback transaction must wait until all transactions with commit LSNs or rollback LSNs smaller than its own rollback LSN have completed execution before it can be released, which affects synchronization efficiency. This approach corresponds to the following: when the target-side data synchronization service receives a rolled-back transaction, all operations in the transaction are discarded without execution, and only the retained transaction information is added to the pending execution queue. For example, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to locate the transaction based on the transaction ID in the operation, and then mark the transaction to indicate that some operations in the transaction are rolled back. If the operation is a rollback operation, then all operations in the transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed. The transaction information of the rolled back transaction cannot be discarded.

[0168] The second approach does not roll back the impact of the rolled-back transaction on the dependencies in hash table R. Instead, the conflicting TRXIDs of the rolled-back transaction's dependencies are collected, for example, recorded in the conflicting TRXID dependency list L. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending list for execution to ensure the continuity of the dependencies. When this approach is adopted, the conflicting TRXIDs in the rolled-back transaction are collected before the operation is released. Although this reduces the processing performance of the rolled-back transaction, it improves the release performance of the rolled-back transaction during execution. Operations that depend on the rolled-back transaction must wait until the rolled-back transaction is released from the pending list before they can be executed. Since the conflicting transactions of the rolled-back transaction were collected during the release operation, the rolled-back transaction can be released only after the execution of these conflicting transactions, without having to wait for the completion of all transactions with a commit LSN or a rollback LSN smaller than the rollback LSN. This partially improves synchronization efficiency compared to the first approach. This method corresponds to that when the target-side data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation. For the rollback operation, the corresponding transaction cannot be discarded, but is added to the queue of pending transactions.

[0169] The third method: roll back the impact of the rolled-back transaction operation on the operation dependencies in the hash table R. Then, the rolled-back transaction will not be dependent on other operations, and the rolled-back transaction does not need to be added to the pending execution linked list. When this solution is adopted, after receiving the rollback operation, the transaction needs to roll back the impact of all operations in the transaction on the operation dependencies in the hash table R. Since the transaction scale is uncontrollable, when a large transaction is rolled back, the rollback action will affect the progress of receiving the operation, thereby affecting the efficiency of synchronization. This method corresponds to that when the target-side data synchronization service receives a rollback or partial rollback operation, the conflicting TRXID saved in the rolled-back operation is written back to the hash table R. The description of the rollback part in Example 1 of the present invention is an example description under this method.

[0170] The three aforementioned transaction rollback solutions each have their advantages and disadvantages. The first solution is suitable for large-scale transaction rollbacks, the second for medium-scale transactions, and the third for small-scale transactions. Therefore, two parameters, W1 and W2, are set: W1 represents the minimum number of DML operations in large-scale transactions, and W2 represents the minimum number of DML operations in medium-scale transactions. Based on this, after receiving DML operations from the source, the target-side data synchronization service must manage these operations by transaction classification and calculate the number of DML operations received per transaction, S (S being the number of DML operations received in a single transaction). When the target-side data synchronization service receives a rollback operation, it takes appropriate action based on the number of DML operations in the transaction, S.

[0171] When the size S of the rolled-back transaction is greater than or equal to W1, it corresponds to the large-scale transaction rollback in the first solution. In this case, the impact of the rolled-back transaction's operations on the operation dependencies in hash table R is not rolled back, and the conflicting TRXIDs of the rolled-back transaction's operation dependencies are not collected. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations and must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies.

[0172] When the size S of the rolled-back transaction is greater than or equal to W2 and less than W1, it corresponds to the medium-sized transaction rollback in the second solution. In this case, the impact of the rolled-back transaction's operations on the operation dependencies in hash table R is not rolled back. However, the conflicting TRXIDs of the rolled-back transaction's operation dependencies are collected and recorded in the conflicting TRXID dependency list L. This rolled-back transaction will be dependent on other operations and must also be added to the pending execution list for execution to ensure the continuity of the operation dependencies.

[0173] When the size S of the rolled-back transaction is less than W2, this corresponds to the small-scale transaction rollback described in the third solution. In this case, the impact of the rolled-back transaction's operations on the dependencies in hash table R is reversed, and the rolled-back transaction is released directly without being added to the pending execution list. The description of the rollback portion in Example 1 of the present invention describes an example when the size S of the rolled-back transaction is less than W2.

[0174] Furthermore, for partial rollback operations, due to their special nature, they inherently require traversing the rolled-back operations. Therefore, adding the process of collecting conflicting TRXIDs for these operations does not significantly reduce the performance of handling partial rollbacks. Therefore, when the target-side data synchronization service receives a partial rollback operation, it counts the total number of rolled-back rows S1 and, based on the size of S1, performs the following processing: When the number of rolled-back rows S1 is less than W2, the impact of the rollback operation on the operation dependencies in hash table R is rolled back, that is, the conflicting TRXIDs stored in the rolled-back operation are written back to hash table R. When the number of rolled-back rows S1 is greater than W2, the impact of the rolled-back transaction on the operation dependencies in hash table R is not rolled back, that is, the conflicting TRXIDs stored in the rolled-back operation are not written back to hash table R. However, the conflicting TRXIDs of the rolled-back transaction operation dependencies are collected and recorded in the conflicting TRXID dependency list L.

[0175] Based on the above settings, after the target-side data synchronization service creates a worker thread, the worker thread extracts a transaction from the committed transaction list in the order in which the transactions were committed, and determines whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to determine whether the rollback transaction has a conflicting TRXID list L.

[0176] If the rolled-back transaction does not conflict with the TRXID linked list L, it must wait for all transactions with a commit LSN lower than the current transaction to complete before waking up the waiting worker thread. It then removes the current transaction's registration information from hash tables P and G, completing the current transaction. This is because the operations on the rolled-back transaction are no longer needed and are discarded, so the information on the transactions they depend on is unclear. Therefore, when committing, it must wait until all transactions submitted earlier than the current transaction have completed their commit to ensure the correct order of transaction dependencies.

[0177] If the rolled back transaction has a conflicting TRXID linked list L, it is necessary to wait for all conflicting transactions on the conflicting transaction linked list L to commit before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and hash table G to complete the execution of the current transaction. Specifically, extract a conflicting TRXID from the current transaction conflicting transaction linked list L in turn, and check whether it exists in the hash table P. If it exists, it is necessary to wait for the transaction to be completed before waking itself up. After waking up, it extracts the next conflicting TRXID from the conflicting transaction linked list L and performs the same check until all conflicting transactions in the conflicting transaction linked list L are completed. This is because the operations on the rolled back transaction do not need to be executed and are discarded. When the rollback operation is discarded, the conflicting TRXIDs on the operations are collected, so the transaction information they depend on is very clear. Then, when releasing, it only needs to wait for these conflicting transactions to be committed to ensure the correctness of the transaction dependency order.

[0178] In summary, since the rollback transaction will not be executed, it will generate three schemes during conflict detection. Each scheme has its own advantages. It is necessary to set appropriate values ​​to give full play to the advantages of the three schemes in order to achieve the purpose of improving the overall synchronization performance. In addition, for the partial rollback operations of the committed transactions, the conflicting TRXIDs referenced by them are collected in a targeted manner during the rollback operation, which can effectively reduce the performance loss caused by the partial rollback operation writing back the conflict detection hash table R, so as to achieve the purpose of improving the overall synchronization performance. This embodiment processes the rollback transaction and partial rollback operations in a targeted manner according to the agreed operation scale, and finds the balance point between the cost of the operation rollback and the execution of the rollback transaction, thereby improving the overall synchronization performance.

[0179] In addition, in some embodiments, the KEY in the hash table R can be stored in a file or in a memory chip. Specifically, the difference between the two is that when using files for storage, although the storage performance is low and affects the efficiency of conflict detection, after a data synchronization failure, the state of the hash table R before the failure can be restored by storing the KEY value in the file. In this way, when performing conflict detection on the received operations after the failure, there is continuity with the failure, and the synchronization performance will not drop drastically due to the failure. When using memory chips for storage, the conflict detection performance will be greatly improved compared to the file storage solution, but the data of the previous conflict detection will be lost after the failure. When restoring the detection, it can only start from the currently received LSN, which will cause a discontinuity between the subsequent conflict detection and the failure. During synchronous execution, the synchronization performance will drop drastically at the failure location. When the storage performance is high (for example, using SSD fixed storage), it can meet the performance requirements of conflict detection, and the efficiency of operation reception can be balanced with synchronous execution. Then, the method of storing the KEY in the hash table R through a file can effectively avoid the loss of synchronization performance caused by failures. When the storage performance affects the efficiency of conflict detection, the bottleneck of synchronization exists in the operation reception. At this time, the method of storing the KEY in the hash table R through a memory chip can effectively improve the reception performance of the operation, thereby improving the overall synchronization performance. Example 1 of the present application is an example of a method of storing the KEY in the hash table R through memory.

[0180] In some embodiments, you can choose to calculate the MD5 value as the operation key on either the source or target side. The source-side data synchronization service detects conflicts for each operation, allowing for a more accurate calculation of the conflicting MD5 value. However, if the target-side data synchronization service calculates the MD5 value based on the target database table's primary key information, errors may occur if the target database table's primary key structure or column order differs from that on the source side. For this reason, the MD5 value is typically calculated on the source side as the operation key. Upon receiving the DML operation from the source side, the target-side data synchronization service directly extracts the MD5 value calculated by the source side as the key.

[0181] In some embodiments, when calculating the KEY of an operation, since the order of columns in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the KEY is calculated according to the fixed order of the primary key or unique index combination columns. After the source-side data synchronization service captures a DML log operation, it obtains the table ID in the log and uses the table ID to obtain the primary key information and unique index of the table in the source-side database. After parsing the current DML log operation, the source-side data synchronization service determines whether the table has a primary key or unique index. If there is a primary key or unique index, the table ID and the column values ​​of the primary key or unique index are combined to calculate an MD5 value as the KEY for this operation. This value is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when a table has both a primary key and a unique index, the table ID and the primary key are preferably combined to calculate the MD5 value as the KEY; when a table has no primary key but has multiple unique indexes, the table ID and any unique index are combined to calculate the MD5 value. If there is no primary key or unique index, after parsing the current DML log operation, the source-side data synchronization service combines the table ID and the row's ROWID value or custom selected columns to calculate an MD5 value as the key for this operation. This value is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when calculating the key for a table without a primary key or unique constraint, if it is clear that all columns in the table have no duplicate values, the table's ROWID can be used directly to calculate the key, effectively reducing the cost of calculating the key. If the table has duplicate rows, custom selected columns are required to calculate the key. The key can be calculated based on the distribution of column values, which columns or all column values ​​are used to calculate the key. This allows the target-side synchronization service to use the key to prevent deadlocks caused by concurrent synchronization transactions operating on rows with duplicate values.

[0182] Example 4:

[0183] Based on the transaction operation parallel execution method based on mapping function container provided in the above embodiment 1, the present invention also provides a transaction operation parallel execution device based on mapping function container that can be used to implement the above method and system, such as Figure 6 FIG. 1 is a schematic diagram of the device architecture of an embodiment of the present invention. The transaction operation parallel execution device based on the mapping function container of this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0184] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The bus connection is taken as an example.

[0185] Memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the method for parallel execution of transaction operations based on mapping function containers in Example 1. Processor 21 executes the non-volatile software programs, instructions, and modules stored in memory 22 to execute various functional applications and data processing of the device for parallel execution of transaction operations based on mapping function containers, thereby implementing the method for parallel execution of transaction operations based on mapping function containers in Example 1.

[0186] The memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21, and such remote memory may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the method for executing transaction operations in parallel based on the mapping function container in the above embodiment 1 is executed. For example, the method described above is executed. Figures 1-4 The steps shown.

[0188] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0189] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0190] Through the description of the above embodiments, it can be clearly understood by those skilled in the art that each embodiment can be implemented by means of software plus a general hardware platform, or of course by hardware. It can be understood by those skilled in the art that all or part of the processes in the above embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above. For the sake of simplicity, they are not provided in detail. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A transaction operation parallel execution method based on mapping function container, characterized in that: include: After the target-side data synchronization service is started, a mapping container P is initialized to store the mapping relationship between the source-side database ROWID and the corresponding ROWID in the target-side database. After receiving the DML operation from the source, the target data synchronization service extracts the KEY calculated from the table ID and ROWID in the operation, and then extracts the transaction ID in the operation as the TRXID. It combines the KEY and TRXID into a record and saves it to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice. Determine the conflicting TRXID and conflicting LSN of the operation according to KEY, TRXID and starting LSN; The target-side data synchronization service creates several working threads, processes the operations according to the conflicting TRXID and conflicting LSN of the operations, and when executing the current operation through the working thread, determines the type of the current operation, and classifies and processes it based on the container P according to the different types.

2. The transaction operation parallel execution method based on mapping function container according to claim 1 is characterized in that: After the target-side data synchronization service is started, a hash table R is initialized, and the record address in the hash item of the hash table R is initialized to empty, which is used to store and locate the received operations; the target-side data synchronization service creates two memory slices to store the KEY in the hash table R, and each memory slice is incremented in the order of creation. The version number is set, and the two memory slices are added to the memory linked list.

3. The transaction operation parallel execution method based on mapping function container according to claim 1, characterized in that: The generation of the KEY specifically includes: generating an MD5 value as the KEY of the operation according to the table ID and ROWID in the DML operation.

4. The transaction operation parallel execution method based on mapping function container according to claim 2, characterized in that: The conflicting TRXID and conflicting LSN of the operation determined according to KEY, TRXID and starting LSN specifically include: Use the KEY of the current operation to construct the detection record X, and merge the detection record X into the hash table R; During the merging process, conflict detection and judgment are performed. If there is a conflicting KEY, the TRXID value corresponding to the conflicting KEY is returned; otherwise, 0 is returned as the corresponding TRXID, indicating no conflict; The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

5. The transaction operation parallel execution method based on mapping function container according to claim 4 is characterized in that: The conflict detection and judgment specifically includes: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item, and return 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next step of conflict detection and judgment; Determine whether the memory chip version number in the address recorded in the hash item is less than the version number value of the first memory chip in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment; According to the address recorded in the hash item, the corresponding record Y is loaded from the memory chip, and it is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, and record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item, and the record address in the hash item is set to the next address space of the current detection record X, and the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflicting TRXID for return; if not, the next step of conflict detection and judgment is carried out; Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record linked list.

6. The transaction operation parallel execution method based on mapping function container according to claim 2, characterized in that: The memory linked list includes two memory slices. When the first memory slice is full, it switches to the second memory slice. When the second memory slice is full, the first memory slice in the memory linked list is eliminated first, and then a new memory slice is created with an increasing version number and added to the memory linked list.

7. The transaction operation parallel execution method based on mapping function container according to any one of claims 1 to 6, characterized in that: When the current operation is executed by the working thread, the type of the current operation is determined, and classification processing is performed based on the container P according to the type, specifically including: If the current operation is an INSERT operation, a ROWID return expression is constructed when constructing the INSERT statement. After the execution is completed, the ROWID of the current operation and the ROWID returned by the database are inserted into the mapping container P; If the current operation is an UPDATE operation, the ROWID of the row of data in the target database is queried in the mapping container P through the ROWID on the current operation, and then the update statement is executed using the ROWID of the target database as the update condition; If the current operation is a DELETE operation, the ROWID of the row of data in the target database is queried in the mapping container P through the ROWID on the current operation, and then the delete statement is executed using the ROWID of the target database as the deletion condition, and the mapping relationship is cleared from the container P.

8. A transaction operation parallel execution device based on mapping function container, characterized in that: The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the transaction operation parallel execution method based on mapping function container as described in any one of claims 1-7.

9. A transaction operation parallel execution system based on mapping function container, applying the transaction operation parallel execution method based on mapping function container as claimed in any one of claims 1 to 7, characterized in that: The system includes a ROWID mapping module, a KEY acquisition module, a conflict information acquisition module and a classification processing module, wherein: The ROWID mapping module is used to initialize a mapping function container P after the target-side data synchronization service is started, which is used to store the mapping relationship between the source-side database ROWID and the corresponding ROWID in the target-side database; The KEY acquisition module is used to extract the KEY calculated according to the table ID and ROWID in the operation after the target-side data synchronization service receives the DML operation from the source side, and then extract the transaction ID in the operation as the TRXID, and combine the KEY and TRXID into a record and save it in the current memory slice. If the record is the first record of the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice; The conflict information acquisition module is used to determine the conflicting TRXID and conflicting LSN of the operation according to the KEY, TRXID and the starting LSN; The classification processing module is used to process operations according to the conflicting TRXID and conflicting LSN of the operations after creating several working threads in the target end data synchronization service, and to determine the type of the current operation when executing the current operation through the working thread, and to perform classification processing based on the container P according to the different types.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the transaction operation parallel execution method based on mapping function container as described in any one of claims 1-7.

Images