A method, apparatus, and system for parallel execution of transaction operations based on mapping functional containers.

Abstract

This invention relates to a method, apparatus, and system for parallel execution of transaction operations based on a mapping function container. The method mainly includes: after the target-side data synchronization service starts, initializing a mapping function container P to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database; determining the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID, and starting LSN; the target-side data synchronization service creating several worker threads to process the operation according to the conflicting TRXID and conflicting LSN, and determining the type of the current operation when the worker thread executes the current operation, and classifying and processing it based on the type using container P. This invention can effectively increase the concurrency of operation execution, thereby improving synchronization performance.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method, apparatus and system for parallel execution of transaction operations based on a mapping function container. Background Technology

[0002] In a real-time database synchronization system based on log parsing architecture (supporting heterogeneous databases), the source-side data synchronization service is responsible for capturing the operation logs of the source database, parsing the logs to restore the corresponding database operations, and then sending them to the target-side data synchronization service. The target-side data synchronization service is responsible for executing these operations in the target database, thereby completing the data synchronization process.

[0003] When a target-side data synchronization service performs synchronization operations on a target database, it often needs to execute according to the transaction commit order. This is because if the previous transaction hasn't completed synchronization, the next transaction's operation cannot be executed concurrently with previous transactions and its data entry cannot proceed until all transactions preceding the current operation have completed their commits. This significantly impacts the concurrency of transaction data entry. Currently, an effective method is to determine whether parallel execution is possible by comparing the log LSN of the current operation with the LSN of a previously committed transaction. If the LSN of the currently executing transaction is smaller than the LSN of another currently executing transaction, then this operation can be executed concurrently with that transaction. While this method of parallel execution based on transaction commit order improves performance, the granularity of conflict detection is too large and not fine enough, resulting in low parallelism and a small performance improvement.

[0004] In view of this, how to overcome the shortcomings of existing technologies and how to improve the parallel execution of target-side data synchronization services to enhance data synchronization performance have become important technical problems that the industry urgently needs to solve. Summary of the Invention

[0005] Addressing the shortcomings or improvement needs of existing technologies, this invention aims to enhance the parallel execution of target-side data synchronization services to improve data synchronization performance. The invention proposes that by determining the dependencies between transaction operations and clearly identifying which transactions need to be waited for during synchronization, the parallelism of synchronization can be maximized. Based on this, the invention provides a method, apparatus, and system for parallel execution of transaction operations based on a mapping functional container. In this method, the target-side data synchronization service receives synchronization transaction operations in the order of generation of the source database logs. The order of each received operation forms the basis for detecting conflicts. This invention extracts the ROWID from each received operation based on the order of operation reception and checks whether this ROWID, within a defined range, is identical to the ROWID of other previously received transactions. If so, the execution of this operation must wait for the conflicting transaction to commit before it can begin; otherwise, it can be executed concurrently with transactions that have already committed, maximizing the parallelism of transaction execution and effectively improving synchronization performance.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for parallel execution of transaction operations based on a mapping function container, comprising:

[0008] After the target data synchronization service starts, it initializes a container P with mapping function to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database.

[0009] After receiving the DML operation from the source end, the target end data synchronization service extracts the KEY calculated based on the table ID and ROWID in the operation, and then extracts the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice.

[0010] Determine the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID, and starting LSN;

[0011] The target data synchronization service creates several worker threads to process operations based on their conflicting TRXIDs and conflicting LSNs. When a worker thread executes an operation, it determines the type of the operation and classifies it based on the container P according to the type.

[0012] In some embodiments, after the target data synchronization service starts, it initializes a hash table R and initializes the record addresses in the hash entries of hash table R to be empty, which is used to store and locate the received operations; the target data synchronization service creates two memory slices to store the KEY in hash table R, each memory slice increments the version number in the order of creation, and adds the two memory slices to the memory linked list.

[0013] In some embodiments, the generation of the KEY specifically includes: generating an MD5 value based on the ID and ROWID of the DML operation table as the KEY of the operation.

[0014] In some embodiments, determining the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID, and starting LSN specifically includes:

[0015] Construct a detection record X using the KEY of the current operation, and merge the detection record X into the hash table R;

[0016] During the merging process, conflict detection is performed. If a conflicting key is found, the TRXID value corresponding to the conflicting key is returned; otherwise, 0 is returned as the corresponding TRXID, indicating that there is no conflict.

[0017] Set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN.

[0018] In some embodiments, the conflict detection judgment specifically includes:

[0019] Calculate the hash value of the KEY of the current detection record X, and locate the hash entry using the hash value; determine whether the record address in the hash entry is empty. If so, set the record address in the hash entry to the next address space of the current detection record X, fill the hash entry with the address of the detection record X, and return 0 as the conflict TRXID, indicating that no conflict was detected; otherwise, proceed to the next conflict detection and judgment step.

[0020] Determine if the memory slice version number at the address recorded in the hash entry is less than the version number of the first memory slice in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the record address in the hash entry to the next address space of the current detection record X, and use the address of the detection record X to fill back into the hash entry. Return 0 as the conflict TRXID, indicating that no conflict was detected. If not, proceed to the next conflict detection and judgment step.

[0021] Based on the address recorded in the hash entry, load the corresponding record Y from the memory slice. Determine if the KEY of record Y is equal to the KEY of the currently detected record X and if the TRXID of record Y is not equal to the TRXID of the current operation. If so, it indicates a conflict. Remove record Y from the record list of the hash entry to shorten the length of the linked list in the hash entry. Set the address of the record in the hash entry to the next address space of the currently detected record X. Fill the address of the currently detected record X back into the hash entry and return it using the TRXID stored in record Y as the conflict TRXID. If not, proceed to the next conflict detection step.

[0022] Iterate through the next record of record Y and perform conflict detection and judgment until the record linked list ends.

[0023] In some embodiments, the memory list includes two memory blocks. When the first memory block is full, the system switches to the second memory block. When the second memory block is full, the first memory block in the memory list is first removed, and then a new memory block is created and added to the memory list using an incrementing version number.

[0024] In some embodiments, when the current operation is executed by the worker thread, determining the type of the current operation and classifying it based on the container P according to the type specifically includes:

[0025] If the current operation is an INSERT operation, construct the ROWID return expression when constructing the INSERT statement. After execution, insert the ROWID of the current operation and the ROWID returned by the database into the mapping container P.

[0026] If the current operation is an UPDATE operation, the ROWID of the row of data in the target database is retrieved from the mapping container P using the ROWID of the current operation, and then the ROWID of the target database is used as the update condition to execute the update statement.

[0027] If the current operation is a DELETE operation, the ROWID of the row of data in the target database is retrieved from the mapping container P using the ROWID of the current operation. Then, the ROWID of the target database is used as the deletion condition to execute the deletion statement, and the mapping relationship is cleared from container P.

[0028] Secondly, the present invention also provides a parallel execution device for transaction operations based on a mapping function container, the device comprising:

[0029] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the parallel execution method of transaction operations based on the mapping function container described in the first aspect.

[0030] Thirdly, the present invention also provides a parallel execution system for transaction operations based on a mapping functional container, applying the parallel execution method for transaction operations based on a mapping functional container as described in the first aspect. The system includes a ROWID mapping module, a KEY acquisition module, a conflict information acquisition module, and a classification processing module, wherein:

[0031] The ROWID mapping module is used to initialize a container P with mapping function after the target data synchronization service starts, which is used to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database.

[0032] The KEY acquisition module is used to extract the KEY calculated based on the table ID and ROWID in the operation after the target data synchronization service receives the DML operation from the source end, and then extract the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record of the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice.

[0033] The conflict information acquisition module is used to determine the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID and the starting LSN.

[0034] The classification processing module is used to process operations based on the conflict TRXID and conflict LSN after creating several worker threads in the target data synchronization service. When the worker thread executes the current operation, it determines the type of the current operation and performs classification processing based on the container P according to the different types.

[0035] Fourthly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the parallel execution method for transaction operations based on a mapping functional container as described in the first aspect.

[0036] Compared with the prior art, the present invention provides a method, apparatus and system for parallel execution of transaction operations based on a mapping functional container, the advantages of which are:

[0037] First, using ROWID from the source operation to calculate conflicting keys has the biggest advantage that the ROWID length is fixed, generally not exceeding 20 bytes, and the calculation cost is stable and less than that of user-defined columns, which helps to reduce resource consumption.

[0038] Secondly, when worker threads perform operations, they need to execute them according to the operation type. When performing INSERT operations, it is necessary to construct a ROWID mapping relationship between the source database and the target database. When performing update and delete operations, it is necessary to first determine the ROWID of the operation data in the target database through the mapping relationship, and then use the ROWID of the target database as the location condition to improve synchronization performance. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0040] Figure 1 A flowchart of a method for parallel execution of transaction operations based on a mapping function container provided in Embodiment 1 of the present invention;

[0041] Figure 2 This is a flowchart of determining the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID and the starting LSN provided in Embodiment 1 of the present invention;

[0042] Figure 3 This is a flowchart of the conflict detection and judgment process provided in Embodiment 1 of the present invention;

[0043] Figure 4 This is a flowchart of a classification process based on container P according to different operation types, provided in Embodiment 1 of the present invention.

[0044] Figure 5 This is a schematic diagram of a parallel execution system for transaction operations based on a mapping function container, provided in Embodiment 1 of the present invention.

[0045] Figure 6 This is a schematic diagram of a parallel execution device for transaction operations based on a mapping function container, provided in Embodiment 3 of the present invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention. It should be noted that, unless otherwise specified, the various features in the embodiments of the present invention can be combined with each other, all within the protection scope of this application. Furthermore, although functional modules may be divided in the device schematic diagram, and a logical order may be shown in the flowchart, in some cases, the steps shown or described may be executed differently from the module division in the device or the order in the flowchart.

[0047] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of the invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] Example 1:

[0051] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for parallel execution of transaction operations based on a mapping function container, the method comprising the following steps.

[0052] Step 100: After the target data synchronization service starts, initialize a container P with mapping function to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database.

[0053] It should be noted that in this embodiment, LSN is the log sequence number; TRXID is the ID that identifies a transaction; DML refers to data insertion, update, and deletion operations; KEY is the key used to identify a specific row in the table. ROWID is a pseudo-column used to uniquely identify rows in the table. MD5 is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value to ensure the integrity and consistency of transmitted information.

[0054] Step 200: After receiving the DML operation from the source end, the target end data synchronization service extracts the KEY calculated based on the table ID and ROWID in the operation, and then extracts the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice.

[0055] Step 300: Determine the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID, and starting LSN. Specifically, construct a detection record X using the KEY of the current operation, and merge the detection record X into the hash table R; during the merging process, perform conflict detection and judgment. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict; set the TRXID obtained from the conflict detection and judgment as the conflicting TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation, setting it as the conflicting LSN.

[0056] Step 400: The target-side data synchronization service creates several worker threads to handle operations based on their conflicting TRXIDs and LSNs. When a worker thread executes an operation, it determines the type of the operation and categorizes it based on the container P. Specifically, during an INSERT operation, a ROWID mapping relationship needs to be constructed between the source and target databases. During updates and deletions, the ROWID of the data in the target database is first determined through the mapping relationship, and then used as the location condition to improve synchronization performance.

[0057] In the target data synchronization service of this invention, each received operation is marked with transaction information that the operation depends on and the conflict LSN value for dependency detection. When the operation is executed, conflict detection is performed based on the conflict transaction information attached to the operation. If no conflicting transaction is detected, and no transaction with a commit LSN less than the commit LSN of the transaction to which the current operation belongs or a transaction start LSN less than the conflict LSN of the current operation is detected in the running transaction, then the operation can be executed in parallel with the operation that was committed earlier than itself, without waiting for the transaction that was committed earlier to complete synchronization, thereby improving the synchronization performance.

[0058] The following embodiment will provide a more detailed description of the above method.

[0059] In one embodiment, a synchronization system needs to be deployed in both the source and destination databases. The source database synchronization system reads logs from the source database and parses them into operations. It then uses the ID and ROWID from the DML operation table to generate an MD5 value as the key for the operation. The destination database synchronization system is responsible for classifying and collecting the synchronization operations sent from the source database by transaction and setting the dependency relationship between each received operation and any conflicting transactions received before it.

[0060] When calculating the key for an operation, using ROWID is more efficient than using the primary key, index, or custom column. This is because the length of the ROWID in the operation row is fixed, generally within 20 bytes, while the values ​​of other user columns in the operation row are uncontrollable. Columns with excessively long values ​​will increase the cost of calculating the operation key.

[0061] In addition, since calculating the key consumes computer resources, the key calculation method can also be placed in the target data synchronization service, which can be selected according to actual needs.

[0062] In one embodiment, after the target data synchronization service starts, it needs to initialize a hash table R of size N and initialize the record addresses in the hash entries to be empty, which is used to store and locate the received operations; and create two memory slices of size M to store the KEY in the hash table R. The memory slice names are set with version numbers incremented in the order of creation, and these two memory slices are added to the memory linked list.

[0063] For hash table R, the structure of a hash entry in its hash table consists of a 4-byte memory slice version number, denoted as K; and a 4-byte memory offset, denoted as F. K and F together form a hash entry in the hash table, as shown in the table below:

[0064] Memory chip version number (K) Memory offset (F)

[0065] This indicates that the first key in the HASH entry is stored in the memory slice with version number K, and its offset in that memory slice is F.

[0066] In one embodiment, the target data synchronization service also needs to initialize a container P with a mapping function to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database.

[0067] In one embodiment, after the source data synchronization service captures a DML log operation, it obtains the table ID in the log and the ROWID value of the operation in the table, and calculates the MD5 value together as the KEY of the operation.

[0068] In one embodiment, after the target data synchronization service receives the DML operation from the source, it extracts the KEY calculated by the source in the operation, and then extracts the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved at offset F of memory slice K. If the record is the first record of the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice, which is also the conflict LSN of the operation in the memory slice.

[0069] Since a hash record needs to store the address of the next record, the record stored in the memory chip also needs to reserve the address for storing the next record, as shown in the table below.

[0070] KEY TRXID Next record memory chip version number (K) Next record memory offset (F)

[0071] refer to Figure 2 As shown, in one embodiment, determining the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID, and starting LSN specifically includes the following steps.

[0072] Step 101: Construct a detection record X using the KEY of the current operation, and merge the detection record X into the hash table R.

[0073] Step 102: During the merging process, perform conflict detection and judgment. If there is a conflicting key, return the TRXID value corresponding to the conflicting key; otherwise, return 0 as the corresponding TRXID, indicating that there is no conflict.

[0074] Step 103: Set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation, setting it as the conflict LSN. This yields the conflict information for this operation.

[0075] Based on the above steps, the target data synchronization service receives the next message and saves it to memory slice K in the manner described above. The memory linked list includes two memory slices. When the first memory slice is full (the written data exceeds the set size M), the system switches to the second memory slice. When the second memory slice is full, the first memory slice in the linked list is first discarded, and then a new memory slice is created and added to the linked list using an incrementing version number. When a record is saved to a new memory slice, if it is the first record in that memory slice, the operation LSN corresponding to that record is marked as the starting LSN of that memory slice.

[0076] refer to Figure 3 As shown, in one embodiment, the conflict detection and judgment in step 102 above specifically includes the following steps.

[0077] Step 201: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, fill the hash item with the address of the detection record X, and return 0 as the conflict TRXID, indicating that no conflict was detected; if not, proceed to the next conflict detection judgment.

[0078] Step 202: Determine whether the memory slice version number K at the address recorded in the hash entry is less than the version number K of the first memory slice in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the record address in the hash entry to the next address space of the current detection record X, fill the hash entry with the address of the detection record X, and return 0 as the conflict TRXID, indicating that no conflict was detected. If not, proceed to the next conflict detection and judgment step.

[0079] Step 203: Load the corresponding record Y from the memory slice based on the address recorded in the hash item. Determine whether the KEY of record Y is equal to the KEY of the currently detected record X and whether the TRXID of record Y is not equal to the TRXID of the current operation. If so, it indicates that there is a conflict in the operation. Remove record Y from the record linked list of the hash item to shorten the linked list length in the hash item. Set the record address in the hash item to the next address space of the currently detected record X. Fill the address of the currently detected record X back into the hash item and return it using the TRXID stored in record Y as the conflict TRXID. If not, proceed to the next conflict detection and judgment step.

[0080] Step 204: Iterate through the next record after record Y and perform conflict detection until the record list ends. The condition for the end of the list is: the version number at the address of the next record is less than the version number of the first memory slice in the current memory list, or the version number at the address of the next record is null.

[0081] In one embodiment, when the target data synchronization service receives a rollback or partial rollback operation, it needs to write the conflict TRXIDs saved in the rolled-back operation back to the conflict detection HASH table, i.e., hash table R. It should be noted that since the transaction dependency information of the rolled-back transactions and partial rollback operations is restored, the dependency relationships of each transaction received in chronological order are clear. These transactions can be committed immediately upon commit, without waiting for transactions with a smaller LSN to commit, thereby further improving synchronization performance.

[0082] Specifically, the transaction requiring rollback is located using the transaction ID of the rollback or partial rollback operation, and the number of rows to be rolled back in the partial rollback operation or all rows involved in the transaction in the rollback operation are extracted. The rollback operation is extracted backwards from the tail of the transaction operation chain, retrieving the saved KEY value, conflict TRXID, and conflict LSN. The conflict LSN of the operation is compared with the starting LSN of the first memory slice. If it is smaller, the write-back of this partial rollback operation to the HASH table is terminated; otherwise, proceed to the next step. A conflicting operation record with the same KEY is searched in the HASH table. If no record is found, the write-back of this partial rollback operation to the HASH table is terminated; otherwise, the conflict TRXID is written back to the record in the HASH table, and the same write-back process is repeated for the previous record to be rolled back, until the specified number of rows is reached.

[0083] By writing back the conflicting TRXID information in the rollback operation, the continuity problem of operation conflict detection can be solved. These rolled-back operations can be directly discarded. Otherwise, when the transaction is executed, although the partially rolled-back operations do not need to be executed, conflict detection still needs to be performed during execution. Otherwise, other transaction operations that depend on it will not be able to find the conflicting transactions they depend on and will be executed prematurely, causing data inconsistency.

[0084] The above describes the process of setting transaction operation dependencies based on log order. The following describes the execution flow of the target data synchronization service after setting transaction operation dependencies.

[0085] In one embodiment, the target-side data synchronization service creates H worker threads to execute transactions in the order of their committed LSNs received, and then stores the transaction operations in the database. It should be noted that the target-side data synchronization service also needs to create a hash table P to register the currently executing transaction information (transaction ID) and a hash table G to register the table information and the starting LSN of the current transaction.

[0086] In one embodiment, a worker thread retrieves a transaction from the committed transaction list in the order of transaction commits, registers the transaction ID in hash table P, and registers the table information involved in the transaction and the starting LSN of the transaction in hash table G.

[0087] In one embodiment, a worker thread extracts an operation from a transaction, determines whether the current operation is a commit operation, and if so, commits the current transaction, wakes up its waiting worker threads, removes the current transaction's registration information from hash tables P and G, and completes the execution of the current transaction; otherwise, it retrieves the conflict TRXID and conflict LSN stored in the operation. This is because the transactions that each operation on a transaction depends on have already been determined in conflict detection, so the transaction can commit directly after executing all operations (meaning that all its dependent transactions have committed), without waiting for transactions that committed before it to commit first, thus effectively improving synchronization performance.

[0088] In one embodiment, the worker thread uses the conflicting TRXID of the currently pending operation to search in hash table P for a transaction with the conflicting TRXID. If a conflicting TRXID exists, the current execution process is suspended, and the worker thread waits for the conflicting transaction to complete and commit before waking up; otherwise, it proceeds to the next step. It should be noted that the existence of a conflicting TRXID transaction in hash table P indicates that the data involved in the current operation conflicts with the conflicting TRXID transaction. The current operation cannot be executed concurrently with the conflicting transaction; it must wait for the conflicting transaction to commit before execution can begin.

[0089] In one embodiment, the worker thread uses the information from the current pending operation table to search in hash table G for a transaction that operates on the same table, where the commit LSN of this transaction is less than the current transaction's, and the starting LSN of the transaction is less than the conflict LSN of the current operation. If such a transaction exists, the current execution process is suspended, waiting for the conflicting transaction to complete and commit before waking up. Otherwise, it is determined that the conflict detection for the worker thread's current pending operation has not detected a conflict, so the current operation is executed, and then the next operation continues. It should be noted that this step mainly checks whether there are any concurrently executing transactions that exceed the conflict check LSN range. This is because the conflicting transaction information for the pending operation has a valid range: the conflicting transaction information is only valid within the range from the conflicting LSN saved in the operation to the current operation's LSN. The operation does not perform conflict detection on operations before the conflicting LSN. In other words, when there is a transaction whose commit LSN is less than the commit LSN of the transaction to which the current pending operation belongs, and whose starting LSN is less than the conflicting LSN saved in the current pending operation, it is unknown whether there is a conflict between the operations in this transaction and the current pending operation. Therefore, the current pending operation cannot be executed rashly and must wait for the transaction to commit before execution.

[0090] In one embodiment, reference Figure 4 Furthermore, it is necessary to determine the type of the current operation and then process it accordingly. This includes the following steps.

[0091] Step 301: If the current operation is an INSERT operation, when constructing the INSERT statement, a ROWID return expression needs to be constructed. After execution, the ROWID of the current operation and the ROWID returned by the database need to be inserted into the mapping container P.

[0092] Step 302: If the current operation is an UPDATE operation, query the ROWID of the row in the target database in the mapping container P using the ROWID of the current operation, and then use the ROWID of the target database as the update condition to execute the update statement.

[0093] Step 303: If the current operation is a DELETE operation, query the ROWID of the row of data in the target database in the mapping container P using the ROWID of the current operation, then use the ROWID of the target database as the deletion condition to execute the deletion statement, and clear the mapping relationship from the container P.

[0094] It should be noted that when using the source database's ROWID as the conflict detection key, the target data synchronization service must use the mapped target ROWID as the location condition when performing update and delete operations. Otherwise, when updating or deleting duplicate rows, although their ROWIDs are not conflicting, using columns with duplicate values ​​to perform updates and deletes will cause conflicts and result in a deadlock in the synchronization service.

[0095] Based on the above process, this embodiment proposes a method for parallel execution of transaction operations based on a mapping function container. The main method is to perform conflict detection on the operation within a fixed LSN interval, and save the detection result and the LSN interval to the corresponding operation. When the worker thread executes the operation, it can use the conflict transaction information saved on the operation and its detection LSN interval to determine whether there is a conflict in the currently concurrently executed transaction, thereby improving the parallel execution capability.

[0096] refer to Figure 5As shown, this embodiment also provides a parallel execution system for transaction operations based on a mapping function container. Applying the aforementioned parallel execution method for transaction operations based on a mapping function container, the system includes a ROWID mapping module, a KEY acquisition module, a conflict information acquisition module, and a classification processing module. Specifically: the ROWID mapping module initializes a mapping function container P after the target-side data synchronization service starts, used to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database; the KEY acquisition module extracts the KEY calculated based on the table ID and ROWID from the operation after the target-side data synchronization service receives the DML operation from the source, and then extracts the KEY from the operation... The transaction ID in the operation is used as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If this record is the first record in the current memory slice, the log LSN corresponding to this operation is used as the starting LSN of the current memory slice. The conflict information acquisition module is used to determine the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID, and starting LSN. The classification processing module is used to process the operation based on the conflicting TRXID and conflicting LSN after creating several worker threads in the target data synchronization service. When the worker thread executes the current operation, it determines the type of the current operation and classifies it based on the container P according to the different types. The specific functions and principles of the system's functional modules correspond to the specific methods in the method section, and will not be repeated here.

[0097] In summary, this embodiment proposes a method and system for parallel execution of transaction operations based on a mapping function container, which has the following advantages: First, it uses the ROWID on the source operation to calculate the conflicting key. The biggest advantage is that the ROWID length is fixed, generally not exceeding 20 bytes. The calculation cost is stable and less than the calculation cost of user-defined columns, which helps to reduce resource consumption.

[0098] Secondly, when worker threads perform operations, they need to execute them according to the operation type. When performing INSERT operations, it is necessary to construct a ROWID mapping relationship between the source database and the target database. When performing update and delete operations, it is necessary to first determine the ROWID of the operation data in the target database through the mapping relationship, and then use the ROWID of the target database as the location condition to improve synchronization performance.

[0099] Example 2:

[0100] Based on the parallel execution method of transaction operations based on mapping function containers provided in Embodiment 1 above, Embodiment 2 of the present invention provides a specific example for illustration.

[0101] For example: The source database has a table T(ID INT, C1 INT);

[0102] The source application has three transactions performing the following operations on table T in the following order:

[0103] TRX1: INSERT INTO T(ID, C1)VALUES(1, 1);

[0104] Assume the ROWID of the previous INSERT operation row was SROWID1;

[0105] TRX1: INSERT INTO T(ID, C1)VALUES(1, 1);

[0106] Assume the ROWID of the previous INSERT operation row is: SROWID2;

[0107] TRX1: COMMIT;

[0108] TRX2: UPDATE T SET C1 = 1 WHERE ROWID = SROWID1; -- Update the first row inserted by TRX1;

[0109] TRX2: COMMIT;

[0110] TRX3: DELETE FROM T WHERE ROWID = SROWID2; -- Delete the second row inserted by TRX1;

[0111] TRX3: COMMIT;

[0112] The order of the operations described above represents their order within the log stream, resulting in the following numbered table of the generated log stream:

[0113] Transaction ID operate LSN number TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 1 TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 2 TRX1 COMMIT; 3 TRX2 UPDATE T SET C1=1WHERE ROWID=SROWID1; 4 TRX2 COMMIT; 5 TRX3 DELETE FROM T WHERE ROWID=SROWID2; 6 TRX3 COMMIT; 7

[0114] After the target data synchronization service starts, it needs to initialize a hash table R of size N and two memory slices of size M, which are numbered sequentially as 1 and 2.

[0115] The target-side data synchronization service creates a ROWID mapping container P to store the ROWID mapping relationship between the source database and the target database.

[0116] The process is as follows:

[0117] 1. When parsing logs, the source data synchronization service needs to combine the table ID and the ROWID value of the data row to calculate the KEY of each operation after parsing the log operations. For ease of explanation, the ROWID value is used to represent the MD5 value of the KEY in the table below.

[0118] Transaction ID operate LSN number KEY TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 1 SROWID1 TRX1 INSERT INTO T(ID,C1)VALUES(1,1); 2 SROWID2 TRX1 COMMIT; 3 TRX2 UPDATE T SET C1=1WHERE ROWID=SROWID1; 4 SROWID1 TRX2 COMMIT; 5 TRX3 DELETE FROM T WHERE ROWID=SROWID2; 6 SROWID2 TRX3 COMMIT; 7

[0119] The source-side data synchronization service calculates the key of an operation by combining the table ID and the ROWID of the operation.

[0120] 2. The log receiving thread receives operations with LSN 1, extracts its KEY as SROWID1, combines TRXID and the space of the next record, and the length is 32. First, it stores its KEY and transaction ID in the first memory slice, expressed as [memory slice version number (K): memory offset (F)], and its offset in the memory slice is [1:0].

[0121] KEY: SROWID1 TRXID: 1NEXT: empty

[0122] Since this KEY information is the first operation information of the current memory chip, the LSN value of the operation should be set to the starting LSN of the memory chip, which is set to 1.

[0123] 3. Insert the KEY into hash table R. The returned TRXID is 0, which means there is no conflict. Write back the conflict-free TRXID of 0 and the minimum starting LSN of the memory slice of 1 to the operation.

[0124] 4. The log receiving thread receives the operation with LSN 2, extracts its KEY as SROWID2, combines TRXID and the space of the next record, and the length is 32. First, it stores its KEY and transaction ID in the first memory slice, and its offset in the memory slice is [1:32].

[0125] KEY: SROWID1 TRXID: 1NEXT: empty KEY: SROWID2 TRXID: 1NEXT: empty

[0126] 5. Insert the KEY into hash table R. The returned TRXID is 0, which means there is no conflict. Write back the conflict-free TRXID of 0 and the minimum starting LSN of the memory slice of 1 to the operation.

[0127] 6. The log receiving thread receives the operation with LSN 4, extracts its KEY as SROWID1, combines TRXID and the space of the next record, and the length is 32. First, it stores its KEY and transaction ID in the first memory slice, and its offset in the memory slice is [1:64].

[0128] KEY: SROWID1 TRXID: 1NEXT: empty KEY: SROWID2 TRXID: 1NEXT: empty KEY: SROWID1 TRXID: 2NEXT: empty

[0129] 7. Insert the KEY into the hash table R. Since the operation information with KEY SROWID1 already exists in the HASH, the returned conflicting TRXID is 1, which means that the current operation and the transaction with transaction ID 1 have a conflict. Write the conflicting TRXID 1 and the minimum starting LSN of the memory slice of 1 back into the operation.

[0130] 8. Remove the conflicting key information with KEY SROWID1 from the HASH table, and set the address in hash table R with KEY SROWID1 to the next record of the current key information.

[0131] 9. The log receiving thread receives the operation with LSN 6, extracts its KEY as SROWID2, combines TRXID and the space of the next record, and the length is 32. First, it stores its KEY and transaction ID in the first memory slice, and its offset in the memory slice is [1:96].

[0132] KEY: SROWID1 TRXID: 1NEXT: empty KEY: SROWID2 TRXID: 1NEXT: empty KEY: SROWID1 TRXID: 2NEXT: 1:0 KEY: SROWID2 TRXID: 3NEXT: empty

[0133] 10. Insert the KEY into hash table R. The returned TRXID is 1, which means that there is a conflict between the current operation and the transaction with transaction ID 1. Write back the conflicting TRXID 1 and the minimum starting LSN of the memory slice is 1 to the operation.

[0134] After all operation conflict detections are completed, the following detection results are generated:

[0135]

[0136]

[0137] 11. Create three worker threads and execute TRX1, TRX2 and TRX3 respectively in the order of transaction commit.

[0138] 12. Worker thread 1 extracts the TRX1 transaction, registers the transaction information TRXID as 1 in hash table P, and registers the starting LSN of the transaction associated with table information T as 1 in hash table G.

[0139] 13. Worker thread 2 extracts the TRX2 transaction, registers the transaction information TRXID 2 in hash table P, and registers the starting LSN of the transaction associated with table information T, which is 4, in hash table G.

[0140] 14. Worker thread 3 extracts the TRX3 transaction, registers the transaction information TRXID 3 in hash table P, and registers the starting LSN of the transaction associated with table information T, which is 6, in hash table G.

[0141] 15. Worker thread 1 extracts the insertion operation on TRX1, obtains the conflict TRXID and conflict LSN of the operation, and performs conflict detection.

[0142] (1) The first operation is an insertion operation. The conflict TRXID is 0, indicating that there is no conflict.

[0143] (2) Obtain the table information T and the conflict LSN of the operation is 1. Search in the hash table G for conflicting transactions that involve table T and have a commit LSN less than 3 and a minimum starting LSN of the transaction less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0144] (3) When constructing the insert statement, a ROWID return expression needs to be added. After execution, the ROWID mapping relationship [SROWID1, DROWID1] needs to be stored in the ROWID mapping container P.

[0145] (4) The second operation is the insertion operation. The conflict TRXID is 0, indicating that there is no conflict.

[0146] (5) Obtain the table information T and the conflict LSN of the operation is 2. Search in the hash table G for conflicting transactions that involve table T and have a commit LSN less than 3 and a minimum starting LSN of the transaction less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0147] (6) When constructing the insert statement, a ROWID return expression needs to be added. After execution, the ROWID mapping relationship [SROWID2, DROWID2] needs to be stored in the ROWID mapping container P.

[0148] 16. Worker thread 2 extracts the update operation with LSN 4 on TRX2, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0149] The operation conflict TRXID of 1 indicates a conflict with transaction TRX1. The current worker thread needs to be suspended and woken up only after worker thread 1 has completed the synchronization of transaction TRX1.

[0150] 17. Worker thread 3 extracts the deletion operation with LSN 6 on TRX3, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0151] The operation conflict TRXID of 1 indicates a conflict with transaction TRX1. The current worker thread needs to be suspended and woken up only after worker thread 1 has completed the synchronization of transaction TRX1.

[0152] 18. After worker thread 1 executes the commit operation of TRX1, it deletes the current transaction information from hash table P and hash table G respectively, and then wakes up the execution of TRX2 transaction on worker thread 2 and TRX3 transaction on worker thread 3.

[0153] 19. Worker thread 2 extracts the update operation with LSN 4 on TRX2, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0154] (1) The operation conflict TRXID is 1, indicating that TRX1 has a conflict. Since TRX1 has been committed, transaction TRX1 cannot be found in hash table G. Continue to the next step of judgment.

[0155] (2) Obtain the table information T and the conflict LSN of the operation is 1. Search in the hash table G for conflicting transactions that involve table T and have a commit LSN of less than 5 and a minimum starting LSN of less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0156] (3) Since it is an update operation, it is necessary to extract the ROWID value SROWID1 from the operation and obtain its corresponding ROWID value DROWID1 from the target database in the mapping container P. Then, construct a synchronous update statement based on ROWID: UPDATE T SETC1 = 1 WHERE ROWID = DROWID1; to execute the update.

[0157] (4) After executing the commit operation of TRX2, delete the current transaction information from hash table P and hash table G respectively to complete the execution of TRX2 transaction.

[0158] 20. Worker thread 3 extracts the deletion operation with LSN 6 on TRX3, obtains the conflicting TRXID and conflicting LSN of the operation, and performs conflict detection.

[0159] (1) The operation conflict TRXID is 1, indicating that TRX1 has a conflict. Since TRX1 has been committed, transaction TRX1 cannot be found in hash table G. Continue to the next step of judgment.

[0160] (2) Obtain the table information T and the conflict LSN of the operation is 1. Search in the hash table G for conflicting transactions that involve table T and whose commit LSN is less than 7 and whose minimum starting LSN is less than 1. If no conflicting transactions are found, the operation can be executed directly.

[0161] (3) Since it is a deletion operation, the ROWID value SROWID2 of the operation needs to be extracted and its corresponding ROWID value DROWID2 in the target database is obtained in the mapping container P. The synchronous deletion statement is constructed with ROWID as the condition: DELETE FROM TWHERE ROWID = SROWID2; to execute the deletion.

[0162] (4) Remove the ROWID mapping relationship [SROWID2, DROWID2] from the mapping container P.

[0163] (5) After executing the commit operation of TRX3, delete the current transaction information from hash table P and hash table G respectively to complete the execution of TRX3 transaction.

[0164] As can be seen from the above process, transaction TRX1 inserted two rows with the same value, transaction TRX2 updated the first row, and TRX3 deleted the second row. If the user column is used to calculate the conflict key, then the conflict keys for the two insert operations of TRX1, the update operation of TRX2, and the delete operation of TRX3 are all the same. This will cause the execution of these three transactions to form a dependency relationship based on the order of transaction commits; that is, TRX2 depends on the execution of TRX1, and TRX3 depends on the execution of TRX2. When the target synchronization service executes TRX2 and TRX3, these two transactions cannot be concurrent. If only ROWID is used as the conflict key for conflict detection, but ROWID mapping is not used on the target side to locate the rows operated on by TRX2 and TRX3, and the user column is directly used as the condition for update and delete, then when TRX2 and TRX3 are executed concurrently, the update and delete operations will locate the same row, resulting in a conflict deadlock. This invention combines the use of ROWID as the key for conflict detection with a ROWID mapping method. Transaction TRX2 and Transaction TRX3, which can only be executed serially in traditional schemes, can be executed concurrently in this scheme, effectively solving the data synchronization performance problem of the source database when there are duplicate rows of data.

[0165] Example 3:

[0166] It should be noted that for rollback transactions, since the rollback transaction will not be executed, there are three ways to handle the rollback transaction operation in the hash table R.

[0167] The first approach: This method does not roll back the impact of rollback transactions on operations dependent on them in hash table R, nor does it collect conflicting TRXIDs of rollback transaction dependencies. It directly releases all operations of the rollback transaction. This rollback transaction will be depended upon by other operations, and it will still be added to the pending execution list for execution to ensure the continuity of operation dependencies. When using this approach, the entire rollback transaction is released, which can improve the performance of handling rollback transactions. However, operations dependent on the rollback transaction can only execute after the rollback transaction is released in the pending execution list. Since conflicting transactions were not collected during the release operation, the rollback transaction can only be released after all transactions with commit LSNs or rollback LSNs less than its own rollback LSN have been executed, which affects synchronization efficiency. The corresponding approach is that when the target data synchronization service receives a rollback transaction, all operations on this transaction do not need to be executed; they are discarded directly, and only the retained transaction information is added to the pending execution queue. For example, when the target data synchronization service receives a rollback or partial rollback operation, it needs to locate the transaction based on the transaction ID of the operation, and then mark the transaction to indicate that some operations of the transaction have been rolled back. If the operation is a rollback operation, then all operations on this transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed. The transaction information of the rollback transaction cannot be discarded.

[0168] The second approach: While not reversing the impact of rollback transactions on operation dependencies in hash table R, it collects conflicting TRXIDs of rollback transaction dependencies, for example, by recording them in a conflicting TRXID dependency list L. This rollback transaction will be depended upon by other operations, and it will also be added to the pending execution list for execution to ensure the continuity of operation dependencies. When using this approach, conflicting TRXIDs in the operation must be collected before the rollback transaction is released. Although this reduces the processing performance of rollback transactions, it improves the release performance of rollback transactions during execution. Operations dependent on rollback transactions can only be executed after the rollback transaction is released in the pending execution list. However, since conflicting transactions that depend on the rollback transaction were collected during the release operation, the rollback transaction can be released only after these conflicting transactions have completed, without waiting for all transactions with commit LSNs or rollback LSNs less than its own rollback LSN to complete. Compared to the first approach, this partially improves synchronization efficiency. This approach corresponds to the following: when the target data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation. Furthermore, for the rollback operation, the corresponding transaction cannot be discarded, but is added to the queue of pending transactions.

[0169] The third approach involves rolling back the impact of a transaction's operations on the dependencies of operations in hash table R. This rolled-back transaction will not be depended upon by other operations, and it does not need to be added to the pending execution list. When using this approach, after receiving a rollback operation, the transaction needs to roll back the impact of all operations within the transaction on the dependencies of operations in hash table R. Since the transaction size is uncontrollable, rolling back a large transaction will affect the progress of receiving operations, thus impacting synchronization efficiency. In this method, when the target data synchronization service receives a rollback or partial rollback operation, it writes the conflicting TRXIDs saved in the rolled-back operations back to hash table R. The description of the rollback portion in Embodiment 1 of this invention is an example description of this approach.

[0170] The three schemes for handling transaction rollback each have their advantages and disadvantages. The first scheme is suitable for large-scale transaction rollbacks, the second scheme for medium-scale transaction rollbacks, and the third scheme for small-scale transaction rollbacks. Therefore, two parameters, W1 and W2, are set: W1 represents the minimum number of DML operations in a large-scale transaction, and W2 represents the minimum number of DML operations in a medium-scale transaction. Based on this, after receiving DML operations from the source end, the target-side data synchronization service needs to classify and manage the received DML operations by transaction and count the scale S of the DML operations received in each transaction (S is the number of DML operations received in a single transaction). When the target-side data synchronization service receives a rollback operation, it takes corresponding actions based on the scale S of the DML operations in the transaction.

[0171] When the size S of the rollback transaction is greater than or equal to W1, it corresponds to a large-scale transaction rollback in the first scheme. In this case, the impact of the rollback transaction's operations on the operation dependencies in the hash table R is not rolled back, and conflicting TRXIDs of the rollback transaction's operation dependencies are not collected. All operations of the rollback transaction are directly released. This rollback transaction will be depended on by other operations, and the rollback transaction will also be added to the pending execution list for execution to ensure the continuity of operation dependencies.

[0172] When the size S of the rollback transaction is greater than or equal to W2 and less than W1, it corresponds to a medium-sized transaction rollback in the second scheme. In this case, the operation of not rolling back the rollback transaction does not affect the operation dependencies in the hash table R. However, the conflicting TRXIDs of the rollback transaction operation dependencies are collected, and they can be recorded in the conflicting TRXID dependency list L. This rollback transaction will be depended on by other operations, and the rollback transaction should also be added to the pending execution list for execution to ensure the continuity of operation dependencies.

[0173] When the size S of the rollback transaction is less than W2, it corresponds to the small-scale transaction rollback in the third scheme. In this case, the rollback operation affects the operation dependencies in the hash table R, and the rollback transaction does not need to be added to the pending execution list; it is released directly. The description of the rollback part in Embodiment 1 of this invention is an example description when the size S of the rollback transaction is less than W2.

[0174] Furthermore, for partial rollback operations, due to the special nature of partial rollback, which inherently requires traversing all rollback operations, adding a process to collect conflicting TRXIDs from these operations does not significantly degrade the performance of handling partial rollbacks. Based on this, when the target data synchronization service receives a partial rollback operation, it needs to count the total number of rolled-back rows S1. Depending on the size of S1, the following processing is performed: When the number of rollback rows S1 is less than W2, the impact of the rollback operation on the operation dependencies in hash table R is rolled back, i.e., the conflicting TRXIDs saved in the rolled-back operations are written back to hash table R. When the number of rollback rows S1 is greater than or equal to W2, the impact of the rollback operation on the operation dependencies in hash table R is not rolled back, i.e., the conflicting TRXIDs saved in the rolled-back operations are not written back to hash table R, but the conflicting TRXIDs dependent on the rollback operation are collected and recorded in the conflicting TRXID dependency linked list L.

[0175] Based on the above settings, after the target data synchronization service creates a worker thread, the worker thread extracts a transaction from the committed transaction list in the order of transaction commits, and then determines whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to determine whether the rollback transaction has a conflict with the TRXID list L.

[0176] If the rollback transaction does not conflict with the TRXID chain L, it needs to wait for all transactions with a commit LSN less than the current transaction to complete before waking up its waiting worker thread. Then, it removes the current transaction's registration information from hash tables P and G, completing the execution of the current transaction. This is because the operations on the rollback transaction do not need to be executed and are discarded, so the transaction information they depend on is unclear. Therefore, when committing, it must wait for all transactions that committed before it to commit to ensure the correctness of the transaction dependency order.

[0177] If a rollback transaction has a conflicting TRXID chain L, it needs to wait for all conflicting transactions in chain L to be committed before waking up its waiting worker thread. Then, it removes the current transaction's registration information from hash tables P and G, completing the execution of the current transaction. Specifically, it sequentially extracts a conflicting TRXID from chain L, checks if it exists in hash table P, and if it does, waits for that transaction to complete before waking up its own thread. After waking up, it extracts the next conflicting TRXID from chain L and performs the same check until all conflicting transactions in chain L are completed. This is because the operations on the rollback transaction do not need to be executed and are discarded. When discarding the rollback operations, the conflicting TRXIDs of the operations are collected, so their dependent transaction information is clear. Therefore, during release, it is only necessary to wait for these conflicting transactions to be committed to ensure the correctness of the transaction dependency order.

[0178] In summary, since rollback transactions are not executed, three conflict detection schemes arise, each with its own advantages. Appropriate values ​​need to be set to leverage the strengths of each scheme to improve overall synchronization performance. Furthermore, for partial rollback operations of committed transactions, specifically collecting the conflicting TRXIDs referenced during the rollback operation can effectively mitigate the performance loss caused by writing back to the conflict detection hash table R during partial rollback operations, thus improving overall synchronization performance. This embodiment handles rollback transactions and partial rollback operations according to a predetermined operational scale, seeking a balance between the costs of operation rollback and rollback transaction execution, thereby improving overall synchronization performance.

[0179] In some embodiments, the keys in hash table R can be stored either in files or in memory slices. Specifically, the difference lies in the storage method. While file storage reduces performance and impacts collision detection efficiency, the state of hash table R before the failure can be restored using the key values ​​in the stored files after a data synchronization failure. This ensures continuity in collision detection for received operations after the failure, preventing a precipitous drop in synchronization performance. Conversely, while memory slice storage significantly improves collision detection performance compared to file storage, it loses previously detected collision data after a failure. Recovery requires starting from the currently received LSN, leading to a discontinuity in subsequent collision detection and a sharp drop in synchronization performance at the failure point. When storage performance is high (e.g., using SSD fixed storage), the performance requirements for conflict detection can be met, and the efficiency of operation reception and synchronization execution can be balanced. In this case, storing the keys in the hash table R in files can effectively avoid the loss of synchronization performance caused by failures. When storage performance affects the efficiency of conflict detection, the bottleneck of synchronization lies in operation reception. In this case, storing the keys in the hash table R in memory slices can effectively improve the operation reception performance, thereby improving the overall synchronization performance. Embodiment 1 of this application is an example of storing the keys in the hash table R in memory.

[0180] In some embodiments, the MD5 value can be calculated at either the source or the target end as the key for the operation. Calculating the MD5 value for each operation by detecting conflicts in the source-end data synchronization service allows for more accurate conflict MD5 value calculation. However, if the calculation is performed by the target-end data synchronization service, the MD5 value calculated using the primary key information of the target database table will be inaccurate if the primary key structure or column order is inconsistent with the source table. Therefore, it is generally preferred to calculate the MD5 value at the source end as the key for the operation. After receiving the DML operation from the source end, the target-end data synchronization service directly extracts the MD5 value calculated at the source end as the key.

[0181] In some embodiments, when calculating the key for an operation, since the column order in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the key is calculated according to the fixed order of the primary key or unique index combination columns. After the source data synchronization service captures a DML log operation, it obtains the table ID in the log and uses the table ID to retrieve the primary key information and unique index of the table in the source database. After parsing the current DML log operation, the source data synchronization service determines whether the table has a primary key or unique index. If a primary key or unique index exists, it calculates an MD5 value by combining the table ID and the column values ​​of the primary key or unique index as the key for this operation, stores it in the current DML log operation, and sends it to the target synchronization service. It should be noted that when the table has both a primary key and a unique index, the MD5 value calculated by combining the table ID and the primary key is preferred as the key; when the table has no primary key but has multiple unique indexes, the MD5 value calculated by combining the table ID and any one of the unique indexes is selected. If there is no primary key or unique index, after parsing the current DML log operation, the source data synchronization service calculates an MD5 value by combining the table ID and the ROWID value or a custom selection column of the corresponding row. This MD5 value is then stored in the current DML log operation and sent to the target synchronization service. It should be noted that when calculating the key for a table without a primary key or unique constraint, if it is known that there are no duplicate values ​​in any combination of columns, the table's ROWID can be used directly to calculate the key, effectively reducing the cost of key calculation. If the table contains duplicate rows, then a custom selection column approach must be used to calculate the key. The selection column can be chosen based on its distribution, allowing the target synchronization service to prevent deadlocks caused by concurrent synchronization transactions operating on rows with duplicate values.

[0182] Example 4:

[0183] Based on the parallel execution method of transaction operations based on mapping function containers provided in Embodiment 1 above, the present invention also provides a parallel execution device of transaction operations based on mapping function containers that can be used to implement the above method and system, such as... Figure 6 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The parallel execution device for transaction operations based on a mapping function container in this embodiment includes one or more processors 21 and a memory 22. Figure 6 Take a processor 21 as an example.

[0184] Processor 21 and memory 22 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0185] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the parallel execution method of transaction operations based on mapped function containers in Embodiment 1. The processor 21 executes various functional applications and data processing of the parallel execution device for transaction operations based on mapped function containers by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the parallel execution method of transaction operations based on mapped function containers in Embodiment 1.

[0186] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] The program instructions / modules are stored in memory 22. When executed by one or more processors 21, they execute the parallel execution method of transaction operations based on the mapped function container described in Embodiment 1 above. For example, they execute the methods described above. Figures 1-4 The steps shown.

[0188] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0189] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above. For the sake of brevity, they are not provided in detail; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for parallel execution of transaction operations based on a mapping function container, characterized in that, include: After the target data synchronization service starts, it initializes a container P with mapping function to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database. After receiving the DML operation from the source end, the target end data synchronization service extracts the KEY calculated based on the table ID and ROWID in the operation, and then extracts the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice. The conflict TRXID and conflict LSN of the operation are determined based on the KEY, TRXID, and starting LSN. This includes: constructing a detection record X using the current operation's KEY and merging it into hash table R; performing conflict detection during the merging process; calculating the hash value of the KEY of the current detection record X and locating the hash entry using the hash value; determining if the record address in the hash entry is empty; if so, setting the record address in the hash entry to the next address space of the current detection record X, filling the hash entry with the address of the detection record X, and returning 0 as the conflict TRXID, indicating no conflict was detected; otherwise, proceeding to the next conflict detection step; determining if the memory slice version number at the address recorded in the hash entry is less than the version number value of the first memory slice in the memory linked list; if so, it indicates that this record Y has exceeded the inspection range, and setting the record address in the hash entry to the next address space of the current detection record X. The system uses an address space, fills the hash entry with the address of the detected record X, and returns 0 as the conflict TRXID, indicating that no conflict was detected. If not, it proceeds to the next conflict detection step. Based on the address recorded in the hash entry, it loads the corresponding record Y from the memory slice, checks if the KEY of record Y is equal to the KEY of the current detected record X, and if the TRXID of record Y is not equal to the TRXID of the current operation. If so, it indicates a conflict, removes record Y from the record list of the hash entry to shorten the list length, sets the address of the record in the hash entry to the next address space of the current detected record X, fills the hash entry with the address of the current detected record X, and returns the TRXID stored in record Y as the conflict TRXID. If not, it proceeds to the next conflict detection step. It iterates through the next record of record Y and performs conflict detection until the record list ends. The target data synchronization service creates several worker threads to process operations based on their conflicting TRXIDs and conflicting LSNs. When a worker thread executes an operation, it determines the type of the operation and classifies it based on the container P according to the type.

2. The method for parallel execution of transaction operations based on a mapping function container according to claim 1, characterized in that, After the target data synchronization service starts, it initializes a hash table R and initializes the record addresses in the hash entries of hash table R to be empty, which is used to store and locate the received operations. The target data synchronization service creates two memory slices to store the keys in hash table R. Each memory slice is set with an incrementing version number in the order of creation, and the two memory slices are added to the memory linked list.

3. The method for parallel execution of transaction operations based on a mapping function container according to claim 1, characterized in that, The generation of the KEY specifically includes: generating an MD5 value based on the ID and ROWID of the DML operation table as the KEY for the operation.

4. The method for parallel execution of transaction operations based on a mapping function container according to claim 2, characterized in that, The determination of conflicting TRXIDs and conflicting LSNs based on KEY, TRXID, and starting LSN specifically includes: During the merging process, conflict detection is performed. If a conflicting key is found, the TRXID value corresponding to the conflicting key is returned; otherwise, 0 is returned as the corresponding TRXID, indicating that there is no conflict. Set the TRXID obtained from the conflict detection as the conflict TRXID, and write the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R into the current operation and set it as the conflict LSN.

5. The method for parallel execution of transaction operations based on a mapping function container according to claim 2, characterized in that, The memory list includes two memory blocks. When the first memory block is full, the system switches to the second memory block. When the second memory block is full, the first memory block in the memory list is first removed, and then a new memory block is created and added to the memory list using an incrementing version number.

6. The method for parallel execution of transaction operations based on a mapping function container according to any one of claims 1-5, characterized in that, The step of determining the type of the current operation when it is executed by a worker thread, and classifying and processing it based on the container P according to the different types, specifically includes: If the current operation is an INSERT operation, construct the ROWID return expression when constructing the INSERT statement. After execution, insert the ROWID of the current operation and the ROWID returned by the database into the mapping container P. If the current operation is an UPDATE operation, query the ROWID of the travel data in the target database in the mapping container P using the ROWID of the current operation, and then use the ROWID of the target database as the update condition to execute the update statement. If the current operation is a DELETE operation, query the ROWID of the travel data in the target database in the mapping container P using the ROWID of the current operation, then execute the deletion statement using the ROWID of the target database as the deletion condition, and clear the mapping relationship from container P.

7. A device for parallel execution of transaction operations based on a mapping function container, characterized in that, The apparatus includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the parallel execution method of transaction operations based on a mapping function container as described in any one of claims 1-6.

8. A system for parallel execution of transaction operations based on a mapping function container, executing the method for parallel execution of transaction operations based on a mapping function container as described in any one of claims 1-6, characterized in that, The system includes a ROWID mapping module, a KEY acquisition module, a conflict information acquisition module, and a classification processing module, wherein: The ROWID mapping module is used to initialize a container P with mapping function after the target data synchronization service starts, which is used to store the mapping relationship between the source database ROWID and the corresponding ROWID in the target database. The KEY acquisition module is used to extract the KEY calculated based on the table ID and ROWID in the operation after the target data synchronization service receives the DML operation from the source end, and then extract the transaction ID in the operation as the TRXID. The KEY and TRXID are combined into a record and saved to the current memory slice. If the record is the first record of the current memory slice, the log LSN corresponding to the operation is used as the starting LSN of the current memory slice. The conflict information acquisition module is used to determine the conflicting TRXID and conflicting LSN of the operation based on the KEY, TRXID and the starting LSN. The classification processing module is used to process operations based on the conflict TRXID and conflict LSN after creating several worker threads in the target data synchronization service. When the worker thread executes the current operation, it determines the type of the current operation and performs classification processing based on the container P according to the different types.

9. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the parallel execution method of transaction operations based on a mapping function container as described in any one of claims 1-6.

Images