Optimization methods, devices, and systems for target-side concurrency in database synchronization

Abstract

This invention relates to the field of database technology, and provides an optimization method, apparatus, and system for target-side concurrency in database synchronization. The method includes: merging the DML masks of each transaction in a corresponding thread to obtain a thread operation mask; before executing a first DML operation, the first thread compares the DML mask of the first DML operation with the thread operation masks of each second thread to determine whether there is any overlap in operation positions between the first DML operation and the second thread; if it is determined that there is overlap in operation positions between the first DML operation and the second thread, the first DML operation is executed after the second thread submits all second transactions. This invention uses DML masks to represent the operation positions of DML operations, and by merging the DML masks, it achieves control over the operation positions of each thread. By judging the overlap of operation positions, it can determine whether there may be concurrency conflicts between threads, ensuring the normal processing of threads with concurrency conflicts.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to an optimization method, apparatus and system for target-end concurrency in database synchronization. Background Technology

[0002] Currently, heterogeneous database synchronization technology based on database log analysis is widely used. This technology captures incremental data from the source database and sends it to the target database. On the target database, the incremental data is applied through a common database access interface, thus achieving data replication. Because this technology uses a common database interface, it supports replication of heterogeneous database systems and heterogeneous operating system environments. Furthermore, the target standby database system is readable and writable, making it a "dual-active" system.

[0003] To improve the performance of database insertion operations, the target side typically employs multiple execution threads. This increases concurrency by executing different transactions simultaneously while maintaining the transaction commit order. However, this approach can lead to dependencies between transactions, resulting in concurrency conflicts. Some operations may need to wait for earlier transactions to complete before execution, or the correct execution order must be guaranteed; otherwise, synchronous operations may fail or deadlocks may occur. Therefore, accurately identifying concurrency conflicts between different transactions has become a crucial technical problem that urgently needs to be solved in order to maximize concurrency and improve performance.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem to be solved by this invention is how to resolve concurrent conflicts between different things.

[0006] The present invention adopts the following technical solution:

[0007] Firstly, this embodiment provides an optimization method for target-side concurrency in database synchronization, the method including:

[0008] The DML masks of each transaction in the corresponding thread are merged to obtain the thread operation mask of the thread; wherein, a DML operation in a transaction has a DML mask to represent the operation position of the DML operation, and a thread operation mask corresponds to a type of operation of a table entry by a thread to represent all operation positions of the thread performing the corresponding type of operation on the corresponding table entry; the operation position is the operation position in the source database;

[0009] Before executing the first DML operation, the first thread compares the DML mask of the first DML operation with the thread operation masks of each second thread to determine whether there is any overlap in operation positions between the first DML operation and the second thread; wherein, the second thread is the thread that precedes the first thread.

[0010] If it is determined that there is an overlap between the operation positions of the first DML operation and the second thread, the first DML operation is executed after the second thread submits its second transactions, in order to resolve the concurrency conflict between the first and second threads.

[0011] Preferably, the DML mask is obtained based on DML operation analysis, specifically including:

[0012] Extract the primary key or ROWID from the DML operation, generate N bits, set the value of the i%Nth bit to 1, and set the values ​​of the other bits to 0 to obtain the DML mask; where i is the primary key or ROWID of the DML operation.

[0013] Preferably, the DML mask includes multiple types, including DML insertion masks, DML update masks, and DML deletion masks; the merging of the DML masks of each transaction in the first thread to obtain the thread operation mask of the first thread specifically includes:

[0014] Perform an OR operation on the DML masks of the same type for the same table entry in each transaction of the first thread to obtain the thread operation mask for the corresponding category of operation of the first thread on that table entry.

[0015] Preferably, comparing the DML mask of the first DML operation with the thread operation masks of each second thread to determine whether there is any overlap in operation positions between the first DML operation and the second thread specifically includes:

[0016] According to the preset conflict rules, find the first operation category that conflicts with the first DML operation;

[0017] If the operation category corresponding to the thread operation mask of the second thread is the first operation category, then the DML mask of the first DML operation and the thread operation mask of the second thread are ANDed to obtain the operation result.

[0018] If the result of the operation is greater than 0, it is determined that there is an overlap between the operation positions of the first DML operation and the second thread.

[0019] Otherwise, it is determined that there is no overlap between the operation positions of the first DML operation and the second thread.

[0020] Preferably, transactions are processed in batches by multiple threads, specifically including:

[0021] Retrieve the nth batch of transactions, distribute the nth batch of transactions to multiple threads for processing, and after the nth batch of transactions is processed, process the (n+1)th batch of transactions; where a batch of transactions includes multiple transactions.

[0022] Secondly, this embodiment also provides an optimization method for target-side concurrency in database synchronization, the method including:

[0023] Parse the operation log to obtain DML operations, fill the DML operations with one or more of the corresponding table entry information, primary key, and ROWID, and send the filled DML operations to the target end so that the target end can analyze the filled DML operations to obtain the DML mask, thereby executing the optimization method for target end concurrency in database synchronization described in the first aspect.

[0024] Thirdly, the present invention also provides an optimization apparatus for target-end concurrency in database synchronization, used to implement the optimization method for target-end concurrency in database synchronization described in the first aspect, the apparatus comprising:

[0025] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the optimization method for target concurrency in database synchronization as described in the first aspect.

[0026] Fourthly, the present invention also provides an optimization apparatus for target-end concurrency in database synchronization, used to implement the optimization method for target-end concurrency in database synchronization described in the second aspect, the apparatus comprising:

[0027] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the optimization method for target concurrency in database synchronization as described in the second aspect.

[0028] Fifthly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect and / or the method described in the second aspect.

[0029] A sixth aspect provides a chip comprising: a processor and an interface for calling and running a computer program stored in a memory, performing methods as described in the first to second aspects and any one thereof.

[0030] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a computer or processor, cause the computer or processor to perform the methods of the first to sixth aspects and any one thereof.

[0031] Eighthly, an optimization system for target-end concurrency in database synchronization is provided, comprising an optimization device for target-end concurrency in database synchronization as described in the third aspect and an optimization device for target-end concurrency in database synchronization as described in the fourth aspect, and each using an optimization method for target-end concurrency in database synchronization as described in the first aspect and the second aspect to complete the interaction between the optimization device for target-end concurrency in database synchronization and the optimization device for target-end concurrency in database synchronization.

[0032] This invention uses DML masks to represent the operation positions of DML operations, and then merges the DML masks to control the operation positions of each thread. Based on this, by judging the overlap of operation positions, it can determine whether there may be concurrent conflicts between threads, and ensure the normal processing of each thread with concurrent conflicts. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0034] Figure 1 This is a flowchart illustrating the first optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0035] Figure 2 This is a flowchart illustrating the second optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram illustrating an application scenario of an optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0037] Figure 4 This is a flowchart illustrating the third optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of preset conflict rules in an optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0039] Figure 6 This is a flowchart illustrating the fourth optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0040] Figure 7 This is a schematic diagram illustrating an application scenario of another optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0041] Figure 8 This is a schematic diagram of an optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0042] Figure 9 This is a schematic diagram of another optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0043] Figure 10 This is a schematic diagram of another optimization method for target-end concurrency in database synchronization provided by an embodiment of the present invention;

[0044] Figure 11 This is a schematic diagram of the architecture of an optimization device for target-side concurrency in database synchronization provided by an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0047] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0048] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0049] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0050] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] Example 1:

[0052] Embodiment 1 of this invention provides an optimization method for concurrency on the target end in database synchronization. The execution subject in this embodiment is the target end, such as... Figure 1 As shown, the method includes:

[0053] In step 201, the DML masks of each transaction in the corresponding thread are merged to obtain the thread operation mask. Each DML operation within a transaction has a DML mask representing the operation position. A thread operation mask represents all operation positions performed by the thread on a given table entry for a specific type of operation. Operation types include insert, update, and delete operations, and the operation positions are those in the source database. In some application scenarios, a thread performs only one type of operation on a table entry; in this case, a thread operation mask corresponds to a single operation on the table entry. If a thread performs multiple types of operations on a table entry, such as insert, update, and delete operations, then a thread operation mask corresponds to each type of operation performed by the thread on that table entry.

[0054] The operation location can be a precise row within the corresponding table entry, or it can be a general area within the corresponding table entry, such as multiple rows being treated as a single operation location. In practice, transactions are processed in batches by multiple threads, specifically including:

[0055] The nth batch of transactions is retrieved and distributed to multiple threads for processing. After the nth batch of transactions is processed, the (n+1)th batch of transactions is processed. A single batch of transactions comprises multiple transactions. Transaction processing includes transaction execution and transaction commit. In practical applications, a single thread may handle multiple transactions. In this case, the thread will merge the multiple transactions, meaning that multiple transactions are committed only after all of them have been executed.

[0056] In step 202, before executing the first DML operation, the first thread compares the DML mask of the first DML operation with the thread operation masks of each second thread to determine whether there is any overlap in operation positions between the first DML operation and the second thread; wherein, the second thread is the thread that precedes the first thread; the second thread being the thread that precedes the first thread is described based on a preset order, which is a preset order analyzed in advance by those skilled in the art, and is the order in which each thread is woken up under the condition that there is no conflict between threads.

[0057] In step 203, if it is determined that there is an overlap between the operation positions of the first DML operation and the second thread, the first DML operation is executed after the second thread submits each second transaction to resolve the concurrency conflict between the first thread and the second thread.

[0058] In step 204, if it is determined that there is no overlap between the operation positions of the first DML operation and the second thread, it is considered that there is no concurrent conflict between the first DML operation and the second thread. Therefore, there is no restriction on the execution and submission order of the first DML operation and the second thread, and the first DML operation can be executed directly.

[0059] This embodiment uses DML masks to represent the operation positions of DML operations, and then merges the DML masks to control the operation positions of each thread. Based on this, by judging the overlap of operation positions, it can determine whether there may be concurrent conflicts between threads, and ensure the normal processing of each thread with concurrent conflicts.

[0060] In practical application scenarios, the execution subject of the method described in this embodiment is the target end, and the DML mask is obtained based on DML operation analysis. In optional implementations, such as... Figure 2 As shown, it specifically includes:

[0061] In step 301, the primary key or ROWID is extracted from the DML operation to generate N bits; N is a preset value obtained by those skilled in the art based on experience.

[0062] In step 302, the value of the i%Nth bit in the N bits is set to 1, and the values ​​of the other bits are set to 0 to obtain the DML mask; where i is the primary key or ROWID of the DML operation.

[0063] by Figure 3 Taking Table A as an example, assuming N is 7, when there is a DML operation INSERT INTO A(ID,STATE)VALUES(4,0), that is, when the primary key corresponding to the DML operation is 4, then the DML mask corresponding to the DML operation can be 0010000, that is, the value of the 4th % 7 = 4th bit of the DML mask is 1, and the value of other bits is 0. It should be noted that here it is calculated in the order of the 0th to the 6th bit from right to left. In actual use, it is also feasible to calculate the DML mask in the order of the 0th to the 6th bit from left to right.

[0064] If, as the number of data rows in table A increases, a subsequent DML operation, INSERT INTO A(ID,STATE)VALUES(11,0), occurs, the primary key corresponding to this DML operation is 11. Since 11%7 = 4, the DML mask corresponding to this DML operation is also 0010000. Therefore, it can be understood that the DML mask 0010000 represents all data rows in table A whose primary key satisfies key%7 = 4. That is, one DML mask represents multiple data rows. This design is based on the consideration that data tables typically have a large number of rows. Using a single DML mask to represent a unique data row would result in an excessively long DML mask, consuming significant resources and hindering the normal operation of transaction processing. Furthermore, considering that in existing technologies, this embodiment is mainly applied to incremental data ingestion scenarios, which often exhibit block operation characteristics, meaning that a batch of transactions operates on a single piece of data... In a database, data rows located close to each other are relatively rare, and operations on distant data rows are less common. Based on this characteristic, this embodiment proposes a method to obtain a DML mask by setting the value of the i%Nth bit out of N bits to 1. By appropriately setting the value of N to cover the size of one or more blocks (i.e., the area of ​​data rows that may be modified by a batch of transactions), the purpose of one DML mask representing only one data row in a batch can be achieved. This makes it possible to determine whether there are overlapping operation positions in a batch of transactions while reducing the resources occupied by the DML mask.

[0065] Correspondingly, the DML mask includes multiple types, including DML insertion masks, DML update masks, and DML deletion masks; the merging of the DML masks of each transaction in the first thread to obtain the thread operation mask of the first thread specifically includes:

[0066] Perform an OR operation on the DML masks of the same type for the same table entry in each transaction of the first thread to obtain the thread operation mask for the corresponding category of operation of the first thread on that table entry.

[0067] The step involves comparing the DML mask of the first DML operation with the thread operation masks of each second thread to determine whether there is any overlap in operation positions between the first DML operation and the second thread. Figure 4 As shown, it specifically includes:

[0068] In step 401, according to the preset conflict rules, the first operation category that conflicts with the first DML operation is found;

[0069] In step 402, if the operation category corresponding to the thread operation mask of the second thread is the first operation category, then the DML mask of the first DML operation and the thread operation mask of the second thread are ANDed to obtain the operation result.

[0070] In step 403, if the calculation result is greater than 0, it is determined that there is an overlap in operation positions between the first DML operation and the second thread; otherwise, it is determined that there is no overlap in operation positions between the first DML operation and the second thread. The determination that there is no overlap in operation positions between the first DML operation and the second thread includes two cases: the first case: the calculation result is not greater than 0; the second case: the operation category corresponding to the thread operation mask of the second thread is not the first operation category.

[0071] The preset conflict rules are as follows: Figure 5 As shown, when the type of the first DML operation is an insert operation (i.e. Figure 5 When performing an insert operation, the first operation category is a delete operation (i.e., ...). Figure 5 In the context of `delete`, when the type of the first DML operation is an update operation (i.e., `update`),... Figure 5 When performing an update operation, the first operation category is either an update operation or an insert operation. When the first DML operation is of type delete, the first operation category is either an update operation or an insert operation.

[0072] Figure 5 The preset conflict rules shown are designed to simplify the determination of overlapping operation positions to the greatest extent and maximize the concurrent execution of transactions. Specifically, they take into account that in actual use, there is no situation where an update operation is performed on the same row of data before an insert operation (because the row already exists before the update operation); similarly, there is no situation where an insert operation is performed on the same row of data before another insert operation; there is no situation where a delete operation is performed on the same row of data before another delete operation; and after performing a delete operation on the corresponding row (assuming the rowID is n), the row with the original rowID n+1 will replace the row, that is, the rowID will become n. If an update operation is subsequently performed on the row with rowID n, the update operation and the previous delete operation, although having the same rowID, still target different rows of data, that is, the operation positions do not overlap. Therefore, the above four situations are considered to be non-conflicting, and other situations are considered to be potentially conflicting. Figure 5 The preset conflict rules are shown.

[0073] The following table shows initial data as an example:

[0074] ROWID ID DATA 1 1 A 2 2 A 3 3 A

[0075] Operations on different rows can be executed concurrently, using the same rowid or primary key value to define the same row in the table. While rowids and primary key values ​​must not be duplicated, from the perspective of dynamic data changes, primary key values ​​can be reused (i.e., after a delete and then insert, the primary key values ​​can be the same). Whether rowids can be reused depends on the database type. When operations A and B are executed in a preset order, if a conflict is detected according to the following conditions, it means that the execution order must be guaranteed. These conditions include:

[0076] (1) Operation A: UPDATE T SET DATA = 'B' WHERE ID = 1; where ROWID is 1;

[0077] Operation B: INSERT INTO T(ID,DATA)VALUES(4,'A'); Here, the ROWID of operation B cannot be 1, because the row with ROWID 1 already exists, so this case is not considered and it is determined to be conflict-free.

[0078] (2) Operation A: Execute DELETE; Operation B: Execute INSERT. There may be a conflict. The conflict occurs when ROWID is reused. For example, if the ROWID of Operation A is 3, the ROWID of Operation B is also 3.

[0079] (3) Operation A: Execute INSERT; Operation B: Execute INSERT. There is no conflict because ROWID cannot be repeated.

[0080] (4) Operation A: Execute UPDATE; Operation B: Execute UPDATE. There may be conflicts, such as: Operation A: UPDATE TSET DATA = 'B' WHERE ID = 1; where ROWID is 1; Operation B: UPDATE T SET DATA = 'C' WHERE ID = 1; where ROWID is 1.

[0081] (5) Operation A: Execute DELETE; Operation B: Execute UPDATE. There is no conflict because it is impossible to update a non-existent row.

[0082] (6) Operation A: Execute INSERT; Operation B: Execute UPDATE, then there may be a conflict.

[0083] (7) Operation A: Execute UPDATE; Operation B: Execute DELETE, then there may be a conflict.

[0084] (8) Operation A: Execute DELETE; Operation B: Execute DELETE. There is no conflict because it is impossible to delete a non-existent row.

[0085] (9) Operation A: Execute INSERT; Operation B: Execute DELETE, which may result in a conflict.

[0086] It should be noted here that... Figure 5 The preset conflict rule shown is a preferred implementation of this embodiment. In another optional implementation, all cases can be considered as potentially conflicting, and delete, update, and insert operations can be treated as the same type of operation. All operations performed by a thread on a table entry are mapped to a single thread operation mask, and the thread operation mask is directly ANDed with the first DML operation. If the result is greater than 0, it is determined that the first DML operation and the second thread have overlapping operation positions. This optional implementation can also solve the problem of transaction concurrency conflicts, but because some actual non-overlapping operation positions are also judged as overlapping, it will affect the degree of concurrent execution of transactions.

[0087] In practical applications, besides the implementation method described above (hereinafter referred to as the first implementation method), there are several other optional implementation methods for the DML mask. Different DML masks may lead to different methods for merging the DML masks of each transaction in the first thread and for comparing the DML mask of the first DML operation with the thread operation masks of each second thread. The following examples illustrate two optional implementation methods:

[0088] The second implementation method is as follows: Generate N1+N2 bits, set the value of the (i / N2)%N1th bit to 1, set the value of the i%N2+N1th bit to 1, and set the values ​​of the other bits to 0 to obtain the DML mask; where i is the primary key or ROWID of the DML operation; that is, the first N1 bits are used to represent the large area where the operation position of the DML operation is located, and the last N2 bits are used to represent the more precise position of the operation position of the DML operation in the large area, thereby expanding the scope of concurrent conflict judgment.

[0089] The third implementation method is to further combine information such as table ID and type ID corresponding to different types of DML operations, based on the first or second implementation method.

[0090] It should be noted that in practical use, a table operation mask may also be introduced. That is, for a table involved in a transaction, a table operation mask is used to represent all the operation positions of all DML operations performed by the transaction on the table. Taking the first implementation method mentioned above as an example, the table operation mask is the result of ORing the DML masks corresponding to all DML operations of the transaction on the table. In order to reduce resource consumption, the table operation mask may also be stored directly, and the thread operation mask of the thread may be obtained by merging the table operation masks. Finally, the DML mask and the thread operation mask are compared to obtain the concurrency conflict result.

[0091] Example 2:

[0092] Having provided an optimization method for target-side concurrency in database synchronization as described in Embodiment 1, this invention will further provide another optimization method for target-side concurrency in database synchronization. This is to elaborate on the method of Embodiment 1 from the perspective of the source end and to further analyze its design principles in depth. It should be noted that the execution subject of the method in Embodiment 1 is the target end, while the execution subject of the method in Embodiment 2 is the source end. Concurrency conflict handling is achieved through the cooperation between the source and target ends. The method of Embodiment 1 will not be described again in this embodiment.

[0093] This embodiment provides an optimization method for target-side concurrency in database synchronization. The method includes: parsing operation logs to obtain DML operations, filling the DML operations with one or more of the following: corresponding table entry information (also known as table description information), primary key, and ROWID, and sending the filled DML operations to the target end so that the target end can analyze the filled DML operations to obtain the DML mask, thereby executing the optimization method for target-side concurrency in database synchronization described in Embodiment 1.

[0094] It is understandable that the source end has a log parsing service deployed. The logs include DML operations, description information of tables in the database (table name, column name) and primary key information, as well as ROWID, new value and old value of the DML operation row. The description information, primary key or ROWID are filled into the DML operation and sent to the target end so that the target end can generate transactions and DML masks according to the filled DML operation, thereby performing database synchronization.

[0095] Example 3:

[0096] Based on the method described in Embodiment 1, this invention, combined with specific application scenarios, and using technical descriptions within those scenarios, elucidates the implementation process of the invention's features in those scenarios. It should be noted that, for ease of explanation, this embodiment uses the example of a single thread performing only one type of operation on a single table entry.

[0097] The optimization method for target-side concurrency in database synchronization described in this embodiment specifically includes:

[0098] The target end receives logs from the source end and categorizes operations according to transaction IDs. Each transaction maintains an N-byte "table operation mask" initialized to 0 for each DML operation type on each table X involved, denoted as i. X d X and u X These correspond to insert, delete, and update operations, respectively. Each mask has a maximum of 8*N available mask positions. To reduce hash collisions, the number of available mask positions is a prime number Q. This mask is used to describe the table, operation, and row information involved in the transaction, as follows: When a DML operation is received from the source, the value that uniquely identifies the row in the operation, such as the primary key or ROWID, is extracted. A hash algorithm, such as n = value % Q, is used to calculate the mask position n, and the mask position n corresponding to the table operation type is set to 1.

[0099] Since the execution thread retrieves a batch of transactions for execution each time, there is a possibility of transaction merging. Before execution begins, the table operation masks of all transactions with the same operation are merged (i.e., the table operation masks of the same operation category within the same thread are merged). This is done by performing a bitwise OR operation to obtain the merged mask (i.e., the thread operation mask in Example 1), which is denoted as I. X D X and U X The execution threads have a sequential order. Before executing each DML operation, it is necessary to determine whether there is a potential conflict with an earlier transaction. If a conflict is possible, the operation must wait for the potentially conflicting transaction to commit before it can continue. The determination method is as follows: Similar to the method used during reception, first calculate the mask position n corresponding to the operation, then traverse the "merge mask" of the corresponding table backward, and check whether the mask position n is 1. If it is, it indicates that a conflict exists.

[0100] like Figure 6 As shown, the specific operation steps are as follows:

[0101] In step 501, a log parsing service is deployed at the source end to populate the table description information, column values, and ROWID values ​​involved in the parsed DML operations.

[0102] In step 502, the target data synchronization service receives log operations sent by the source and categorizes them according to transactions. When a DML operation is received, the value that uniquely identifies a row, such as a primary key or ROWID, is extracted. Using the hash algorithm n = value % Q, the mask position n is calculated, and then the table operation mask position corresponding to the transaction is set to 1, thus generating the table operation mask. When a commit operation is received, the transaction is added to the waiting execution queue.

[0103] In step 503, the target data synchronization involves multiple execution threads, each retrieving a batch of transactions from the waiting execution queue and executing them into the database.

[0104] In step 504, after the execution thread retrieves the transaction, it first performs a merging of the table operation masks, i.e., a bitwise OR operation, to obtain the merge mask I for each table X. X D X and U X .

[0105] Before step 505, before executing the DML operation (i.e., the first DML operation), extract the value that uniquely identifies a row, such as the primary key or ROWID. Using the hash algorithm n = value % Q, calculate the mask position n, which can be understood as the bit position where the value of 1 is located in the DML mask. Traverse the merge mask of the table forward according to the order of submission. First, determine if there is a conflict based on the operation type (i.e., according to the preset conflict rules, find the first operation category that conflicts with the first DML operation; if the operation type matches the first operation category, it is determined that a conflict may exist). If a conflict is determined, then check the mask. If the value of the merge mask position n is 1, it indicates a conflict, and execution needs to wait for all conflicting threads to submit before proceeding. One method to determine if the value of the merge mask position n is 1 is to directly perform a bitwise AND operation between the merge mask and the DML operation mask. If the result of the AND operation is 0, there is no conflict; if the result is greater than 0, a conflict exists.

[0106] In step 506, the execution thread traverses the DML operations of the transaction, repeating step 505 above until all operations are completed, then commits and wakes up those waiting for it, and then returns to step 504 to execute the next batch of transactions.

[0107] The above steps can be understood as follows: First, the target data synchronization service receives logs sent by the source and categorizes them according to transactions. When a DML operation is received, it extracts the value that uniquely identifies a row in the operation, calculates the mask position using a hash algorithm, and then sets the table operation mask position corresponding to the transaction to 1. When a commit message is received, the transaction is added to the waiting execution queue, and the execution thread retrieves a batch of transactions from this queue and executes them into the database. Second, after retrieving a transaction, the execution thread first merges the table operation masks before starting execution. Next, before the execution thread executes the DML operation, it extracts the value that uniquely identifies a row in the operation, calculates the mask position using a hash algorithm, and then traverses the merge mask of the table backward in the commit order, determining conflicts based on the operation type and mask position. If a conflict exists, it needs to wait for the conflicting thread to commit before execution.

[0108] Finally, after the execution thread has traversed and executed all operations of the transaction, it commits and wakes up the waiting threads, and then retrieves the next batch of transactions from the waiting queue for execution.

[0109] To exist Figure 3 Table A shown and existence Figure 7 For example, in Table B shown, Figure 3 The data shown is the initial data in Table A. Figure 7 The data shown is the initial data for Table B. Figure 3 and Figure 7 In this context, ID represents the primary key.

[0110] Assuming the available number of bits in the DML mask is Q = 7, meaning the DML mask has 7 bits, the initial value is 0000000, and the mask positions are 0, 1, ..., 6 from right to left. The hash algorithm for the DML mask positions is n = ID % 7. The source end sequentially generates and submits the following transactions to the target end, so that the target end can retrieve the following at a certain time: Figure 8 Taking a batch of transactions as an example, the DML mask for each transaction is as follows: Figure 8 As shown, the table operation masks corresponding to each transaction are as follows: Figure 9 As shown.

[0111] Assuming there are 8 execution threads, each executing the above transactions, the specific details of the transactions for each thread are as follows: Figure 10 As shown, the table operation mask is further merged to obtain the merged mask, which is the thread operation mask, as shown below. Figure 10 As shown, the index of the merge mask is the table name + execution sequence number.

[0112] The concurrency conflicts between threads can be analyzed using the DML mask and the thread operation mask, as shown in the table below:

[0113]

[0114]

[0115]

[0116] Taking the DML operation UPDATE B SET STATE = 1 WHERE ID = 4 in sequence number 2 (i.e., thread 2) as a specific example, since its corresponding DML mask U = 0010000 and the thread operation mask I in thread 1 that precedes this DML mask... B1 The result of the AND operation on 1010000 is 0010000, which means the result of the AND operation is greater than 0. Therefore, this DML mask conflicts with thread 1, and the DML operation can only be executed after all transactions of thread 1 have been committed.

[0117] Example 4:

[0118] like Figure 11 The diagram shown is an architectural schematic of an optimization device for target-side concurrency in database synchronization according to an embodiment of the present invention. This optimization device for target-side concurrency in database synchronization includes one or more processors 21 and a memory 22. Figure 11 Take a processor 21 as an example.

[0119] Processor 21 and memory 22 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0120] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the optimization method for target-end concurrency in database synchronization in Embodiment 1 or Embodiment 2. The processor 21 executes the optimization method for target-end concurrency in database synchronization by running the non-volatile software programs and instructions stored in the memory 22.

[0121] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the optimization method for target-end concurrency in database synchronization described in Embodiment 1 or Embodiment 2.

[0123] This embodiment also provides an optimization system for target-end concurrency in database synchronization, including at least two of the above-described optimization devices for target-end concurrency in database synchronization, each using the optimization method for target-end concurrency in database synchronization described in claim 1 and executing the optimization method for target-end concurrency in database synchronization described in claim 2, to complete the interaction between the two optimization devices for target-end concurrency in database synchronization.

[0124] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0125] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An optimization method for target-end concurrency in database synchronization, characterized in that the method... include: The DML masks of each transaction in the corresponding thread are merged to obtain the thread operation mask of the thread; wherein, a DML operation in a transaction has a DML mask to represent the operation position of the DML operation, and a thread operation mask corresponds to a type of operation of a table entry by a thread to represent all operation positions of the thread performing the corresponding type of operation on the corresponding table entry; the operation position is the operation position in the source database; Before executing the first DML operation, the first thread compares the DML mask of the first DML operation with the thread operation masks of each second thread to determine whether there is any overlap in operation positions between the first DML operation and the second thread; wherein, the second thread is the thread that precedes the first thread. If it is determined that there is an overlap between the operation positions of the first DML operation and the second thread, the first DML operation is executed after the second thread submits all the second transactions, so as to resolve the concurrency conflict between the first thread and the second thread. The step of comparing the DML mask of the first DML operation with the thread operation masks of each second thread to determine whether there is any overlap in operation positions between the first DML operation and the second thread specifically includes: According to the preset conflict rules, the first operation category that conflicts with the first DML operation is found; if the operation category corresponding to the thread operation mask of the second thread is the first operation category, the DML mask of the first DML operation and the thread operation mask of the second thread are ANDed to obtain the operation result; if the operation result is greater than 0, it is determined that there is an overlap between the operation positions of the first DML operation and the second thread; otherwise, it is determined that there is no overlap between the operation positions of the first DML operation and the second thread.

2. The optimization method for target-end concurrency in database synchronization according to claim 1, characterized in that, The DML mask is obtained based on DML operation analysis, specifically including: Extract the primary key or ROWID from the DML operation, generate N bits, set the value of the i%Nth bit to 1, and set the values ​​of the other bits to 0 to obtain the DML mask; where i is the primary key or ROWID of the DML operation.

3. The optimization method for target-end concurrency in database synchronization according to claim 2, characterized in that, The DML mask includes multiple types, including DML insert mask, DML update mask, and DML delete mask; the merging of the DML masks of each transaction in the corresponding thread to obtain the thread operation mask of the corresponding thread specifically includes: Perform an OR operation on the DML masks of the same type for the same table entry in each transaction of the first thread to obtain the thread operation mask for the corresponding category of operation of the first thread on that table entry.

4. The optimization method for target-end concurrency in database synchronization according to claim 1, characterized in that, Transactions are processed in batches by multiple threads, specifically including: Retrieve the nth batch of transactions, distribute the nth batch of transactions to multiple threads for processing, and after the nth batch of transactions is processed, process the (n+1)th batch of transactions; where a batch of transactions includes multiple transactions.

5. An optimization method for target-side concurrency in database synchronization, characterized in that, The methods include: Parse the operation log to obtain DML operations, fill the DML operations with one or more of the corresponding table entry information, primary key, and ROWID, and send the filled DML operations to the target end so that the target end can analyze the filled DML operations to obtain the DML mask, thereby executing the optimization method for target end concurrency in database synchronization as described in any of claims 1-4.

6. An optimization device for target-end concurrency in database synchronization, characterized in that, The device includes: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the optimization method for target-end concurrency in database synchronization as described in any one of claims 1-4.

7. An optimization device for target-end concurrency in database synchronization, characterized in that, The device includes: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the optimization method for target-end concurrency in database synchronization as described in claim 5.

8. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the optimization method for target-end concurrency in database synchronization as described in any one of claims 1-4, or the optimization method for target-end concurrency in database synchronization as described in claim 5.

9. An optimization system for target-end concurrency in database synchronization, characterized in that, This includes the optimization device for target-end concurrency in database synchronization as described in claim 6 or the optimization device for target-end concurrency in database synchronization as described in claim 7.

Images