Data processing method, system, device and storage medium for distributed database

Abstract

The embodiment of the present application provides a data processing method, system, device and storage medium for a distributed database, which uses the global timestamp TSO to sort the transactions in the transQueue, creates a local transaction list through the XA Event and the waitTrans in the transQueue, and generates a primary sorting queue according to the local transaction list; performs multi-way merge sorting on the primary sorting queue of each DN to generate a global sorting queue; merges local transactions with the same global timestamp TSO for the global sorting queue, and generates a global binary log Binlog. This ensures the global orderliness and integrity of transactions in the data processing and replication process of the distributed database.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and system for a distributed database, an electronic device and a storage medium. Background Art

[0002] MySQL is an open source relational database system that has a place in most enterprise technology stacks around the world. It is estimated that there are tens of thousands of downloads every day. MySQL database is the first choice for many high-performance database developers, administrators and IT managers because of its high reliability, affordability and ease of use. MySQL has an open and powerful CDC (Change Data Capture) capability. Based on MySQL's Binlog, various replication links can be built. By synchronizing the changed data to the cache system, index system, big data platform and other systems, it can support a variety of business scenarios.

[0003] In the prior art, CDC solutions for distributed databases are presented in the following forms:

[0004] 1. Row-level ordered replication scheme ensures that the replication order of a single row of data is consistent with the upstream change order. Generally, hashing (hash, hash function operation) is performed based on the primary key, and data distribution and subscription are carried out in conjunction with the message middleware;

[0005] 2. Table-level ordered replication scheme ensures that the replication order of all data in a single table is consistent with the upstream change order. Generally, hashing is performed based on the table name TableName, and data distribution and subscription are performed in conjunction with the message middleware;

[0006] 3. Node-level ordered replication scheme, which ensures that the replication order of all data in a single storage node is consistent with the upstream change order, is generally used in distributed MySQL Sharding scenarios;

[0007] 4. A globally ordered replication solution ensures that the replication order of all data in the database is consistent with the execution order of upstream distributed transactions. This generally requires the introduction of a global timestamp.

[0008] Although the three replication schemes mentioned above, namely row-level ordered, table-level ordered, and node-level ordered, are friendly to high concurrency and high throughput, they are very unfriendly to DDL (Data Definition Language, database model definition language), cannot implement DDL replication, and have weak ability to ensure data consistency.

[0009] As for the globally ordered replication solution, most of the current distributed databases still do not have the ability to replicate globally ordered. Most of them use message middleware as the medium for subscription, and the data exchange formats are also different, resulting in weak CDC service capabilities and high access costs. Summary of the invention

[0010] The embodiment of the present application provides a data processing method for a distributed database to solve the problem that the global orderliness and integrity of transactions cannot be guaranteed during the data replication process for the distributed database, and is fully compatible with the MySQL data replication standard.

[0011] Correspondingly, the embodiments of the present application also provide a data processing system of a distributed database, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0012] In order to solve the above problem, an embodiment of the present application discloses a data processing method for a distributed database, wherein the distributed database has multiple storage nodes DN, the DN has a corresponding physical binary log Binlog, the physical binary log Binlog is used to store local transaction operation information XA Event of a distributed transaction XA, the XA includes at least one local transaction Transaction, the local transaction Transaction is composed of the XA Event, the XA has a corresponding global timestamp TSO, and the method may include:

[0013] The global timestamp TSO is used to sort the Transactions in the transQueue, a local transaction list is created through the XA Event and the waitTrans in the transQueue, and a first-level sorting queue is generated according to the local transaction list;

[0014] Perform multi-way merge sorting on the primary sorting queue of each DN to generate a global sorting queue;

[0015] For the global sorting queue, local transactions Transaction with the same global timestamp TSO are merged, and a global binary log Binlog is generated.

[0016] Optionally, the XA may be associated with at least one DN. For the same XA, all local transactions Transaction on the DN associated with the same XA may have the same TSO.

[0017] Optionally, the XA Event includes first-phase commit information XA Prepare, second-phase commit information XA Commit, and rollback information XA Rollback; for a local transaction Transaction, the XA Prepare, the XA Commit, and the XA Rollback have the same transaction identifier xid, and the step of sorting the local transactions Transaction corresponding to each physical binary log Binlog to create a local transaction list may include:

[0018] Read the XA Event in the physical binary log Binlog, and push the read XA Event to a preset sorting item queue sortItemsQueue;

[0019] When the XA Prepare is read, the xid is pushed to the preset waiting transmission queue waitTrans;

[0020] When the XA Commit or the XA Rollback is read, the xid is removed from the waitTrans;

[0021] When the XA Commit is read, the XA Prepare corresponding to the XA Commit is merged through the xid to generate a local transaction Transaction, and the Transaction is pushed to the preset transmission queue transQueue;

[0022] The global timestamp TSO is used to sort the Transactions in the transQueue, a local transaction list is created through the XA Event and the waitTrans in the transQueue, and a first-level sorting queue is generated according to the local transaction list.

[0023] Optionally, the step of creating a local transaction list through the XA Event in the transQueue and the waitTrans may further include:

[0024] Obtain the XA Event from the sortItemsQueue in sequence;

[0025] When the XA Prepare is obtained, the xid is extracted, and it is determined whether the XA Commit with the same xid is received;

[0026] If yes, remove the XA Prepare from the sortItemsQueue, and get the XA Event from the sortItemsQueue again;

[0027] If not, stop obtaining the XA Event from the sortItemsQueue;

[0028] When the XA Commit is obtained, the XA Commit is removed from the sortItemsQueue;

[0029] When the XA Rollback is obtained, the XA Rollback is removed from the sortItemsQueue;

[0030] Determine the maximum timestamp maxTSO in the transQueue, and judge whether the global timestamp TSO corresponding to the XA Commit is greater than the maxTSO;

[0031] If yes, the global timestamp TSO is used as the new maximum timestamp maxTSO;

[0032] The transactions whose TSO in the transQueue is less than or equal to maxTSO are used as safe transactions to create a local transaction list, and the local transaction list is used to generate the first-level sorting queue, wherein the first-level sorting queue is composed of ordered local transactions.

[0033] Optionally, after the step of sequentially obtaining the XA from the sortItemsQueue, the method further includes:

[0034] Determine whether the XA Event obtained from the sortItemsQueue is empty;

[0035] If so, stop obtaining the XA Event from the sortItemsQueue.

[0036] Optionally, after the step of sequentially obtaining the XA from the sortItemsQueue, the method may further include:

[0037] Determine whether the duration of the XA Event obtained from the sortItemsQueue exceeds a preset threshold;

[0038] If so, stop obtaining the XA Event from the sortItemsQueue.

[0039] Optionally, the step of performing multi-path merge sorting on the primary sorting queue of each DN to generate a global sorting queue includes:

[0040] Pushing the transactions in the first-level sorting queue in the local transaction list to the preset global transaction queue;

[0041] The TSO is used to sort the Transactions in the global transaction queue to generate a global sorting queue.

[0042] Optionally, the step of merging local transactions Transaction with the same global timestamp TSO for the global sorting queue and generating a global binary log Binlog may include:

[0043] Reading transactions in the global sorting queue in sequence, and determining whether the global timestamp has changed;

[0044] If yes, a complete global transaction is generated by merging transactions with the same global timestamp TSO;

[0045] Extracting the characteristic event Event of the complete global transaction in the global sorting queue;

[0046] The characteristic event Event is deleted, and the complete global transaction in the global sorting queue is used to generate a global binary log Binlog.

[0047] Optionally, it may also include:

[0048] In the global binary log Binlog, a query log RowsQueryLogEvent is added for the complete global transaction; wherein the RowsQueryLogEvent is used to record the global timestamp TSO corresponding to the complete global transaction.

[0049] Optionally, the global binary log Binlog may have a corresponding logical library table, the logical library table may have a corresponding logical library table name, the physical binary log Binlog may have a corresponding physical library table, the physical library table may have a corresponding physical library table name; the logical library table name may be used to replace the physical library table name when generating the global binary log Binlog; the logical library table and the physical library table may have a mapping relationship, and the mapping relationship may be acquired by the metadata system of the distributed database.

[0050] The embodiment of the present application further discloses a data processing system of a distributed database, wherein the distributed database has multiple storage nodes DN, the DN has a corresponding physical binary log Binlog, the physical binary log Binlog is used to store local transaction operation information XA Event of a distributed transaction XA, the XA includes at least one local transaction Transaction, the local transaction Transaction is composed of the XA Event, the XA has a corresponding global timestamp TSO, and the system may include:

[0051] A primary sorting module, used to sort the transactions in the transQueue using the global timestamp TSO, create a local transaction list through the XA Event and the waitTrans in the transQueue, and generate a primary sorting queue according to the local transaction list;

[0052] The global sorting module is used to perform multi-way merge sorting on the primary sorting queues of each DN to generate a global sorting queue;

[0053] The transaction merging module is used to merge local transactions Transaction with the same global timestamp TSO for the global sorting queue, and generate a global binary log Binlog.

[0054] Optionally, the XA is associated with at least one DN, and for the same XA, all local transactions Transaction on the DN associated with the same XA have the same TSO.

[0055] Optionally, the XA Event may include first-phase commit information XA Prepare, second-phase commit information XA Commit, and rollback information XA Rollback; for a local transaction Transaction, the XA Prepare, the XA Commit, and the XA Rollback may have the same transaction identifier xid, and the primary sorting module may include:

[0056] A local transaction reading submodule, used for reading the XA Event in the physical binary log Binlog, and pushing the read XA Event to a preset sorting item queue sortItemsQueue;

[0057] The identifier push submodule is used to push the xid to the preset waiting transmission queue waitTrans when reading the XA Prepare;

[0058] An identifier removal submodule, used for removing the xid from the waitTrans when reading the XA Commit or the XA Rollback;

[0059] The local transaction push submodule is used for merging the XA Prepare corresponding to the XACommit through the xid when reading the XA Commit, generating a local transaction Transaction, and pushing the Transaction to a preset transmission queue transQueue;

[0060] The local transaction list generation submodule is used to sort the Transactions in the transQueue using the global timestamp TSO, create a local transaction list through the XA Event and the waitTrans in the transQueue, and generate a primary sorting queue according to the local transaction list.

[0061] Optionally, the local transaction list generation submodule may further include:

[0062] A transaction acquisition unit, used for sequentially acquiring the XA Event from the sortItemsQueue;

[0063] an identifier determination unit, configured to extract the xid when the XA Prepare is obtained, and determine whether the XA Commit with the same xid is received; if the XA Commit with the same xid is received, call the first transaction removal unit; if the XA Commit with the same xid is not received, call the first stop acquisition unit;

[0064] A first transaction removal unit, configured to remove the XA Prepare from the sortItemsQueue and obtain the XA Event from the sortItemsQueue again;

[0065] A first stop acquisition unit, used to stop acquiring the XA Event from the sortItemsQueue;

[0066] A second transaction removing unit, configured to remove the XA Commit from the sortItemsQueue when the XA Commit is obtained;

[0067] A third transaction removing unit, configured to remove the XA Rollback from the sortItemsQueue when the XA Rollback is acquired;

[0068] A timestamp determination unit, used to determine the maximum timestamp maxTSO in the transQueue, and determine whether the global timestamp TSO corresponding to the XA Commit is greater than the maxTSO; if the global timestamp TSO corresponding to the XA Commit is greater than the maxTSO, then call a timestamp replacement unit;

[0069] A timestamp replacement unit, configured to adopt the global timestamp TSO as a new maximum timestamp maxTSO;

[0070] The local transaction list creation unit is used to use the transactions in the transQueue whose TSO is less than or equal to maxTSO as safe transactions to create a local transaction list, and use the local transaction list to generate the first-level sorting queue, wherein the first-level sorting queue is composed of ordered local transactions.

[0071] Optionally, the transaction acquisition submodule may further include:

[0072] a transaction empty set judgment unit, used to judge whether the XA Event obtained from the sortItemsQueue is empty; if the XA Event obtained from the sortItemsQueue is empty, calling a second stop acquisition unit;

[0073] The second stop obtaining unit is used to stop obtaining the XA Event from the sortItemsQueue.

[0074] Optionally, the transaction acquisition submodule further includes:

[0075] a duration judgment unit, configured to judge whether the duration of the XA Event obtained from the sortItemsQueue exceeds a preset threshold; if the duration of the XA Event obtained from the sortItemsQueue exceeds the preset threshold, calling a third stop acquisition unit;

[0076] The third stop obtaining unit is used to stop obtaining the XA Event from the sortItemsQueue.

[0077] Optionally, the global sorting module may include:

[0078] A transaction push submodule is used to push the transactions in the first-level sorting queue in the local transaction list to the preset global transaction queue;

[0079] The global sorting submodule is used to sort the transactions in the global transaction queue using the TSO to generate a global sorting queue.

[0080] Optionally, the XA has rotation information Commit TSO Rotation, and the transaction merging module may include:

[0081] A global timestamp change judgment submodule is used to read transactions in the global sorting queue in sequence and judge whether the global timestamp has changed; when the global timestamp has changed, the complete global transaction generation submodule is called;

[0082] The complete global transaction generation submodule is used to generate a complete global transaction by merging transactions with the same global timestamp TSO;

[0083] A characteristic event extraction submodule, used for extracting the characteristic event Event of the complete global transaction in the global sorting queue;

[0084] The characteristic event deletion submodule is used to delete the characteristic event Event and use the complete global transaction in the global sorting queue to generate a global binary log Binlog.

[0085] Optionally, it may also include:

[0086] The query log adding module is used to add a query log RowsQueryLogEvent for a complete global transaction in the global binary log Binlog; wherein the RowsQueryLogEvent is used to record a global timestamp TSO corresponding to the complete global transaction.

[0087] Optionally, the global binary log Binlog may have a corresponding logical library table, the logical library table may have a corresponding logical library table name, the physical binary log Binlog may have a corresponding physical library table, the physical library table may have a corresponding physical library table name; the logical library table name may be used to replace the physical library table name when generating the global binary log Binlog; the logical library table and the physical library table may have a mapping relationship, and the mapping relationship may be acquired by the metadata system of the distributed database.

[0088] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the data processing method of the distributed database as described in one or more of the embodiments of the present application.

[0089] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the data processing method of the distributed database as described in one or more of the embodiments of the present application.

[0090] Compared with the prior art, the embodiments of the present application have the following advantages:

[0091] As one of the most mainstream databases, MySQL has an excellent ecological foundation. Building a distributed database based on the idea of ​​distributed MySQL Sharding (MySQL Sharding, sharded MySQL) is also the current mainstream development direction. So far, there is no technical solution that can guarantee the global orderliness and integrity of transactions when performing data replication for the distributed MySQL Sharding architecture. In the embodiment of the present application, the Transaction in the transQueue is sorted by using the global timestamp TSO, and a local transaction list is created by the XA Event and the waitTrans in the transQueue, and a first-level sorting queue is generated according to the local transaction list; multi-way merge sorting is performed on the first-level sorting queue of each DN to generate a global sorting queue; for the global sorting queue, local transactions Transaction with the same global timestamp TSO are merged, and a global binary log Binlog is generated. Thus, it is ensured that in the data replication process of the distributed database, the transaction has global orderliness and integrity, and at the same time, a strong consistency effect can be achieved, so that data replication can be realized in scenarios with high data consistency requirements. For example, in the transfer scenario, the user can always find the correct result when querying the total balance of the account in the standby database. In addition, the global binary log Binlog is fully compatible with the native MySQL Binlog format in terms of data format and has strong ecological compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a flowchart of steps of an embodiment of a data processing method for a distributed database of the present application;

[0093] Figure 2 It is a flow chart of the generation process of a primary sorting queue of the present application;

[0094] Figure 3 It is a schematic diagram of a first-level sorting queue of the present application;

[0095] Figure 4 This is a flowchart of the global binary log Binlog generation process of this application;

[0096] Figure 5 It is a structural block diagram of a data processing system embodiment of a distributed database of the present application;

[0097] Figure 6 It is a structural diagram of a system provided in one embodiment of the present application. DETAILED DESCRIPTION

[0098] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0099] Reference Figure 1 , is a flowchart of a method for processing data in a distributed database according to an embodiment of the present invention, which may specifically include the following steps:

[0100] Step 102: sort the transactions in the transQueue using the global timestamp TSO, create a local transaction list using the XA Event and the waitTrans in the transQueue, and generate a primary sorting queue according to the local transaction list;

[0101] Step 104: Perform multi-way merge sorting on the primary sorting queue of each DN to generate a global sorting queue;

[0102] Step 106: For the global sorting queue, merge the local transactions Transaction with the same global timestamp TSO, and generate a global binary log Binlog.

[0103] In an embodiment of the present application, the distributed database may have multiple storage nodes DN, the DN may have a corresponding physical binary log Binlog, the physical binary log Binlog may be used to store local transaction operation information XA Event of the distributed transaction XA, the XA includes at least one local transaction Transaction, the local transaction Transaction may be composed of the XA Event, and the XA may have a corresponding global timestamp TSO.

[0104] In order to enable those skilled in the art to better understand the present application, the following is a brief description of the terms involved in the present application:

[0105] DN (storage node, Data Node)

[0106] MySQL InnoDB (database engine)

[0107] TSO (Global Timestamp, Timestamp Oracle)

[0108] XA (eXtended Architecture, distributed transactions)

[0109] XA protocol (a distributed transaction processing specification)

[0110] Binary log Binlog: used to record DDL and DML (data manipulation language) statements in the form of events, including the time consumed by the statement execution.

[0111] 2PC (two-phase commit): 2PC is a strongly consistent, centralized atomic commit protocol that can guarantee atomicity and durability. There are multiple components involved in the transaction, and each component records its own operation log. The operation log is separate rather than centralized.

[0112] MVCC (Multi-version Cocurrent Control): It is used to perform multi-version control on data during concurrent access to the database to avoid blockage of read operations due to write locks, thereby optimizing the concurrency blockage problem.

[0113] while loop: It is a basic loop mode of computer. It enters the loop when the condition is met and exits the loop when the condition is not met.

[0114] Logical library table: used in distributed MySQL Sharding scenarios, with multiple Sharding partitions, that is, the library table that is directly visible and used externally.

[0115] Physical library table: used in distributed MySQL Sharding scenarios, the Sharding library table on each MySQL node belongs to a certain logical library table.

[0116] Physical Binlog: used for Binlog on each MySQL node in distributed MySQL Sharding scenarios.

[0117] In a specific implementation, the embodiment of the present application can implement the MVCC mechanism in the form of XA protocol + 2PC + TSO based on MySQL InnoDB before the distributed database copies data, and can extend a sequence event Sequence before the second-phase commit event XA Commit Event of the native MySQL local transaction to record the second-phase commit event timestamp Commit TSO of the transaction, and write the recorded Commit TSO into the physical Binlog.

[0118] In practical applications, the distributed database of the embodiment of the present application may include multiple storage node DNs, the storage node DN may be MySQL, and each DN may have a corresponding physical binary log Binlog, the physical binary log Binlog may be used to store local transaction operation information XA Event of the distributed transaction XA, XA includes at least one local transaction Transaction, the local transaction Transaction may be composed of XA Event, and XA may have a corresponding global timestamp TSO.

[0119] In a specific implementation, the embodiment of the present application can sort the local transactions Transaction recorded in each physical binary log Binlog, so as to obtain a local transaction list corresponding to each storage node DN. After the local transaction list is created, a first-level sorting queue corresponding to the local transaction list can be generated for the local transaction list. This means that a DN corresponds to a group of first-level sorting queues, and at the same time, this first-level sorting queue can also be regarded as a partially ordered set.

[0120] In practical applications, a distributed database can have multiple DNs, each of which can have its own physical local transaction list. After sorting the first-level sorting queue of each DN, each DN can obtain a locally ordered first-level sorting queue. The local transaction list can be used to generate the first-level sorting queue. At this time, the first-level sorting queues corresponding to the multiple local transaction lists can be merged together, and then sorted by the global timestamp TSO to generate a global sorting queue. The global sorting queue can be composed of globally ordered local transactions Transaction. At the same time, the global sorting queue can be regarded as a fully ordered set.

[0121] In a preferred embodiment of the present application, the XA is associated with at least one DN, and for the same XA, all local transactions Transaction on the DN associated with the same XA have the same TSO.

[0122] In a preferred embodiment of the present application, the XA is associated with at least one DN. For the same XA, all local transactions Transaction on the DN associated with the same XA have the same TSO.

[0123] In actual applications, for an XA, there will be at least one DN participating in it. For the same XA, the Commit TSO on all DN nodes participating in the XA is the same.

[0124] From the above, it can be seen that the embodiment of the present application can merge local transactions Transaction with the same global timestamp TSO in the global sorting queue, and then generate a global binary log Binlog, which can record local transactions Transaction in multiple DNs, and the local transactions Transaction recorded in the global binary log Binlog are globally ordered.

[0125] The embodiment of the present application uses the global timestamp TSO to sort the transactions in the transQueue, creates a local transaction list through the XA Event and the waitTrans in the transQueue, and generates a first-level sorting queue based on the local transaction list; performs multi-way merge sorting on the first-level sorting queue of each DN to generate a global sorting queue; for the global sorting queue, merges the local transaction transactions with the same global timestamp TSO, and generates a global binary log Binlog. This ensures that in the data replication process of the distributed database, the transactions have global orderliness and integrity. At the same time, a strong consistency effect can also be achieved, so that data replication can be achieved in scenarios with high data consistency requirements. For example, in the transfer scenario, the user can always find the correct result when querying the total balance of the account in the backup database. In addition, the global binary log Binlog is fully compatible with the native MySQL Binlog format in data format and has strong ecological compatibility.

[0126] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0127] In a preferred embodiment of the present application, step 102, the step of sorting the local transactions Transaction corresponding to each physical binary log Binlog to create a local transaction list includes:

[0128] Read the XA Event in the physical binary log Binlog, and push the read XA Event to a preset sorting item queue sortItemsQueue;

[0129] When the XA Prepare is read, the xid is pushed to the preset waiting transmission queue waitTrans;

[0130] When the XA Commit or the XA Rollback is read, the xid is removed from the waitTrans;

[0131] When the XA Commit is read, the XA Prepare corresponding to the XA Commit is merged through the xid to generate a local transaction Transaction, and the Transaction is pushed to the preset transmission queue transQueue;

[0132] The global timestamp TSO is used to sort the Transactions in the transQueue, a local transaction list is created through the XA Event and the waitTrans in the transQueue, and a first-level sorting queue is generated according to the local transaction list.

[0133] In an example of an embodiment of the present application, the XA Event includes first-phase commit information XA Prepare, second-phase commit information XA Commit, and rollback information XA Rollback; for a local transaction Transaction, the XA Prepare, the XA Commit, and the XA Rollback have the same transaction identifier xid.

[0134] In practical applications, the XA of the embodiment of the present application may include first-phase commit information XA Prepare, second-phase commit information XA Commit, rollback information XA Rollback and other transaction information. For a local transaction Transaction, the XA Prepare, the XA Commit, and the XA Rollback have the same transaction identifier xid. Therefore, for the same local transaction Transaction, regardless of whether its XA Prepare, XA Commit, and XARollback are stored in the same DN, its global timestamp TSO is the same.

[0135] Specifically, the basis of XA transactions can be the two-phase commit protocol 2PC, in which a transaction coordinator is required to ensure that all transaction participants have completed preparation, that is, the first-phase commit message XA Prepare. If the coordinator receives a message that all participants are ready, it will notify all transactions that they can be committed, that is, the second-phase commit message XA Commit.

[0136] XA transactions can be divided into internal XA and external XA. External XA can participate in external distributed transactions and requires the application layer to intervene as a coordinator. Internal XA transactions can be used across multiple MySQL InnoDBs under the same instance and are coordinators by Binlog. For example, when a MySQL InnoDB commits, the commit information needs to be written to the binary log Binlog. This is a distributed internal XA transaction, and the participant of the binary log Binlog can be MySQL itself.

[0137] In order to enable those skilled in the art to better understand the embodiments of the present application, a specific example is used below to explain the distributed transaction XA.

[0138] In MySQL, XA transactions can include:

[0139] XA{START|BEGIN}xid[JOIN|RESUME] - Start an XA transaction [xid must be a unique value; (JOIN|RESUME) clause is not supported]

[0140] XA END xid[SUSPEND(FOR MIGRATE)] - ends an XA transaction (the [SUSPEND[FORMIGRATE]] clause is not supported)

[0141] XA PREPARE xid -- prepare

[0142] XA COMMIT xid[ONE PHASE]——Commit XA transaction

[0143] XA ROLLBACK xid——Rollback an XA transaction

[0144] XA RECOVER - View all XA transactions in the PREPARE phase

[0145] The transaction identifier xid is:

[0146] xid is a transaction identifier that is provided by the client or generated by the MySQL server.

[0147] The format of xid is generally xid:gtrid[,bqual[,formatID]]; gtrid is a global transaction identifier, bqual is a branch qualifier, and formatID is a number that identifies the format used by the gtrid and bqual values. bqual and formatID are optional, as indicated by the syntax. If not given, the default bqual value is ". If not given, the default fromatID value is 1.

[0148] The XA transaction status progress process is:

[0149] 1. Use XA START to start an XA transaction and set it to the ACTIVE state.

[0150] 2. For an ACTIVE XA transaction, issue the SQL statements that constitute the transaction, and then issue an XA END statement. XAEND sets the transaction to the IDLE state.

[0151] 3. For an IDLE XA transaction, issue an XA PREPARE statement or a XA COMMIT ... ONEPHASE statement: the former sets the transaction to the PREPARE state, and the output of the XA RECOVER statement contains the transaction xid value (the XARECOVER statement will list all XA transactions in the PREPARE state); the latter is used to prepare and commit transactions and will not be listed by XARECOVER because the transaction has been terminated.

[0152] 4. For a PREPARE XA transaction, you can issue an XA COMMIT statement to commit and terminate the transaction, or issue an XA ROLLBACK to roll back and terminate the transaction.

[0153] In actual applications, the global timestamp TSO is out of order in the physical Binlog file. In order to achieve global order in MySQL, the first-level sorting of DN can be completed first. The embodiment of the present application can first push the xid to the preset waiting transmission queue waitTrans when reading XA Prepare, and remove the xid from waitTrans when reading XA Commit or XA Rollback. In the subsequent first-level sorting process, it is known whether the corresponding XA Commit is received when the XA Prepare is received. When the XA Commit is read, the Transaction corresponding to the XA Commit is pushed to the preset transmission queue transQueue, and the Transaction is sorted in the ransQueue by the obtained XA Prepare, XA Commit, XARollback and the global timestamp TSO corresponding to the XA Commit, so as to obtain a first-level sorting queue for the Transaction in DN.

[0154] In a more preferred embodiment of the present application, the step of creating a local transaction list through the XA Event and the waitTrans in the transQueue further includes:

[0155] Obtain the XA Event from the sortItemsQueue in sequence;

[0156] When the XA Prepare is obtained, the xid is extracted, and it is determined whether the XA Commit with the same xid is received;

[0157] If yes, remove the XA Prepare from the sortItemsQueue, and get the XA Event from the sortItemsQueue again;

[0158] If not, stop obtaining the XA Event from the sortItemsQueue;

[0159] When the XA Commit is obtained, the XA Commit is removed from the sortItemsQueue;

[0160] When the XA Rollback is obtained, the XA Rollback is removed from the sortItemsQueue;

[0161] Determine the maximum timestamp maxTSO in the transQueue, and judge whether the global timestamp TSO corresponding to the XA Commit is greater than the maxTSO;

[0162] If yes, the global timestamp TSO is used as the new maximum timestamp maxTSO;

[0163] The transactions whose TSO in the transQueue is less than or equal to maxTSO are used as safe transactions to create a local transaction list, and the local transaction list is used to generate the first-level sorting queue, wherein the first-level sorting queue is composed of ordered local transactions.

[0164] In another more preferred embodiment of the present application, after the step of sequentially obtaining the XA from the sortItemsQueue, the following may also be included:

[0165] It can be determined whether the XA Event obtained from the sortItemsQueue is empty;

[0166] If so, stop obtaining the XA Event from the sortItemsQueue.

[0167] In another more preferred embodiment of the present application, after the step of sequentially obtaining the XA from the sortItemsQueue, the following may further be included:

[0168] It can be determined whether the duration of the XA Event obtained from the sortItemsQueue exceeds a preset threshold;

[0169] If so, stop obtaining the XA Event from the sortItemsQueue.

[0170] In another more preferred embodiment of the present application, before the step of using a Transaction less than or equal to the maxTSO to create a physical binary log Binlog, the following steps may also be included:

[0171] It can be determined whether the Transaction is empty;

[0172] If so, the XA Event in the physical binary log Binlog may be re-read, and the read XA Event may be pushed to a preset sorting item queue sortItemsQueue.

[0173] Specifically, for an XA (XA Prepare + XA Commit), if there are XA Prepare or XA Commit of other transactions between its XA Prepare and XA Commit, there is a transaction hole between these transactions. Transaction holes are specifically manifested in Binlog as the commit logs of other local transactions between the XA Prepare and XA Commit of a local transaction.

[0174] For example, suppose there are transactions XA1 and XA2, and XA Prepare is abbreviated as P and XA Commit is abbreviated as C. If there are holes, they are represented in the Binlog as: P1 P2 C1 C2, P1 P2 C2 C1, P2 P1 C1 C2, P2 P1 C2 C1; if there are no holes, they are represented in the Binlog as: P1 C1 P2 C2, P2 C2 P1 C1.

[0175] Combining the above XA transaction status progress process, three theorems can be derived, including:

[0176] Theorem 1

[0177] Definition: If there is a hole between two transactions, there is no happenbefore constraint between the two transactions, that is, there must be no write conflict between the two local transactions, and adjusting the before and after relationship of the transactions will not cause data consistency problems.

[0178] Proof by contradiction: Because concurrent transactions with write conflicts will be converted into serial relationships by exclusive locks; therefore, transactions with write conflicts cannot enter the XA Prepare stage at the same time; therefore, there is no condition to trigger the void; proof obtained.

[0179] Theorem 2

[0180] Definition: If there is a hole between two transactions, the Commit TSO of the two transactions is not related to the Commit order, that is, the Commit TSO of the transaction that commits earlier is not necessarily larger than the Commit TSO of the transaction that commits later. For example: In the P1 P2 C1 C2 scenario, the Commit TSO of C1 may be larger or smaller than that of C2.

[0181] Proof: From Theorem 1, we know that there must be no write conflict between transactions with a void relationship, so the commit of the transaction is random. Since obtaining the commit TSO and committing are not atomic operations, the transaction that initiates the commit first may hold a smaller commit TSO. This is proved.

[0182] Theorem 3

[0183] Definition: If there is no gap between two transactions, the Commit TSO of the two transactions is positively correlated with the Commit order, that is, the Commit TSO of the transaction that commits earlier must be less than or equal to the Commit TSO of the transaction that commits later. For example: In the P1 C1 P2 C2 scenario, the Commit TSO of C1 must be less than or equal to the Commit TSO of C2.

[0184] Proof: When there is no hole between transactions T1 and T2, that is, when the form P1 C1 P2 C2 is satisfied, combined with the characteristics of the above 2PC transactions, we can know that:

[0185] a. For transaction T1, when C1 is committed, P1 of all shards must have been committed.

[0186] b. For transaction T2, when C2 is committed, P2 of all shards must have been committed.

[0187] Therefore, from the order of occurrence, because C1=>P2, P2=>C2, so C1=>C2 ("=>" means "happens before"), that is: C1 must occur before C2, C1 and C2 will not occur concurrently, which is proved.

[0188] Conclusion: From Theorem 1 and Theorem 2, we can see that the out-of-order scenario of Commit TSO does exist, but sorting will not cause data consistency problems. From Theorem 3, we can see that out-of-order only occurs in the scenario where there are transaction holes.

[0189] Figure 2 A flow chart of the generation process of a first-level sorting queue of the present application is shown, such as Figure 2 As shown, in a specific implementation, the embodiment of the present application can generate a first-level sorting queue in the following manner.

[0190] Step 201, read the XA Event in the physical binary log Binlog, step 202, push the read XAEvent to the preset sorting item queue sortItemsQueue, and in the reading process, execute step 203 to determine the transaction type of the read XA Event; when XA Prepare is read, step 204 can be executed to push xid to the preset waiting transmission queue waitTrans, when XA Commit or XA Rollback is read, step 205 can be executed to remove xid from waitTrans; execute step 206 to determine the transaction type of the read XA Event, when XA Commit is read, step 207 can be executed to push the Transaction corresponding to XA Commit to the preset transmission queue transQueue; after completing the reading of the XA Event in sortItemsQueue, step 208 can be executed to obtain XA Events from sortItemsQueue in sequence. Event, and in the process of obtaining XAEvent from sortItemsQueue, step 209 can be executed to determine the transaction type of the obtained XA Event. When XAPrepare is obtained, xid can be extracted, and step 210 can be executed to find out whether there is the same xid from waitTrans. If not, step 211 can be executed to remove the XA Prepare from sortItemsQueue, and step 208 can be executed again. If so, the loop can be exited to stop obtaining XA Event from sortItemsQueue. When XACommit is obtained, step 211 can be executed to remove XA Commit from sortItemsQueue; when XARollback is obtained, step 211 can be executed to remove XA Rollback from sortItemsQueue;Execute step 212 to determine the transaction type of the XA Event removed from the sortItemsQueue. When the removed XA is XA Commit, execute step 213 to determine the maximum timestamp maxTSO in the transQueue, and determine whether the global timestamp TSO corresponding to the XA Commit is greater than maxTSO. If the global timestamp TSO corresponding to the XA Commit is less than or equal to maxTSO, execute 214 to determine whether the duration of the obtained XA is within the preset duration. If the duration of the obtained XA exceeds the preset duration, exit the loop or execute step 208. If the duration of the obtained XA Event is within the preset duration, execute step 215 to obtain the Transaction corresponding to the C-TSO less than or equal to maxTSO and push it to the downstream transaction list. If XA If the global timestamp TSO corresponding to Commit is greater than maxTSO, the global timestamp TSO can be used as the new maximum timestamp maxTSO. After completing step 215, step 216 can be executed to determine whether Transaction is empty. If Transaction is empty, the loop can be exited, or step 201 can be executed. If Transaction is not empty, step 217 can be executed to use the Transaction in the transQueue whose TSO is less than or equal to maxTSO as the safe transaction to create a local transaction list, and use the local transaction list to generate the first-level sorting queue, wherein the first-level sorting queue is composed of ordered local transactions Transaction. ;

[0191] refer to Figure 3 The schematic diagram of a first-level sorting queue of the present application shown can push the read XA Event to the preset sorting item queue sortItemsQueue. At this time, the XA Event 301 in the sortItemsQueue is unordered, and the Transaction 302 in the transQueue can include the corresponding XA Prepare, XACommit and global timestamp TSO, and the Transaction in the transQueue can be an ordered sequence sorted according to the global timestamp TSO.

[0192] In order to enable those skilled in the art to better understand the embodiments of the present application, a complete example is provided below to describe how the embodiments of the present application sort the local transactions of each storage node DN to generate a primary sorting queue.

[0193] T stands for local transaction Transaction; P stands for XA Prepare; C stands for XA Commit; R stands for XA Rollback; TSO stands for global timestamp TSO.

[0194] Step 1. Pull and consume transaction logs in the natural order of the Binlog file.

[0195] Step 2. Every time a P, C, or R is received, it will be put into the sortItemsQueue.

[0196] Step 3. Every time a P is received, extract the xid it holds and put the xid into waitTrans.

[0197] Step 4. Each time an R is received, extract the xid it holds and remove the xid from waitTrans

[0198] Step 5. Every time a C is received, extract the xid it holds and remove the xid from waitTrans

[0199] Step 6. Every time a C is received, extract the T it holds and put T into the transQueue.

[0200] Step 7. After receiving each C, complete the operations in steps 2, 5, and 6 above, perform the following operations to obtain a list of transactions that can be exported:

[0201] a. Start the while loop and set the loop timeout

[0202] i gets the element from the queue head of sortItemsQueue, recorded as Item, if Item is empty, exit the loop, if not empty, continue

[0203] ii. If the item is a P, extract the xid it holds, and then use waitTrans to determine whether the corresponding C has been received. If it has been received, remove the item from sortItemsQueue and jump to step i to continue the next round; if it has not been received, exit the while loop directly

[0204] iii. If the item is a C or R, remove the item from the sortItemsQueue, and then compare C's Commit TSO with the current maxTSO. If C's TSO is greater than maxTSO, replace maxTSO with C's TSO. Check whether it has timed out. If not, jump to step i to continue the next loop; if it has timed out, exit the while loop.

[0205] b. Get all transactions less than or equal to maxTSO from transQueue, construct a local transaction list, and output it to the downstream if the list is not empty (these transactions can be safely output, which can be proved by Theorem 3).

[0206] Step 8. Return to step 1 to continue processing.

[0207] Of course, those skilled in the art may adopt other algorithms to perform primary sorting according to actual conditions, and this application does not need to impose any limitation on this.

[0208] In a preferred embodiment of the present application, step 104, performing multi-way merge sorting on the primary sorting queue of each DN to generate a global sorting queue may include the following steps:

[0209] Pushing the transactions in the first-level sorting queue in the local transaction list to the preset global transaction queue;

[0210] The TSO is used to sort the Transactions in the global transaction queue to generate a global sorting queue.

[0211] In a specific implementation, the embodiment of the present application can deliver the Transaction to the downstream after constructing the local transaction list, that is, the first-level sorting queue corresponding to each physical Binlog contains at least one local transaction Transaction. The embodiment of the present application can push the Transaction in the physical binary log Binlog corresponding to each storage node DN to the preset global transaction queue, and then take the transaction with the smallest global timestamp TSO and output it to the downstream of the global transaction queue, and repeat this cycle to obtain an unbounded global sorting queue.

[0212] Of course, those skilled in the art may adopt other algorithms for global sorting according to actual conditions, and this application does not need to impose any limitation on this.

[0213] In a preferred embodiment of the present application, step 106, merging local transactions Transaction with the same global timestamp TSO for the global sorting queue and generating a global binary log Binlog may include the following steps:

[0214] Reading transactions in the global sorting queue in sequence, and determining whether the global timestamp has changed;

[0215] If yes, a complete global transaction is generated by merging transactions with the same global timestamp TSO;

[0216] Extracting the characteristic event Event of the complete global transaction in the global sorting queue;

[0217] The characteristic event Event is deleted, and the complete global transaction in the global sorting queue is used to generate a global binary log Binlog.

[0218] In the specific implementation, local transaction merging can be completed based on the global sorting queue. Whenever Commit TSO Rotation occurs, it means that the global timestamp has changed, that is, it means the end of the previous XA and the beginning of the new XA. Different XAs are distinguished based on this. As can be seen from the above, for the same Transaction, all DNs associated with the same Transaction have the same TSO, so local transactions scattered across various DNs can be merged into a complete transaction. In addition, in the process of merging, characteristic events corresponding to XA characteristics such as XA Start, XA End, and XA Prepare can be eliminated, and only single-machine characteristic events are retained.

[0219] The embodiment of the present application can eliminate the complexity of the global binary log Binlog by deleting the characteristic events of the transaction before generating the global binary log Binlog.

[0220] In a preferred embodiment of the present application, a query log RowsQueryLogEvent may be added for a complete global transaction in the global binary log Binlog; wherein the RowsQueryLogEvent is used to record a global timestamp TSO corresponding to the complete global transaction.

[0221] In actual applications, the global binary log Binlog generated by the global sorting queue can load an event of type RowsQueryLogEvent after each XA compared to the native MySQL Binlog. RowsQueryLogEvent can be used to record the global timestamp TSO of each transaction to support fault recovery. When the link that produces the global binary log Binlog is interrupted or restarted, the link can be restored based on the last global timestamp TSO recorded in the file. The appended RowsQueryLogEvent can be dumped according to the provisions of the MySQL Dump protocol, and the event will be ignored by the downstream system. The downstream system cannot perceive the internal details of the distributed system and will not affect subscription consumption.

[0222] Similarly, the embodiment of the present application can also refer to MySQL to implement a dump program, that is, to provide external consumption subscription capabilities. The downstream system can consume the global binary log Binlog of the distributed database just like subscribing to a single-machine MySQL Binlog.

[0223] The embodiment of the present application can add RowsQueryLogEvent that complies with the database dump MySQL Dump protocol to XA in the global binary log Binlog, so that the global binary log Binlog is compatible with both the MySQL Binlog file format and the Dump protocol, thereby preventing the downstream system from perceiving the internal details of the distributed system and not affecting subscription consumption.

[0224] In a preferred embodiment of the present application, the global binary log Binlog may have a corresponding logical library table, the logical library table may have a corresponding logical library table name, the physical binary log Binlog may have a corresponding physical library table, the physical library table may have a corresponding physical library table name; the logical library table name may be used to replace the physical library table name when generating the global binary log Binlog; the logical library table and the physical library table may have a mapping relationship, and the mapping relationship may be acquired by the metadata system of the distributed database.

[0225] In the specific implementation, after the transaction is merged, the physical library table name in the physical Binlog can be replaced with the name of the logical library table to make the global binary log Binlog directly visible and usable to the outside world. The mapping relationship between the physical library table and the logical library table can be obtained through the metadata system of the distributed database.

[0226] like Figure 4 As shown, Figure 4This is a flowchart of the global binary log Binlog generation process of the present application. The core of the embodiment of the present application lies in sorting, and the basis of sorting is the global timestamp TSO. In practical applications, the embodiment of the present application can be used in the core component of the CDC of a distributed database. For an XA, at least one DN node will participate in it. For the same Transaction, its global timestamp TSO on all DN nodes participating in the transaction can be the same. Determined by the characteristics of XA, the global timestamp TSO is not naturally ordered in the physical Binlog of each DN node, so a first-level sorting queue can be generated for each DN first, and on the basis of generating the first-level sorting queue, the local transactions in the local transaction list of all DN nodes are multi-way merged, and a global sorting queue can be generated by global sorting. On the basis of global order, XAs holding the same global timestamp TSO can be merged into a complete transaction, and a global binary log Binlog can be generated and output in the form of a MySQL stand-alone transaction.

[0227] The embodiment of the present application can sort the Transaction in the transQueue by using the global timestamp TSO, create a local transaction list through the XA Event and the waitTrans in the transQueue, and generate a first-level sorting queue according to the local transaction list; perform multi-way merge sorting on the first-level sorting queue of each DN to generate a global sorting queue; for the global sorting queue, merge the local transaction Transaction with the same global timestamp TSO, and generate a global binary log Binlog. This ensures that in the data replication process of the distributed database, the transaction has global orderliness and integrity. At the same time, a strong consistency effect can also be achieved, so that data replication can be achieved in scenarios with high data consistency requirements. For example, in the transfer scenario, the user can always find the correct result when querying the total balance of the account in the backup database. In addition, the global binary log Binlog is fully compatible with the native MySQL Binlog format in data format and has strong ecological compatibility.

[0228] Furthermore, the embodiment of the present application adopts the TSO strategy and implements distributed MVCC based on the global timestamp (TSO), which can enable the distributed database to have correct linear consistency and good performance; at the same time, the distributed local transactions are converted into the global binary log Binlog in the single-machine transaction log format to meet the strong data consistency requirements; and before generating the global binary log Binlog, the characteristic events of the transaction are deleted, thereby eliminating the complexity of the global binary log Binlog; in addition, by adding the RowsQueryLogEvent that complies with the database dump MySQL Dump protocol to the XA in the global binary log Binlog, the global binary log Binlog is compatible with the MySQL Binlog file format and the Dump protocol, so that the downstream system cannot perceive the internal details of the distributed system and will not affect subscription consumption.

[0229] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0230] On the basis of the above embodiments, this embodiment further provides a data processing system of a distributed database, which is applied to electronic devices such as terminal devices and servers.

[0231] Reference Figure 5 , shows a structural block diagram of a data processing system embodiment of a distributed database of the present application, which may specifically include the following modules:

[0232] The primary sorting module 501 is used to sort the transactions in the transQueue using the global timestamp TSO, create a local transaction list through the XA Event and the waitTrans in the transQueue, and generate a primary sorting queue according to the local transaction list;

[0233] The global sorting module 502 is used to perform multi-way merge sorting on the primary sorting queue of each DN to generate a global sorting queue;

[0234] The transaction merging module 503 is used to merge local transactions Transaction with the same global timestamp TSO for the global sorting queue, and generate a global binary log Binlog.

[0235] Optionally, the XA is associated with at least one DN, and for the same XA, all local transactions Transaction on the DN associated with the same XA have the same TSO.

[0236] Optionally, the XA Event may include first-phase commit information XA Prepare, second-phase commit information XA Commit, and rollback information XA Rollback; for a local transaction Transaction, the XA Prepare, the XA Commit, and the XA Rollback have the same transaction identifier xid, and the primary sorting module 501 may include:

[0237] The local transaction reading submodule can be used to read the XAEvent in the physical binary log Binlog, and push the read XA Event to a preset sorting item queue sortItemsQueue;

[0238] The identifier push submodule may be used to push the xid to a preset waiting transmission queue waitTrans when reading the XA Prepare;

[0239] The identifier removal submodule may be used to remove the xid from the waitTrans when the XA Commit or the XA Rollback is read;

[0240] The local transaction push submodule may be used to merge the XA Prepare corresponding to the XA Commit through the xid when reading the XA Commit, generate a local transaction Transaction, and push the Transaction to a preset transmission queue transQueue;

[0241] The local transaction list generation submodule can be used to sort the Transactions in the transQueue using the global timestamp TSO, create a local transaction list through the XA Event and the waitTrans in the transQueue, and generate a primary sorting queue according to the local transaction list.

[0242] Optionally, the local transaction list generation submodule may further include:

[0243] A transaction acquisition unit, which can be used to sequentially acquire the XA Event from the sortItemsQueue;

[0244] an identifier determination unit, which may be used to extract the xid when the XA Prepare is obtained, and determine whether the XA Commit with the same xid is received; if the XA Commit with the same xid is received, call the first transaction removal unit; if the XA Commit with the same xid is not received, call the first stop acquisition unit;

[0245] A first transaction removal unit may be used to remove the XA Prepare from the sortItemsQueue and obtain the XA Event from the sortItemsQueue again;

[0246] A first stop acquisition unit may be used to stop acquiring the XA Event from the sortItemsQueue;

[0247] A second transaction removal unit may be configured to remove the XA Commit from the sortItemsQueue when the XA Commit is obtained;

[0248] A third transaction removal unit may be configured to remove the XA Rollback from the sortItemsQueue when the XA Rollback is acquired;

[0249] A timestamp determination unit may be used to determine the maximum timestamp maxTSO in the transQueue, and determine whether the global timestamp TSO corresponding to the XA Commit is greater than the maxTSO; if the global timestamp TSO corresponding to the XA Commit is greater than the maxTSO, then a timestamp replacement unit is called;

[0250] A timestamp replacement unit, which can be used to adopt the global timestamp TSO as a new maximum timestamp maxTSO;

[0251] The local transaction list creation unit can be used to use the transactions in the transQueue whose TSO is less than or equal to maxTSO as safe transactions to create a local transaction list, and use the local transaction list to generate the first-level sorting queue, wherein the first-level sorting queue is composed of ordered local transactions.

[0252] Optionally, the transaction acquisition submodule may further include:

[0253] The transaction empty set judgment unit may be used to judge whether the XAEvent obtained from the sortItemsQueue is empty; if the XA Event obtained from the sortItemsQueue is empty, the second stop acquisition unit is called;

[0254] The second stop acquisition unit may be used to stop acquiring the XA Event from the sortItemsQueue.

[0255] Optionally, the transaction acquisition submodule may further include:

[0256] The duration judgment unit may be used to judge whether the duration of the XA Event obtained from the sortItemsQueue exceeds a preset threshold; if the duration of the XA Event obtained from the sortItemsQueue exceeds the preset threshold, the third stop acquisition unit is called;

[0257] The third stop obtaining unit may be used to stop obtaining the XA Event from the sortItemsQueue.

[0258] Optionally, the global sorting module 502 may include:

[0259] The transaction push submodule can be used to push the transactions in the first-level sorting queue in the local transaction list to the preset global transaction queue;

[0260] The global sorting submodule can be used to sort the transactions in the global transaction queue using the TSO to generate a global sorting queue.

[0261] Optionally, the XA may have rotation information Commit TSO Rotation, and the transaction merging module 503 may include:

[0262] A global timestamp change judgment submodule is used to read transactions in the global sorting queue in sequence and judge whether the global timestamp has changed; when the global timestamp has changed, the complete global transaction generation submodule is called;

[0263] The complete global transaction generation submodule is used to generate a complete global transaction by merging transactions with the same global timestamp TSO;

[0264] A characteristic event extraction submodule, used for extracting the characteristic event Event of the complete global transaction in the global sorting queue;

[0265] The characteristic event deletion submodule is used to delete the characteristic event Event and use the complete global transaction in the global sorting queue to generate a global binary log Binlog.

[0266] Optionally, it may also include:

[0267] The query log adding module can be used to add a query log RowsQueryLogEvent for a complete global transaction in the global binary log Binlog; wherein the RowsQueryLogEvent can be used to record a global timestamp TSO corresponding to the complete global transaction.

[0268] Optionally, the global binary log Binlog may have a corresponding logical library table, the logical library table may have a corresponding logical library table name, the physical binary log Binlog may have a corresponding physical library table, the physical library table may have a corresponding physical library table name; the logical library table name may be used to replace the physical library table name when generating the global binary log Binlog; the logical library table and the physical library table may have a mapping relationship, and the mapping relationship may be acquired by the metadata system of the distributed database.

[0269] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0270] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0271] The embodiments of the present disclosure may be implemented as a system configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the system may include electronic devices such as terminal devices and servers (clusters). Figure 6 An exemplary system 600 that can be used to implement various embodiments described in this application is schematically illustrated.

[0272] For one embodiment, Figure 6An exemplary system 600 is shown having one or more processors 602, a control module (chip set) 604 coupled to at least one of the processor(s) 602, a memory 606 coupled to the control module 604, a non-volatile memory (NVM) / storage device 608 coupled to the control module 604, one or more input / output devices 610 coupled to the control module 604, and a network interface 612 coupled to the control module 604.

[0273] The processor 602 may include one or more single-core or multi-core processors, and the processor 602 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the system 600 can be used as a terminal device, server (cluster), and other devices described in the embodiments of the present application.

[0274] In some embodiments, the system 600 may include one or more computer-readable media (e.g., memory 606 or NVM / storage device 608) having instructions 614 and one or more processors 602 configured to execute the instructions 614 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0275] For one embodiment, control module 604 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 602 and / or any suitable device or component in communication with control module 604 .

[0276] The control module 604 may include a memory controller module to provide an interface to the memory 606. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0277] The memory 606 may be used, for example, to load and store data and / or instructions 614 for the system 600. For one embodiment, the memory 606 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 606 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0278] For one embodiment, control module 604 may include one or more input / output controllers to provide an interface to NVM / storage device 608 and input / output device(s) 610 .

[0279] For example, NVM / storage 608 may be used to store data and / or instructions 614. NVM / storage 608 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0280] NVM / storage device 608 may include storage resources that are physically part of the device on which system 600 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 608 may be accessed via input / output device(s) 610 over a network.

[0281] The input / output device(s) 610 may provide an interface for the system 600 to communicate with any other appropriate device, and the input / output device 610 may include a communication component, an audio component, a sensor component, etc. The network interface 612 may provide an interface for the system 600 to communicate through one or more networks, and the system 600 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0282] For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 604. For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers of the control module 604 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604. For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604 to form a system-on-chip (SoC).

[0283] In various embodiments, the system 600 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the system 600 may have more or fewer components and / or different architectures. For example, in some embodiments, the system 600 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0284] Among them, the main control chip can be used as a processor or control module in the detection system, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0285] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0286] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0287] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable distributed database data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable distributed database data processing terminal device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0288] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable distributed database data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0289] These computer program instructions can also be loaded onto a computer or other programmable distributed database data processing terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0290] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0291] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0292] The above is a detailed introduction to a data processing method and system for a distributed database, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method for a distributed database, characterized in that: The distributed database has multiple storage nodes DN, the DN has a corresponding physical binary log Binlog, the physical binary log Binlog is used to store local transaction operation information XA Event of the distributed transaction XA, the XA includes at least one local transaction Transaction, the local transaction Transaction is composed of the XA Event, the XA has a corresponding global timestamp TSO, and the method includes: The global timestamp TSO is used to sort the Transactions in the transQueue, a local transaction list is created through the XAEvent and waitTrans in the transQueue, and a first-level sorting queue is generated according to the local transaction list; wherein the transQueue is a preset transmission queue, and the waitTrans is a preset waiting transmission queue; Perform multi-way merge sorting on the primary sorting queue of each DN to generate a global sorting queue; For the global sorting queue, local transactions Transaction with the same global timestamp TSO are merged, and a global binary log Binlog is generated.

2. The method according to claim 1, characterized in that: The XA is associated with at least one DN. For the same XA, all local transactions Transaction on the DN associated with the same XA have the same TSO.

3. The method according to claim 1, characterized in that The XA Event includes first-phase commit information XAPrepare, second-phase commit information XA Commit, and rollback information XA Rollback; for a local transaction Transaction, the XA Prepare, the XA Commit, and the XA Rollback have the same transaction identifier xid, and the steps of using the global timestamp TSO to sort the Transaction in the transQueue and creating a local transaction list through the XA Event and waitTrans in the transQueue include: Read the XA Event in the physical binary log Binlog, and push the read XA Event to a preset sorting item queue sortItemsQueue; When the XA Prepare is read, the xid is pushed to the preset waiting transmission queue waitTrans; When the XA Commit or the XA Rollback is read, the xid is removed from the waitTrans; When the XA Commit is read, the XAPrepare corresponding to the XA Commit is merged through the xid to generate a local transaction Transaction, and the Transaction is pushed to the preset transmission queue transQueue; The global timestamp TSO is used to sort the Transactions in the transQueue, a local transaction list is created through the XAEvent and the waitTrans in the transQueue, and a first-level sorting queue is generated according to the local transaction list.

4. The method according to claim 3, characterized in that The step of creating a local transaction list through the XA Event in the transQueue and the waitTrans further includes: Obtain the XAEvent from the sortItemsQueue in sequence; When the XAPrepare is obtained, the xid is extracted, and it is determined whether the XACommit with the same xid is received; If yes, remove the XAPrepare from the sortItemsQueue, and get the XAEvent from the sortItemsQueue again; If not, stop obtaining the XA Event from the sortItemsQueue; When the XA Commit is obtained, the XA Commit is removed from the sortItemsQueue; When the XA Rollback is obtained, the XA Rollback is removed from the sortItemsQueue; Determine the maximum timestamp maxTSO in the transQueue, and judge whether the global timestamp TSO corresponding to the XA Commit is greater than the maxTSO; If yes, the global timestamp TSO is used as the new maximum timestamp maxTSO; The transactions whose TSO in the transQueue is less than or equal to maxTSO are used as safe transactions to create a local transaction list, and the local transaction list is used to generate the first-level sorting queue, wherein the first-level sorting queue is composed of ordered local transactions.

5. The method according to claim 4, characterized in that After the step of sequentially obtaining the XA from the sortItemsQueue, the method further includes: Determine whether the XA Event obtained from the sortItemsQueue is empty; If so, stop obtaining the XA Event from the sortItemsQueue.

6. The method according to claim 4, characterized in that After the step of sequentially obtaining the XA from the sortItemsQueue, the method further comprises: Determine whether the duration of the XA Event obtained from the sortItemsQueue exceeds a preset threshold; If so, stop obtaining the XA Event from the sortItemsQueue.

7. The method according to any one of claims 1 to 6, characterized in that The step of performing multi-path merge sorting on the primary sorting queues of each DN to generate a global sorting queue comprises: Pushing the transactions in the first-level sorting queue in the local transaction list to the preset global transaction queue; The TSO is used to sort the Transactions in the global transaction queue to generate a global sorting queue.

8. The method according to claim 7, characterized in that The step of merging local transactions Transaction with the same global timestamp TSO for the global sorting queue and generating a global binary log Binlog includes: Reading transactions in the global sorting queue in sequence, and determining whether the global timestamp has changed; If yes, a complete global transaction is generated by merging transactions with the same global timestamp TSO; Extracting the characteristic event Event of the complete global transaction in the global sorting queue; The characteristic event Event is deleted, and the complete global transaction in the global sorting queue is used to generate a global binary log Binlog.

9. The method according to claim 8, characterized in that Also includes: In the global binary log Binlog, a query log RowsQueryLogEvent is added for the complete global transaction; wherein the RowsQueryLogEvent is used to record the global timestamp TSO corresponding to the complete global transaction.

10. The method according to claim 9, characterized in that The global binary log Binlog has a corresponding logical library table, and the logical library table has a corresponding logical library table name. The physical binary log Binlog has a corresponding physical library table, and the physical library table has a corresponding physical library table name; the logical library table name is used to replace the physical library table name when generating the global binary log Binlog; the logical library table and the physical library table have a mapping relationship, and the mapping relationship is acquired by the metadata system of the distributed database.

11. A data processing system for a distributed database, characterized in that: The distributed database has multiple storage nodes DN, each DN has a corresponding physical binary log Binlog, and the physical binary log Binlog is used to store local transaction operation information XA Event of a distributed transaction XA, and the XA includes at least one local transaction Transaction, and the local transaction Transaction is composed of the XA Event. The XA has a corresponding global timestamp TSO, and the system includes: A first-level sorting module, used to sort the transactions in the transQueue using the global timestamp TSO, create a local transaction list through the XA Event and waitTrans in the transQueue, and generate a first-level sorting queue according to the local transaction list; wherein the transQueue is a preset transmission queue, and the waitTrans is a preset waiting transmission queue; The global sorting module is used to perform multi-way merge sorting on the primary sorting queues of each DN to generate a global sorting queue; The transaction merging module is used to merge local transactions Transaction with the same global timestamp TSO for the global sorting queue, and generate a global binary log Binlog.

12. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the data processing method of the distributed database as described in one or more of claims 1-10.

13. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the data processing method for a distributed database as claimed in one or more of claims 1-10.

Images