Distributed transaction processing method, system and apparatus

Abstract

Embodiments of the present application disclose a distributed transaction processing method, system and device, which can avoid outdated metadata and transaction processing errors caused by outdated metadata, improve transaction processing efficiency, reduce computing, transmission and storage overhead, and ensure complete fault tolerance and disaster tolerance of the system. The distributed transaction processing method comprises the following steps: firstly, starting a distributed transaction GT and a transaction branch CT of the distributed transaction GT on a computing node by using the computing node according to a command statement, the command statement comprising a data definition language (DDL) statement or a data manipulation language (DML) statement sent by a client, and the local transaction branch CT being completed on the computing node; secondly, obtaining a parsing result by parsing the command statement by using the computing node, the parsing result comprising an internal representation and an execution state of the command statement; and finally, creating or modifying metadata in a local metadata table by using the computing node according to the parsing result, the created or modified metadata being related to the command statement, and the metadata table being stored on the computing node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed transaction processing, and particularly relates to a distributed transaction processing method, system and device. BACKGROUND

[0002] A transaction is a minimum logical unit of work that accesses a database to achieve a specific business function. A distributed transaction refers to a transaction that may involve multiple database operations.

[0003] Generally, a distributed transaction processing system usually includes three types of components: a computing node, a storage node and a metadata node. The storage node is mainly responsible for storing data and forming a cluster based on a data change replication mechanism to achieve a high availability mechanism. The metadata node is responsible for storing metadata of the cluster, including a cluster topology structure and metadata of user data managed by the cluster. The computing node is used to receive a transaction processing request and process a transaction corresponding to the transaction processing request according to a preset business rule.

[0004] In related distributed processing technology, the computing node obtains cluster metadata and user metadata from a metadata cluster or a metadata storage system on the metadata node during transaction processing, and then temporarily caches the obtained metadata locally for use. If the computing node or the metadata node fails, causing metadata loss and disorder, the temporarily cached metadata of the computing node is not the latest, thereby causing processing errors in the transaction processing process. SUMMARY

[0005] Therefore, it is necessary to solve the above problems. The present application provides a distributed transaction processing method, system and device, which can avoid outdated metadata and transaction processing errors caused by outdated metadata, improve transaction processing efficiency, and reduce computing, transmission and storage overheads.

[0006] In a first aspect, the present application provides a distributed transaction processing method, which includes the following steps: first, using a computing node to start a distributed transaction GT and a local transaction branch CT of the distributed transaction GT according to a command statement, the command statement including a data definition language (DDL) statement or a data manipulation language (DML) statement sent by a client, and the local transaction branch CT being completed on the computing node; second, using the computing node to parse the command statement to obtain a parsing result, the parsing result including an internal representation and an execution state of the command statement; and third, using the computing node to newly create or modify metadata in a local metadata table according to the parsing result, the newly created or modified metadata being related to the command statement, and the metadata table being stored on the computing node.

[0007] Optionally, in a possible implementation manner of the first aspect, if the command statement is a DDL statement, the distributed transaction processing method further comprises:

[0008] starting, by the computing node, a transaction branch MT of the distributed transaction GT on the metadata node in the connection between the computing node and the metadata node, the transaction branch MT being executed on the metadata node;

[0009] if the transaction branch MT is successfully executed on the metadata node, sending, by the metadata node, an MT execution success message to the computing node;

[0010] if the transaction branch MT is interrupted in the execution on the metadata node, resuming, by the metadata node, the execution of the transaction branch MT until the execution is successful, and sending, to the computing node, an MT execution failure message.

[0011] Optionally, in a possible implementation manner of the first aspect, the distributed transaction processing method further comprises:

[0012] sending, by the computing node, the DDL transaction log to the metadata node in the connection between the computing node and the metadata node, so that the metadata node stores the DDL transaction log locally.

[0013] Optionally, in a possible implementation manner of the first aspect, if the storage node supports DDL statement execution, the distributed transaction processing method further comprises:

[0014] starting, by the computing node, a transaction branch ST of the distributed transaction GT on the storage node in the connection between the computing node and the storage node;

[0015] receiving, by the storage node, the DDL statement sent by the computing node, and executing the DDL statement in the transaction branch ST;

[0016] if the transaction branch ST is successfully executed on the storage node, sending, by the storage node, an ST execution success message to the computing node;

[0017] if the transaction branch ST is interrupted in the execution on the storage node, resuming, by the storage node, the execution of the transaction branch ST until the execution is successful, and sending, to the computing node, an ST execution failure message.

[0018] Optionally, in a possible implementation manner of the first aspect, if the command statement is a DML statement, the distributed transaction processing method further comprises:

[0019] starting, by the computing node, a transaction branch ST of the distributed transaction GT on the storage node in the connection between the computing node and the storage node, the transaction branch ST being executed on the storage node;

[0020] When the computing node receives the commit transaction command sent by the client, the computing node initiates a GT commit process of the distributed transaction, and the execution process of the commit transaction command includes ending the transaction branch ST by the storage node.

[0021] Optionally, in a possible implementation manner of the first aspect, the commit transaction command includes a first commit command and a second commit command, and initiating the GT commit process of the distributed transaction by the computing node includes:

[0022] sending, by the computing node, the first commit command to the first type of storage node, the first type of storage node being a storage node corresponding to a storage cluster that has performed a write operation in the execution process of the distributed transaction GT;

[0023] sending, by the computing node, the second commit command to the second type of storage node, the second type of storage node being a storage node corresponding to a storage cluster that has performed a read-only operation in the execution process of the distributed transaction GT.

[0024] Optionally, in a possible implementation manner of the first aspect, the distributed transaction processing method further includes:

[0025] if the first commit command is all successfully returned to the computing node, writing, by the computing node, a commit decision of the decision commit into a transaction log;

[0026] if the first commit command is not all successfully returned to the computing node, writing, by the computing node, a commit decision of the decision rollback into the transaction log;

[0027] after the commit decision of the decision commit or the decision rollback is written into the transaction log by the computing node, sending, by the computing node, the first commit command to the first type of storage node;

[0028] committing, by the computing node, the local transaction branch CT.

[0029] Optionally, in a possible implementation manner of the first aspect, the writing, by the computing node, of the commit decision of the decision commit or the decision rollback into the transaction log specifically includes:

[0030] if the metadata cluster is accessible, writing, by the computing node, the commit decision of the decision commit or the decision rollback into a transaction log of the metadata cluster;

[0031] if the metadata cluster is not accessible, writing, by the computing node, the commit decision of the decision commit or the decision rollback into a transaction log of any one of the storage clusters that can be normally written.

[0032] In a second aspect, the present application provides a distributed transaction processing system, comprising: a computing node, a storage node and a metadata node; and the distributed transaction processing method is executed by using the computing node, the storage node and the metadata node.

[0033] In a third aspect, the present application provides a distributed transaction processing device, which is applied to a distributed transaction processing system, wherein the distributed transaction processing system provides multiple transaction processing services, and comprises multiple transaction coordinators and multiple storages, the transaction coordinators and the storages are one-to-one correspondingly arranged, and the distributed transaction processing device comprises:

[0034] A command starting module, configured to start a distributed transaction GT and a transaction branch CT of the distributed transaction GT on the computing node by using the computing node according to a command statement, wherein the command statement comprises a data definition language (DDL) statement or a data manipulation language (DML) statement sent by a client;

[0035] A parsing module, configured to parse the command statement by using the computing node to obtain a parsing result, wherein the parsing result comprises an internal representation and an execution state of the command statement, and the local transaction branch CT is completed on the computing node;

[0036] A processing module, configured to newly create metadata or modify target metadata in a local metadata table on the computing node according to the parsing result, wherein the newly created metadata or the target metadata is related to the command statement, and the metadata table is stored on the computing node.

[0037] In a fourth aspect, the present application provides a distributed transaction processing device, comprising:

[0038] A processor; and

[0039] A storage, which stores executable codes, wherein when the executable codes are executed by the processor, the processor executes the distributed transaction processing method in the first aspect and any one of the implementation manners thereof.

[0040] The technical scheme provided by the present application has the following beneficial effects:

[0041] In the distributed transaction processing method, the metadata table is stored on the computing node and supports local transaction branch processing, the computing node does not need to repeatedly acquire metadata from the metadata node but acquires the metadata from the locally stored metadata table, so that the out-of-date metadata and the transaction processing errors caused by the out-of-date metadata can be avoided, the transaction processing efficiency is improved, the computing, transmission and storage overheads are reduced, and the system has complete fault tolerance and disaster recovery capabilities.

[0042] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout and in which:

[0044] Figure 1 A system structure schematic diagram of a distributed transaction processing system in an embodiment of the present application;

[0045] Figure 2 An embodiment flow schematic diagram of a distributed transaction processing method in an embodiment of the present application;

[0046] Figure 3 A flow schematic diagram of a DDL distributed transaction processing mechanism in a distributed transaction processing method in an embodiment of the present application;

[0047] Figure 4 A flow schematic diagram of a DML distributed transaction processing mechanism in a distributed transaction processing method in an embodiment of the present application;

[0048] Figure 5 A structure schematic diagram of a distributed transaction processing apparatus in an embodiment of the present application;

[0049] Figure 6 Another structure schematic diagram of a distributed transaction processing apparatus in an embodiment of the present application;

[0050] Figure 7 Another structure schematic diagram of a distributed transaction processing apparatus in an embodiment of the present application;

[0051] Figure 8 Another structure schematic diagram of a distributed transaction processing apparatus in an embodiment of the present application. DETAILED DESCRIPTION

[0052] Embodiments of the present application will be described herein below with reference to the accompanying drawings. While the present application is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the application to the particular embodiments described herein, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the application.

[0053] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the accompanying claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0054] It should be understood that although the terms "first," "second," "third," etc. can be employed in describing various information used in the application, such information should not be limited by these terms. These terms are only used to distinguish one piece of information from another. For example, a first piece of information could be termed a second piece of information without departing from the scope of the application. As such, an element proceeded by "first," "second," etc. should not be limited by these terms. In the description of the application, the meaning of "a plurality" is two or more, unless explicitly specified otherwise.

[0055] In order to facilitate understanding of the technical solutions in the embodiments of the application, a basic concept involved in the application will be described first as follows:

[0056] A cluster refers to a group (several) of independent computers, which form a larger computer service system using a high-speed communication network, and each cluster node (i.e. each computer in the cluster) is an independent server running its own service. These servers can communicate with each other, cooperate to provide application programs, system resources and data to users, and are managed in a single system mode. When a user requests a cluster system, the cluster gives the user the feeling of a single independent server, while the user actually requests a group of cluster servers.

[0057] Metadata, also known as intermediate data and relay data, is data about data, mainly information describing data properties (property), used to support functions such as indicating storage location, historical data, resource search, file recording, etc.

[0058] The technical solutions in the present application are mainly used for distributed transaction processing, and are particularly suitable for distributed transaction processing (DTP) of a distributed database management system (DDBMS).

[0059] Figure 1 A system structure diagram of a DDBMS system in an embodiment of the present application.

[0060] As Figure 1 shown, the DDBMS system in the embodiment of the present application includes three types of components: computing nodes 101, storage nodes 102 and metadata nodes 103, wherein the computing nodes 101 are used to receive transaction processing requests and process transactions corresponding to the transaction processing requests according to preset business rules. The storage nodes 102 are mainly responsible for storing data and implementing a high-availability mechanism based on a data change replication mechanism to form a cluster. The metadata nodes 103 are responsible for storing metadata of the cluster, including a cluster topology structure and metadata of user data managed by the cluster.

[0061] Specifically, the computing nodes 101 are mainly responsible for the following work: 1. accepting a client connection request, verifying the request and establishing a connection state; 2. processing a query request sent by the client connection and returning a result to the client; 3. interacting with the storage nodes to complete data reading and writing and implement a coordination function of a distributed transaction coordinator (DTC) to complete reliable submission of a distributed transaction.

[0062] The query requests executed by the DDBMS system are mainly divided into two categories: one is a data definition language (DDL), that is, a statement for defining a data storage method (such as a table, a container and the like) of a user; and the other is a data manipulation language (DML), that is, a statement for reading and writing data in an already established storage unit (such as a table, a container and the like).

[0063] After the DDBMS system is described, the distributed transaction processing method in the embodiment of the present application is described below in combination with the accompanying drawings.

[0064] Figure 2 An embodiment flowchart of the distributed transaction processing method in the embodiment of the present application.

[0065] As Figure 2 shown, the distributed transaction processing method in the embodiment of the present application includes:

[0066] 201. Starting a distributed transaction GT and a transaction branch CT of the distributed transaction GT in a local computing node using the computing node according to a command statement.

[0067] The command statement includes a query request sent by the client, and a connection state is established between the client and the computing node. After receiving the command statement, the computing node starts a corresponding distributed transaction GT according to a preset business rule, and starts a local transaction distribution CT of the distributed transaction GT on the computing node. The command statement includes a DDL statement or a DML statement.

[0068] The distributed transaction GT has a transaction branch in the metadata cluster (i.e., on the metadata node) and a transaction branch in the storage cluster (i.e., on the storage node). The transaction branch of the distributed transaction GT in the metadata cluster is referred to as a transaction branch MT, and the transaction branch of the distributed transaction GT in the storage cluster is referred to as a transaction branch ST. The types of command languages are different, the corresponding business rules are different, and the transaction branches started by the computing node are also different.

[0069] When the command statement is a DDL statement, the computing node executes the DDL statement by using a DDL distributed transaction processing mechanism. The computing node starts the transaction branch MT on the metadata node while starting the local transaction branch CT.

[0070] Optionally, since part of the storage nodes in the DDBMS system support the DDL statement, and another part of the storage nodes do not support the DDL statement, if the storage node supports the DDL statement, the computing node also starts the transaction branch ST on the storage node when executing the DDL statement.

[0071] When the command statement is a DML statement, the computing node executes the DDL statement by using a DML distributed transaction processing mechanism. The computing node starts the transaction branch ST on the storage node while starting the local transaction branch CT.

[0072] The difference from the related art is that the transaction branch CT of the distributed transaction GT on the computing node is running on the computing node, that is, the computing node reads and writes the local metadata in the local transaction branch CT, so that the data consistency of the read and write access of the metadata in the computing node and the user data in the storage node during the running of the distributed transaction can be ensured.

[0073] 202. The command statement is parsed by using the computing node to obtain a parsing result.

[0074] After starting the distributed transaction and the corresponding transaction branch, the command statement is parsed in the local transaction branch CT of the computing node and the local metadata table is queried to establish the internal representation and execution state of the command statement, that is, the parsing result.

[0075] The internal representation and execution state of the command statement refer to a representation manner that can be recognized and executed by the node in the node after the command statement is parsed, such as creating an object, a data structure table, and the like.

[0076] Specifically, the internal representation and execution state of the created object are described by way of example. For example, a transaction starts to execute a statement such as a "start transaction" statement, and after parsing, the transaction manager creates a transaction object. As the subsequent client sends a DDL or DML statement in this connection, the statement is executed in the transaction (labeled GT), and each statement reads and writes a number of storage clusters (i.e., shards or storage shards). The distributed transaction processor module of the computing node records in the Transaction object that GT read from and wrote to which storage clusters; the state of each storage cluster at least includes the connection information (ip, port, username, password, etc.) of its current master node, so as to establish a connection with the target node and repeatedly send statements and then accept the results.

[0077] The above storage cluster is briefly introduced by way of example of a Kunlun distributed database cluster. A Kunlun distributed database cluster includes a plurality of storage clusters and a plurality of computing nodes. Each storage cluster stores a portion of data, and the data between shards has no intersection. All computing nodes of a Kunlun database cluster are completely identical in function and status, and are mainly used to complete distributed query processing and distributed transaction processing, as well as connection authentication and access control, etc.

[0078] The internal representation and execution state of the created object are described by way of example. For example, a query statement such as a "select * from t1" statement is parsed to obtain a 'query plan' data structure plan, which references the opened table t1 (i.e., a data structure of an opened table) that needs to be queried, and the manner and parameters of executing the query after query optimization (i.e., a data structure describing a full table scan operation, which references the opened table t1, and includes a file page number and an offset in the page that are currently scanned, a set of columns to be returned, and a data structure describing a query condition).

[0079] After obtaining the parsing result, the computing node determines the metadata related to the command statement according to the internal representation and execution state of the command statement.

[0080] 203. New or modify the metadata in the local metadata table of the computing node according to the parsing result.

[0081] After determining the metadata related to the command statement, the computing node queries the local metadata table, and after finding, modifies or newly creates the related metadata using the internal representation and execution state of the command statement, so that the computing node completes the transaction branch CT of the distributed transaction GT locally.

[0082] In summary, in the distributed transaction processing method of the present application, the computing node stores the metadata table and supports local transaction branch processing, the computing node does not need to repeatedly obtain metadata from the metadata node but obtains metadata locally, which can avoid outdated metadata and transaction processing errors caused by outdated metadata, improve transaction processing efficiency, reduce computing, transmission and storage overheads, and ensure complete fault tolerance and disaster recovery capability of the system.

[0083] As described above, when the command statement is a DDL statement, the computing node executes the DDL statement using the DDL distributed transaction processing mechanism. When the command statement is a DML statement, the computing node executes the DDL statement using the DML distributed transaction processing mechanism.

[0084] The DDL distributed transaction processing mechanism and the DML distributed transaction processing mechanism in the distributed transaction processing method of the present application embodiment will be described below in connection with the accompanying drawings.

[0085] Figure 3 A flowchart of the DDL distributed transaction processing mechanism in the distributed transaction processing method of the present application embodiment.

[0086] 301. The computing node starts a distributed transaction GT and a transaction branch CT of the distributed transaction GT in the computing node according to the DDL statement.

[0087] After the computing node receives the DDL statement sent by the client, the computing node starts the corresponding distributed transaction GT and a transaction branch CT of the distributed transaction GT in the computing node.

[0088] Further, the computing node uses the computing node to parse the DDL statement to obtain a parsing result, and uses the computing node to newly create or modify metadata in the local metadata table according to the parsing result, to complete the transaction branch CT.

[0089] Next, the computing node starts other transaction branches of the distributed transaction GT, such as the transaction branch MT and the transaction branch ST.

[0090] It should be noted that the execution of the transaction branch CT of the distributed transaction GT in the computing node in step 301 is the same as the execution of the transaction branch CT of the distributed transaction GT in the corresponding embodiment described above, and the detailed description can be referred to the description of the corresponding embodiment described above. Figure 2 The execution of the transaction branch CT of the distributed transaction GT in the corresponding embodiment is the same as the execution of the transaction branch CT of the distributed transaction GT in the corresponding embodiment described above, and the detailed description can be referred to the description of the corresponding embodiment described above. Figure 2 The execution of the transaction branch CT of the distributed transaction GT in the corresponding embodiment is the same as the execution of the transaction branch CT of the distributed transaction GT in the corresponding embodiment described above, and the detailed description can be referred to the description of the corresponding embodiment described above.

[0091] 302. The computing node starts a transaction branch MT of the distributed transaction GT on the metadata node in the connection between the computing node and the metadata node.

[0092] In the DDBMS system, a connection MetaConn is established between the computing node and the metadata node. The computing node initiates a transaction branch MT of the distributed transaction GT on the metadata node through the computing node and the metadata node establishing the connection MetaConn, and the transaction branch MT is executed on the metadata node.

[0093] Further, if the transaction branch MT is successfully executed on the metadata node, the metadata node is used to send an MT execution success message to the computing node to inform the computing node that the transaction branch MT is successfully executed.

[0094] Further, if the transaction branch MT is interrupted during execution on the metadata node, the metadata node is used to resume execution of the transaction branch MT until the execution is successful, and an MT execution failure message is sent to the computing node to inform the computing node that an error occurred during execution of the transaction branch MT.

[0095] It should be noted that, since the metadata node independently supports local transaction branch MT processing, when a fault occurs during execution of the transaction branch MT, causing the execution to be interrupted, the metadata node can resume (i.e., automatically roll back) execution of the transaction branch MT, which can ensure that the transaction branch MT can be correctly restored in the case of a metadata node fault.

[0096] Through the above operations, the metadata node can independently execute and complete the transaction branch MT.

[0097] 303、Using the computing node to send the DDL transaction log to the metadata node in the connection between the computing node and the metadata node.

[0098] During the distributed transaction processing, the computing node obtains a corresponding operation record table such as a DDL transaction log to record the distributed transaction processing flow. The function of the DDL transaction log is to use it to recover or replay the distributed transaction processing to ensure the consistency of the distributed DDL transaction. The consistency refers to that in the distributed transaction processing, a read operation can always read the result of a previously completed write operation.

[0099] In the DDL distributed transaction processing mechanism, the DDL transaction log is added and stored in the local metadata table, so that the computing node can use the transaction log to recover and replay to ensure the consistency of the distributed transaction, and the transaction log can also be used by multiple computing nodes to complete synchronization of metadata data in the metadata table.

[0100] Optionally, 304, using the computing node to initiate a transaction branch ST of the distributed transaction GT on the storage node in the connection between the computing node and the storage node.

[0101] Since not all storage nodes support DDL statement execution in the DDBMS system, some storage nodes do not support DDL statement execution, and therefore, step 304 is optional.

[0102] In the DDBMS system in which the storage nodes support DDL statement execution, the connection StoreConn is established between the computing node and the storage node, and the computing node initiates the transaction branch ST of the distributed transaction GT on the storage node in the connection StoreConn.

[0103] After the transaction branch ST is initiated, the storage node receives the DDL statement sent by the computing node through the connection StoreConn, and then the storage node executes the DDL statement in the transaction branch ST.

[0104] Further, if the transaction branch ST is successfully executed on the storage node, the storage node sends an ST execution success message to the computing node to notify the computing node that the transaction branch ST is successfully executed;

[0105] Further, if the transaction branch ST is interrupted during execution on the storage node, the storage node resumes execution of the transaction branch ST until it is successfully executed, and sends an ST execution failure message to the computing node to notify the computing node that an error occurred during execution of the transaction branch ST.

[0106] Similar to the metadata node, since the storage node independently supports local transaction branch ST processing, when a failure occurs during execution of the transaction branch ST, causing the execution to be aborted, the storage node can resume (i.e., automatically roll back) execution of the transaction branch ST, and can ensure that the transaction branch ST can be correctly restored in the case of a metadata node failure.

[0107] 305、After the distributed transaction GT and its transaction branches are executed, the computing node performs a commit process to commit the distributed transaction GT.

[0108] After the distributed transaction GT and its transaction branches are executed, the computing node submits the transaction branch MT of the distributed transaction GT through the connection MetaConn, and when the storage node supports DDL statement execution, the computing node also submits the transaction branch ST of the distributed transaction GT through the connection StoreConn, and the computing node submits the transaction branch CT of the distributed transaction GT to complete the commit process of the entire distributed transaction.

[0109] To sum up, in the DDL distributed transaction processing mechanism of the distributed transaction processing method in the embodiment of the application, since the computing node and the metadata node can both support local transaction branch processing independently, the distributed distributed transaction processing can be realized between the computing node and the metadata node, on the one hand, the metadata consistency of all nodes of the system can be ensured, meanwhile, the computing node and the metadata node can independently bear a large amount of concurrent load, and the running speed and response speed of the distributed transaction processing are improved; on the other hand, when the transaction branch MT is interrupted during execution, the metadata node can resume the execution of the corresponding transaction branch MT (i.e. automatically rollback) until the execution is successful, so that the distributed transaction MT can be correctly recovered when the execution is interrupted, and the loss or disorder of metadata is avoided.

[0110] Further, in the DDL distributed transaction processing mechanism, when the storage node supports the execution of the DDL statement, the storage node can also have the ability to independently execute the transaction branch ST of the DDL transaction on the storage node, and can independently resume the execution of the transaction branch ST when the transaction branch ST is interrupted, so that the function of the storage node to independently execute the transaction branch ST is realized, and the distributed transaction S can be correctly recovered when the execution is interrupted, and the loss or disorder of storage data is avoided.

[0111] Figure 4 A flowchart of a DML distributed transaction processing mechanism in the distributed transaction processing method in the embodiment of the application.

[0112] 401. A computing node starts a distributed transaction GT and a transaction branch CT of the distributed transaction GT on the computing node according to a DML statement.

[0113] After the computing node receives the DML statement sent by the client, the computing node starts a corresponding distributed transaction GT and a transaction branch CT of the distributed transaction GT on the computing node.

[0114] Further, the computing node uses the computing node to parse the DML statement to obtain a parsing result, and uses the computing node to newly create or modify metadata in the metadata table on the computing node according to the parsing result, so as to complete the transaction branch CT.

[0115] It should be noted that the computing node starts and executes the transaction branch CT of the distributed transaction GT on the computing node in step 401, which is the same as the execution mode of the transaction branch CT of the distributed transaction GT in the corresponding embodiment, and the detailed description can be referred to the description of the related part in the corresponding embodiment, which will not be described here. Figure 2 The execution mode of the transaction branch CT of the distributed transaction GT in the corresponding embodiment is the same as the execution mode of the transaction branch CT of the distributed transaction GT in the corresponding embodiment, and the detailed description can be referred to the description of the related part in the corresponding embodiment, which will not be described here. Figure 2 The execution mode of the transaction branch CT of the distributed transaction GT in the corresponding embodiment is the same as the execution mode of the transaction branch CT of the distributed transaction GT in the corresponding embodiment, and the detailed description can be referred to the description of the related part in the corresponding embodiment, which will not be described here.

[0116] 402、using the computing node to initiate a transaction branch ST of the distributed transaction GT on the storage node in the connection between the computing node and the storage node.

[0117] As described above, in the DDBMS system, the connection StoreConn is established between the computing node and the storage node, and the computing node initiates the transaction branch ST of the distributed transaction GT on the storage node in the connection StoreConn.

[0118] After initiation, the storage node initiates the transaction branch ST and executes the DML statement, wherein the DML statement is sent by the computing node to the storage node through the connection StoreConn.

[0119] 403、when the computing node receives the commit transaction command sent by the client, the computing node initiates a commit process of the distributed transaction GT.

[0120] The commit transaction command sent by the client includes a commit command or a prepare command, which is used to instruct to save the modification made by executing the distributed transaction GT to the storage cluster. In the process of the commit transaction command, the storage node ends the transaction branch ST.

[0121] Optionally, the commit transaction command includes a first commit command and a second commit command, and the computing node initiates the commit process of the distributed transaction GT, including: 1) using the computing node to send the first commit command to the first type of storage node, the first type of storage node being the storage node corresponding to the storage cluster that has executed the write operation in the execution process of the distributed transaction GT; 2) using the computing node to send the second commit command to the second type of storage node, the second type of storage node being the storage node corresponding to the storage cluster that has executed the read-only operation in the execution process of the distributed transaction GT.

[0122] The first commit command and the second commit command can instruct the storage node to save all the modifications (i.e. read and write) made in the distributed transaction GT and its branch transaction to the corresponding storage cluster, so as to end the transaction branch ST on the storage node.

[0123] In addition, when the computing node receives the commit transaction command sent by the client, the computing node also commits the transaction branch CT executed locally by the computing node, so as to end the entire commit process of the distributed transaction GT.

[0124] 404、after the first commit transaction command returns to the computing node, the computing node writes the commit decision of deciding to commit or deciding to roll back into the transaction log of the metadata cluster or the storage cluster.

[0125] After all the first commit commands return to the computing node, the computing node generates corresponding commit decisions and writes the commit decisions into the transaction log of the metadata cluster or the storage cluster. The commit decisions can be stored in the metadata cluster or the storage cluster to prevent loss.

[0126] It should be understood that the first commit commands can be returned to the computing node successfully or not returned to the computing node successfully, and the first commit commands not returned to the computing node successfully means that at least one of the first commit commands is not returned to the computing node successfully. Similarly, the corresponding commit decisions are divided into commit decisions of commit and commit decisions of rollback.

[0127] Specifically, if all the first commit commands are returned to the computing node successfully, the commit decisions of commit are written into the transaction log by the computing node; if the first commit commands are not returned to the computing node successfully, the commit decisions of rollback are written into the transaction log by the computing node.

[0128] Further, after the commit decisions of commit or rollback are written into the transaction log by the computing node, the first commit commands are sent to the first type of storage node by the computing node.

[0129] Finally, the local transaction branch CT is committed by the computing node to complete the distributed transaction processing flow of the entire DML statement.

[0130] The commit decisions of commit or rollback only need to be written into the metadata cluster or the storage cluster for storage, and the storage can be in the transaction log of any metadata node in the metadata cluster or in the transaction log of any normal storage node in the storage cluster.

[0131] In the case that the metadata cluster in the DDBMS system is inaccessible, the commit decisions of commit or rollback can only be written into the transaction log of the storage cluster. Correspondingly, in the case that the metadata cluster is accessible, the commit decisions of commit or rollback can be written into the transaction log of any normal storage node or metadata node, and the commit decisions of commit or rollback are preferably written into the transaction log of the storage cluster.

[0132] Specifically, the commit decisions of commit or rollback are written into the transaction log by the computing node, and the writing specifically includes:

[0133] If the metadata cluster is accessible, the commit decisions of commit or rollback are written into the transaction log of the metadata cluster by the computing node;

[0134] If the metadata cluster cannot be accessed, the computing node writes the commit decision or the commit decision rollback into the transaction log of any storage node in the storage cluster that can be normally written.

[0135] In summary, in the DML distributed transaction processing mechanism of the distributed transaction processing method in the embodiments of the present application, the transaction processing function is implemented in the computing node, the metadata in the local metadata table is read and written in the local transaction of the computing node, and the data consistency of the read and write access of the metadata in the computing node and the user data in the storage node during the running of the distributed transaction is ensured; meanwhile, the computing node and the storage node can independently execute transaction processing, and the interrupted commit of the distributed transaction can be recovered when any computing node or storage node in the cluster fails, thereby improving the reliability.

[0136] Corresponding to the foregoing application function implementation method embodiments, the present application further provides a distributed transaction processing device, system and corresponding embodiments.

[0137] Figure 5 A structural schematic diagram of the distributed transaction processing device in the embodiments of the present application.

[0138] The distributed transaction processing device is applied to a distributed transaction processing system; the distributed transaction processing system provides multiple transaction processing services; the distributed transaction processing system includes multiple transaction coordinators and multiple storage devices, and the transaction coordinators and the storage devices are one-to-one correspondingly arranged.

[0139] As shown in Figure 5 The distributed transaction processing device 50 in the embodiments of the present application includes a command starting module 501, an analysis module 502 and a processing module 503, wherein the command starting module 501 is used for starting the distributed transaction GT and the transaction branch CT of the distributed transaction GT in the computing node according to the command statement by using the computing node, the command statement including the data definition language DDL statement or the data manipulation language DML statement sent by the client; the analysis module 502 is used for analyzing the command statement by using the computing node to obtain an analysis result, the analysis result including the internal representation and the execution state of the command statement, and the local transaction branch CT being completed in the computing node; and the processing module 503 is used for creating or modifying the metadata in the local metadata table by using the computing node according to the analysis result, the created metadata or the target metadata being related to the command statement, and the metadata table being stored in the computing node.

[0140] Figure 6 Another structural schematic diagram of the distributed transaction processing device in the embodiments of the present application.

[0141] Compared with the prior art, Figure 5 Figure 6 ​The device structure shown has a newly added DDL transaction processing module 504.

[0142] Optionally, in one embodiment of the present invention, if the command statement is a DDL statement, the distributed transaction processing device 50 further includes a DDL transaction processing module 504, which is used to perform the following operations:

[0143] The distributed transaction GT is started using the compute node in the connection between the compute node and the metadata node, and the transaction branch MT is executed on the metadata node.

[0144] If the transaction branch MT executes successfully on the metadata node, the metadata node is used to send a message indicating that MT has been executed successfully to the compute node.

[0145] If the execution of transaction branch MT is interrupted on the metadata node, the execution of transaction branch MT is resumed using the metadata node until it is successfully executed, and an MT execution failure message is sent to the compute node.

[0146] Optionally, in one embodiment of the present invention, the DDL transaction processing module 504 is further configured to: use the compute node to send DDL transaction logs to the metadata node in the connection between the compute node and the metadata node, so that the metadata node stores the DDL transaction logs locally.

[0147] Optionally, in one embodiment of the present invention, if the storage node supports DDL statement execution, the DDL transaction processing module 504 is further configured to perform the following operations:

[0148] Initiate a distributed transaction GT on the storage node using the compute node in the connection between the compute node and the storage node;

[0149] The storage node receives DDL statements sent by the compute node and executes the DDL statements in the transaction branch ST.

[0150] If the transaction branch ST executes successfully on the storage node, the storage node sends a message indicating that ST has been executed successfully to the compute node.

[0151] If the execution of transaction branch ST is interrupted on the storage node, the execution of transaction branch ST is resumed using the storage node until it is successfully executed, and an ST execution failure message is sent to the compute node.

[0152] Figure 7 This is another structural schematic diagram of the distributed transaction processing device in an embodiment of the present invention.

[0153] and Figure 5 compared to, Figure 7 The device structure shown has a newly added DDL transaction processing module 505.

[0154] Optionally, in an implementation of the embodiment of the present application, if the command statement is a DML statement, the distributed transaction processing device 50 further comprises a DML transaction processing module 505, and is configured to perform the following operations:

[0155] The computing node initiates a transaction branch ST of the distributed transaction GT on the storage node using the computing node, and the transaction branch ST is executed on the storage node.

[0156] When the computing node receives a commit transaction command sent by the client, the computing node initiates a commit process of the distributed transaction GT, and the commit transaction command includes an operation of ending the transaction branch ST of the storage node in a process of executing the commit transaction command.

[0157] Optionally, in an implementation of the embodiment of the present application, the commit transaction command comprises a first commit command and a second commit command, and the DML transaction processing module 505 is specifically configured to perform the following operations to initiate the commit process of the distributed transaction GT:

[0158] The computing node sends the first commit command to a first type of storage node using the computing node, and the first type of storage node is a storage node corresponding to a storage cluster that has performed a write operation in a process of executing the distributed transaction GT.

[0159] The computing node sends the second commit command to a second type of storage node using the computing node, and the second type of storage node is a storage node corresponding to a storage cluster that has performed a read-only operation in the process of executing the distributed transaction GT.

[0160] Optionally, in an implementation of the embodiment of the present application, the DML transaction processing module 505 is further configured to perform the following operations: if the first commit command is all successfully returned to the computing node, the DML transaction processing module 505 writes a commit decision of deciding to commit into a transaction log using the computing node;

[0161] If the first commit command is not all successfully returned to the computing node, the DML transaction processing module 505 writes a commit decision of deciding to roll back into the transaction log using the computing node;

[0162] After the DML transaction processing module 505 writes the commit decision of deciding to commit or deciding to roll back into the transaction log using the computing node, the DML transaction processing module 505 sends the first commit command to the first type of storage node using the computing node;

[0163] The DML transaction processing module 505 commits the local transaction branch CT using the computing node.

[0164] Optionally, in an implementation of the embodiment of the present application, the DML transaction processing module 505 writes the commit decision of the decision commit or the decision rollback into the transaction log using the computing node specifically performs the following operations:

[0165] If the metadata cluster is accessible, the DML transaction processing module 505 writes the commit decision of the decision commit or the decision rollback into the transaction log of the metadata cluster using the computing node;

[0166] If the metadata cluster is not accessible, the DML transaction processing module 505 writes the commit decision of the decision commit or the decision rollback into the transaction log of any storage node that can be normally written in the storage cluster using the computing node.

[0167] In addition, the embodiment of the present application provides a distributed transaction processing system including a computing node, a storage node and a metadata node; and the distributed transaction processing method as described in the above method embodiment is executed using the computing node, the storage node and the metadata node.

[0168] In summary, in the distributed transaction processing device of the present application, the metadata table is stored on the computing node and the local transaction branch processing is supported, the computing node does not need to repeatedly acquire metadata from the metadata node but acquires from the local, which can avoid the metadata out-of-date and the transaction processing error caused by the metadata out-of-date, improve the transaction processing efficiency, reduce the computing, transmission and storage overhead, and ensure that the system has complete fault tolerance and disaster recovery capability.

[0169] Figure 8 Another structural schematic diagram of the distributed transaction processing device in the embodiment of the present application.

[0170] Figure 8 Another structural schematic diagram of the distributed transaction processing device in the embodiment of the present application.

[0171] As shown in Figure 8 the distributed transaction processing device 80 in the embodiment of the present application includes a memory 801 and a processor 802. The executable code is stored on the memory, and when the executable code is executed by the processor, the processor executes the method in any of the above embodiments.

[0172] The processor 802 can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can be any conventional processor.

[0173] The memory 801 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 802 or other modules of the computer. The permanent storage device can be a read and write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read and write storage device or a volatile read and write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 801 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 801 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and a transient electronic signal transmitted through a wireless or wired transmission.

[0174] The memory 801 stores executable code, which, when processed by the processor 802, can cause the processor 802 to perform part or all of the above-mentioned methods.

[0175] Furthermore, the method according to the present application can also be implemented as a computer program or a computer program product, which comprises computer program code instructions for executing part or all of the steps of the above-mentioned method of the present application.

[0176] Alternatively, the present application can also be implemented as a computer readable storage medium (or machine readable storage medium) having stored thereon executable codes (or computer programs, or computer instruction codes) which, when executed by a processor of a distributed transaction processing device (or electronic equipment, server, etc.), cause the processor to perform part or all of the steps of the above-mentioned method according to the present application.

[0177] The embodiments of the present application have been described above with the aid of numerous drawings. The above description is illustrative only and not exhaustive, and is not intended to limit the disclosed embodiments to the precise steps and / or conditions described. Many modifications and variations are possible in light of the above teachings. The selection of elements and / or configurations is optional. The selection of terms is intended to best describe the principles of the embodiments of the present application, the practical application, or the improvement over the technology in the art, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A method of distributed transaction processing, characterized by, The distributed transaction processing method comprises the following steps: starting a distributed transaction GT and a local transaction branch CT of the distributed transaction GT on the computing node according to a command statement, the command statement comprising a data definition language (DDL) statement sent by a client, the local transaction branch CT being completed on the computing node; parsing the command statement to obtain a parsing result using the computing node, the parsing result comprising an internal representation and an execution state of the command statement; creating or modifying metadata in a local metadata table according to the parsing result using the computing node, the metadata being related to the command statement, the metadata table being stored on the computing node, the metadata comprising cluster topology and user data managed by the cluster; if the command statement is a DDL statement, the distributed transaction processing method further comprises the following steps: starting a transaction branch MT of the distributed transaction GT on a metadata node in a connection between the computing node and the metadata node using the computing node, the transaction branch MT being executed on the metadata node; if the transaction branch MT is successfully executed on the metadata node, sending an MT execution success message to the computing node using the metadata node; if the transaction branch MT is interrupted in execution on the metadata node, resuming execution of the transaction branch MT on the metadata node until the execution is successful, and sending an MT execution failure message to the computing node to inform the computing node that an error has occurred in the execution of the transaction branch MT.

2. The distributed transaction processing method of claim 1, wherein, The distributed transaction processing method further comprises the following steps: sending a DDL transaction log to the metadata node in the connection between the computing node and the metadata node using the computing node, so that the metadata node stores the DDL transaction log locally.

3. The distributed transaction processing method according to claim 1 or 2, characterized by, if the storage node supports execution of the DDL statement, the distributed transaction processing method further comprises the following steps: starting a transaction branch ST of the distributed transaction GT on the storage node in a connection between the computing node and the storage node using the computing node; receiving the DDL statement sent by the computing node and executing the DDL statement in the transaction branch ST using the storage node; if the transaction branch ST is successfully executed on the storage node, sending an ST execution success message to the computing node using the storage node; if the transaction branch ST is interrupted in execution on the storage node, resuming execution of the transaction branch ST on the storage node until the execution is successful, and sending an ST execution failure message to the computing node.

4. A distributed transaction processing system, characterized by The distributed transaction processing apparatus is applied to a distributed transaction processing system, and the distributed transaction processing system provides multiple transaction processing services. The distributed transaction processing apparatus is applied to a distributed transaction processing system, and the distributed transaction processing system provides multiple transaction processing services. ​ 5. A distributed transaction processing apparatus characterized by comprising: ​ The distributed transaction processing system comprises a plurality of transaction coordinators and a plurality of memories, the transaction coordinators are arranged in one-to-one correspondence with the memories, and the distributed transaction processing device comprises: A command starting module is configured to start a distributed transaction GT and a local transaction branch CT of the distributed transaction GT on the computing node according to a command statement, wherein the command statement comprises a data definition language (DDL) statement sent by a client; A parsing module is configured to parse the command statement to obtain a parsing result using the computing node, wherein the parsing result comprises an internal representation and an execution state of the command statement, and the local transaction branch CT is completed on the computing node; A processing module is configured to create new metadata or modify target metadata in a metadata table on the computing node according to the parsing result, wherein the new metadata or the target metadata is related to the command statement, the metadata table is stored on the computing node, and the metadata comprises a cluster topology structure and user data managed by the cluster; If the command statement is a DDL statement, the distributed transaction processing method further comprises: Starting a transaction branch MT of the distributed transaction GT on the metadata node using the computing node in a connection between the computing node and the metadata node, wherein the transaction branch MT is executed on the metadata node; If the transaction branch MT is successfully executed on the metadata node, sending an MT execution success message to the computing node using the metadata node; If the transaction branch MT is interrupted on the metadata node, resuming execution of the transaction branch MT on the metadata node until the execution is successful, and sending an MT execution failure message to the computing node to inform the computing node that an error occurs in the execution process of the transaction branch MT.

Images