Distributed transaction processing method, device and equipment based on append-only file storage

Abstract

The present specification relates to the technical field of big data storage and processing, and provides a distributed transaction processing method, device and equipment based on append-only file storage, which is applied to the master node side of a distributed database system, and includes: receiving a data insertion request sent by a client; dispatching an append-only operation task to multiple slave nodes according to the data insertion request, so that the multiple slave nodes execute the append-only operation task based on lanes in a distributed file system; receiving the append-only operation results returned by the multiple slave nodes; and submitting the append-only operation results returned by the multiple slave nodes as a transaction. The embodiments of the present specification can improve the data consistency and processing efficiency of distributed transaction processing.

Description

Technical Field

[0001] The present specification relates to the technical field of big data storage, and in particular to a distributed transaction processing method, apparatus and device based on append-only file storage. Background Art

[0002] In a distributed scenario, distributed transaction processing is generally implemented according to the XA protocol (an interface standard between a resource manager (database) and a transaction manager). However, this approach may result in data inconsistency. For example, suppose the coordinator sends a transaction commit notification, but due to network problems, the notification is only received by some transaction participants and the commit operation is performed, while the remaining transaction participants are blocked because they have not received the notification. Summary of the invention

[0003] The purpose of the embodiments of this specification is to provide a distributed transaction processing method, device and equipment based on append-only file storage to improve the data consistency and processing efficiency of distributed transaction processing.

[0004] To achieve the above objectives, on the one hand, an embodiment of this specification provides a distributed transaction processing method based on append-only file storage, which is applied to the master node side of a distributed database system, and the method includes:

[0005] Receive data insertion request sent by the client;

[0006] dispatching an append-only operation task to a plurality of slave nodes according to the data insertion request, so that the plurality of slave nodes execute the append-only operation task based on lanes in a distributed file system;

[0007] Receiving append-only operation results returned by the plurality of slave nodes;

[0008] The append-only operation results returned by the multiple slave nodes are submitted as a transaction.

[0009] In the embodiment of this specification, submitting the append-only operation results returned by the multiple slave nodes as a transaction includes:

[0010] Determining whether the append-only operations of the plurality of slave nodes are all successful according to the append-only operation results returned by the plurality of slave nodes;

[0011] When the append-only operations of the plurality of slave nodes are all successful, updating the local metadata;

[0012] When an append-only operation of a slave node is unsuccessful, a rollback operation task is dispatched to the multiple slave nodes, so that the multiple slave nodes execute the rollback operation task based on lanes in the distributed file system.

[0013] In the embodiment of the present specification, an append-only table is provided in the metadata; the append-only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field and a file logical length field before compression.

[0014] In the embodiment of the present specification, a column storage table is provided in the metadata; the column storage table includes a slave node field, a file actual logical length field, a file tuple number field and a file logical length field before compression.

[0015] In the embodiment of this specification, each of the lanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

[0016] In the embodiments of this specification, the distributed database system includes an MPP distributed database system.

[0017] In the embodiments of this specification, the distributed file system includes the Hadoop distributed file system.

[0018] On the other hand, the embodiment of this specification also provides a distributed transaction processing device based on append-only file storage, including:

[0019] A request receiving module, used for receiving a data insertion request sent by a client;

[0020] A task dispatching module, used for dispatching an append-only operation task to a plurality of slave nodes according to the data insertion request, so that the plurality of slave nodes execute the append-only operation task based on a lane in a distributed file system;

[0021] A result receiving module, used for receiving the append-only operation results returned by the plurality of slave nodes;

[0022] The transaction submission module is used to submit the append-only operation results returned by the multiple slave nodes as a transaction.

[0023] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of the above method are executed.

[0024] On the other hand, an embodiment of the present specification further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer device, the instructions of the above method are executed.

[0025] On the other hand, an embodiment of the present specification further provides a computer program product, which, when executed by a processor of a computer device, executes the instructions of the above method.

[0026] On the other hand, the embodiment of this specification also provides another distributed transaction processing method based on append-only file storage, which is applied to the slave node side of the distributed database system, and the method includes:

[0027] Receive append-only operation tasks dispatched by the master node;

[0028] Executing the append-only operation task based on the lane in the distributed file system to obtain the append-only operation result;

[0029] The append-only operation result is returned to the master node, so that the master node commits the append-only operation results returned by the plurality of slave nodes as one transaction.

[0030] In the embodiment of this specification, the performing of the append-only operation task based on a lane in a distributed file system includes:

[0031] Determine a swimming lane corresponding to the append-only operation task in the distributed file system;

[0032] Open the data file corresponding to the lane;

[0033] Inserting the data block specified in the append-only operation task at the end of the content of the data file corresponding to the lane;

[0034] Close the data file corresponding to the lane to prevent handle leakage.

[0035] In the embodiment of this specification, the master node submits the append-only operation results returned by the multiple slave nodes as a transaction, including:

[0036] The master node determines whether the append-only operations of the multiple slave nodes are all successful according to the append-only operation results returned by the multiple slave nodes;

[0037] When the append-only operations of the plurality of slave nodes are all successful, the master node updates the local metadata;

[0038] When the append-only operation of a slave node is unsuccessful, the master node dispatches a rollback operation task to the multiple slave nodes.

[0039] In the embodiment of this specification, the method further includes:

[0040] When a rollback operation task is received, determining a lane corresponding to the rollback operation task in the distributed file system;

[0041] Open the data file corresponding to the lane;

[0042] Clear the last data block of the data file corresponding to the lane;

[0043] Close the data file corresponding to the lane to prevent handle leakage.

[0044] In the embodiment of the present specification, an append-only table is provided in the metadata; the append-only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field and a file logical length field before compression.

[0045] In the embodiment of the present specification, a column storage table is provided in the metadata; the column storage table includes a slave node field, a file actual logical length field, a file tuple number field and a file logical length field before compression.

[0046] In the embodiment of this specification, each of the lanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

[0047] In the embodiments of this specification, the distributed database system includes an MPP distributed database system.

[0048] In the embodiments of this specification, the distributed file system includes the Hadoop distributed file system.

[0049] On the other hand, the embodiment of this specification also provides another distributed transaction processing device based on append-only file storage, the device comprising:

[0050] The task receiving module is used to receive the append-only operation tasks dispatched by the master node;

[0051] A task execution module, used for executing the append-only operation task based on the lane in the distributed file system to obtain the append-only operation result;

[0052] The result returning module is used to return the append-only operation result to the master node, so that the master node can submit the append-only operation results returned by multiple slave nodes as a transaction.

[0053] On the other hand, an embodiment of the present specification also provides another computer device, including a memory, a processor, and a computer program stored in the memory, and when the computer program is executed by the processor, the instructions of the above method are executed.

[0054] On the other hand, the embodiments of this specification also provide another computer storage medium, on which a computer program is stored. When the computer program is executed by a processor of a computer device, the instructions of the above method are executed.

[0055] On the other hand, the embodiments of this specification also provide another computer program product, which executes the instructions of the above method when executed by a processor of a computer device.

[0056] It can be seen from the technical solutions provided in the above embodiments of this specification that in the embodiments of this specification, since transaction submission is only performed on the master node, the slave node does not participate in the transaction submission, and the master node submits the append-only operation results returned by multiple slave nodes as a transaction, which is either successful or failed, and there will be no situation where some transactions are successfully submitted and some transactions fail to be submitted as in the prior art, thereby improving the data consistency of distributed transaction processing. In addition, the slave node performs the append-only operation task based on the lane in the distributed file system, and also realizes the concurrent append-only operation of data, which improves the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0058] Figure 1 A schematic diagram of two-phase submission of distributed transaction processing in the prior art is shown;

[0059] Figure 2 A structural block diagram of a distributed transaction processing system based on append-only file storage in some embodiments of this specification is shown;

[0060] Figure 3 A flow chart of a distributed transaction processing method (master node side) based on append-only file storage in some embodiments of this specification is shown;

[0061] Figure 4 A schematic diagram of a lane model in an embodiment of this specification is shown;

[0062] Figure 5 A schematic diagram of the storage structure of an append-only table in an embodiment of this specification is shown;

[0063] Figure 6 A schematic diagram of the storage structure of a column storage table in an embodiment of this specification is shown;

[0064] Figure 7 A flowchart of a distributed transaction processing method (from a node side) based on append-only file storage according to other embodiments of this specification is shown;

[0065] Figure 8 A distributed transaction processing flow chart based on append-only file storage of a distributed transaction processing system based on append-only file storage in some embodiments of this specification is shown;

[0066] Fig. 9 A structural block diagram of a distributed transaction processing device based on append-only file storage in some embodiments of this specification is shown;

[0067] Fig.10 A structural block diagram of a distributed transaction processing device based on append-only file storage in some other embodiments of this specification is shown;

[0068] Fig.11 The structure block diagram of the computer device in some embodiments of the present specification is shown.

[0069] [Description of Reference Numerals]

[0070] 10. Client;

[0071] 20. Distributed database system;

[0072] 30. Distributed file system;

[0073] 91. Request receiving module;

[0074] 92. Task dispatch module;

[0075] 93. Result receiving module;

[0076] 94. Transaction submission module;

[0077] 11. Task receiving module;

[0078] 12. Task execution module;

[0079] 13. Result return module;

[0080] 1102. Computer equipment;

[0081] 1104. Processor;

[0082] 1106. Memory;

[0083] 1108, driving mechanism;

[0084] 1110, input / output interface;

[0085] 1112. Input device;

[0086] 1114. Output device;

[0087] 1116. Presentation equipment;

[0088] 1118. Graphical user interface;

[0089] 1120, network interface;

[0090] 1122. Communication link;

[0091] 1124. Communication bus. DETAILED DESCRIPTION

[0092] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0093] The embodiments of this specification involve append-only operations; append-only is an operation mode in which all the contents of the appended file are written at the end of the original file content each time an append is made, which is essentially a special write mode. If the appended file does not exist, a file with a specified name will be created at the specified location. Accordingly, the file on which the append operation is performed (i.e., the data file of the distributed file system in the embodiments of this specification) can also be called an append-only file.

[0094] like Figure 1 As shown, the distributed transaction processing based on the XA protocol in the prior art is divided into two stages: the first stage: the transaction manager requires each resource manager involved in the transaction to pre-commit the operation and reflect whether it can be committed. The second stage: the transaction manager requires each resource manager to commit the data or roll back the data. However, if the transaction manager sends a transaction commit notification; but due to network problems, the notification is only received by a part of the resource managers and the commit operation is executed, the remaining resource managers are in a blocked state because they have not received the commit notification, and the resources are not released until the commit is completed. This will cause data inconsistency problems, and this blocking state affects the efficiency of distributed transaction processing.

[0095] In view of this, the embodiments of this specification provide a distributed transaction processing system based on append-only file storage. The distributed transaction processing system based on append-only file storage can be applied to any distributed system with similar problems, such as banking systems, telecommunications business systems, and e-commerce platforms. Figure 2As shown, in some embodiments, the distributed transaction processing system based on append-only file storage may include a client 10, a distributed database system 20, and a distributed file system 30. The client 10 is used to initiate a data insertion request to the distributed database system 20. The distributed database system 20 may include a master node and multiple slave nodes. The master node is used to receive the data insertion request sent by the client; dispatch an append-only operation task to multiple slave nodes according to the data insertion request, so that the multiple slave nodes perform the append-only operation task based on the lane in the distributed file system 30; receive the append-only operation results returned by the multiple slave nodes; and submit the append-only operation results returned by the multiple slave nodes as a transaction.

[0096] Since transaction submission is only performed on the master node, the slave nodes do not participate in transaction submission, and the master node submits the append-only operation results returned by multiple slave nodes as a transaction, which is either successful or failed, instead of the situation where some transactions are successfully submitted and some are failed as in the prior art, thereby improving the data consistency of distributed transaction processing. In addition, the slave nodes perform append-only operation tasks based on lanes in the distributed file system, and also realize data concurrent append-only operations, which improves processing efficiency.

[0097] In some embodiments, the distributed database system may be an electronic device with computing and network interaction functions; or it may be software running in the electronic device to provide business logic for data processing and network interaction. The user terminal may be a self-service terminal device, a mobile terminal (i.e., a smart phone), a display, a desktop computer, a tablet computer, a laptop computer, a digital assistant, or a smart wearable device, etc. Among them, smart wearable devices may include smart bracelets, smart watches, smart glasses, or smart helmets, etc. Of course, the user terminal is not limited to the above-mentioned electronic devices with certain entities, and it may also be software running in the above-mentioned electronic devices. The distributed file system may be an electronic device with computing and network interaction functions; or it may be software running in the electronic device to provide business logic for data processing and network interaction. The distributed file system of the embodiment of this specification supports only append operations and supports the clearing operation (Truncate) of data blocks (Block); the clearing operation of data blocks is to clear the data blocks at the end of the specified file.

[0098] In some embodiments, the distributed database system may be an MPP (Massive Parallel Processing) distributed database system, and the distributed file system may be an HDFS (Hadoop Distributed File System) file system. Thus, by utilizing the append-only feature of the HDFS file system (not capable of update and delete operations) and the MPP distributed transaction processing method, the transaction processing model may be simplified and efficient data writing may be ensured.

[0099] In the embodiments of this specification, the append-only operation is an operation mode in which all the contents of the appended file are written at the end of the original file content each time the appended file is appended, which is essentially a special write mode. If the appended file does not exist, a file with a specified name will be created at the specified location. Accordingly, the file on which the append operation is performed (i.e., the data file of the distributed file system in the embodiments of this specification) can also be called an append-only file.

[0100] refer to Figure 8 As shown, when data needs to be inserted, the user can input an insert statement through the client so as to initiate a data insertion request to the master node of the distributed database system through the client. The data insertion request may carry one or more user data to be inserted. For example, in an exemplary embodiment, the user data to be inserted may include: user XX deposited 10,000 yuan into its account ZZ through the self-service terminal YY at 12:00:00 on September 1, 2021; account MM transferred 10,000 yuan to account NN through the mobile banking APP at 12:00:00 on September 1, 2021; and so on. When the master node receives a data insertion request, one or more (generally multiple) slave nodes can be selected from the slave node set of the distributed database system according to the data insertion request, and these slave nodes can be dispatched to append only operation tasks, so that these slave nodes can perform append only operation tasks based on lanes in the distributed file system, and append user data to the data files of the distributed file system. For example, in an exemplary embodiment, a portion of the selected slave nodes are assigned to insert deposit business data, a portion of the selected slave nodes are assigned to insert transfer business data, a portion of the selected slave nodes are assigned to insert retrieval business data, and so on.

[0101] In order to improve the efficiency of distributed transaction processing, a lane model is used in the embodiments of this specification. Among them, the lane model divides the lanes according to roles (for example, only appending processes), organizes activities (for example, only appending data operations) according to responsibilities, and the activities of each role are scattered in the lanes corresponding to each role. In this way, different activities of different roles are divided into different areas (that is, logically and / or physically isolated), and these areas are called lanes. The lanes will not call each other; if an abnormality occurs in the activities in any lane, it will not affect other lanes, which can also help to achieve error isolation.

[0102] A lane model can be pre-configured in a distributed file system. The number of lanes in the lane model can be determined based on the number of slave nodes in the slave node set and the upper limit of concurrent transactions. Each lane corresponds to a data file (i.e., a user data file). For example, in an exemplary embodiment, the number of slave nodes is 100 and the upper limit of concurrent transactions is 20. Then, the upper limit of the number of lanes in the pre-configured lane model is 100×20=2000. In real time, corresponding lanes can be formed as needed. Each lane can include a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

[0103] In some embodiments, assuming that the distributed database system has N slave nodes (Segments), for a single insert operation of user data, each slave node only needs to write the data into a separate file. In other words, at this time, N data files are opened at the same time and data is written in an append-only manner. For concurrent insert operations of user data, different concurrent operations write to different data files and do not interfere with each other. When there are C concurrent insert operations, the number of files opened and written at the same time is F = N × C.

[0104] For example, in an exemplary embodiment, for a distributed database system with three slave nodes, the corresponding swim lanes and file layout on the distributed file system when performing two concurrent inserts can be as follows: Figure 4 As shown. Figure 4 In the example, the first transaction is insert 1, and the second transaction is insert 2. Each insert has a file on each slave node for concurrent data writing. A total of 6 lanes are formed here, and the write operations of each lane are performed independently without affecting each other. The specific execution steps of each lane are also the same as those of a single insert operation. At this time, the paths of the 6 lanes on the distributed file system are:

[0105] / filespace_oid / database_oid / table_oid / 1,

[0106] / filespace_oid / database_oid / table_oid / 2,

[0107] / filespace_oid / database_oid / table_oid / 3,

[0108] / filespace_oid / database_oid / table_oid / 4,

[0109] / filespace_oid / database_oid / table_oid / 5,

[0110] / filespace_oid / database_oid / table_oid / 6.

[0111] Where filespace_oid is the number of the file space, database_oid is the number of the database, and table_oid is the number of the table.

[0112] In some embodiments, for each slave node (i.e., each slave node to which the master node sends an append-only operation task), the slave node performs the append-only operation task based on the lane in the distributed file system, which may include:

[0113] (1) Receive the append-only operation tasks dispatched by the master node.

[0114] (2) Executing the append-only operation task based on the lane in the distributed file system to obtain the append-only operation result may specifically include:

[0115] 1) Determine the lane corresponding to the append-only operation task in the distributed file system. Since each append-only operation task corresponds to a slave node identifier and content to be appended, and each lane may include a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier; accordingly, the slave node can find the corresponding lane to execute the append-only operation task in the lane.

[0116] 2) Open the data file corresponding to the lane.

[0117] 3) At the end of the content of the data file corresponding to the lane, insert the data block specified in the append-only operation task. For example, before this append, there are five data blocks in the data file corresponding to the lane from the beginning to the end: data block A, data block B, data block C, data block D and data block E; then in this append, a data block F can be added after data block E; that is, after this append, there are six data blocks in the data file corresponding to the lane from the beginning to the end: data block A, data block B, data block C, data block D, data block E and data block F.

[0118] 4) Close the data file corresponding to the lane to prevent handle leakage.

[0119] (2) Returning the append-only operation result to the master node, so that the master node commits the append-only operation results returned by the multiple slave nodes as a transaction.

[0120] In the embodiments of this specification, a complete data append includes not only an append-only operation on a data file in a distributed file system (i.e., writing user data), but also an update operation on metadata stored locally in a distributed database. Metadata is data used to describe a data file. Therefore, each slave node needs to return the append-only operation result to the master node, so that the master node can submit the append-only operation results returned by each slave node as a transaction to ensure data consistency.

[0121] In some embodiments, for a master node (e.g., a transaction manager of the master node), submitting the append-only operation results returned by multiple slave nodes as a transaction may include:

[0122] (1) According to the append-only operation results returned by the multiple slave nodes, it is determined whether the append-only operations of the multiple slave nodes are all successful.

[0123] (2) When the append-only operations of the plurality of slave nodes are all successful, the local metadata is updated.

[0124] When the append-only operations of the multiple slave nodes are all successful, it indicates that the append-only operation for the data insertion request is successful. Therefore, the local metadata can be updated. In this way, the metadata recording the file information and the data file on the append-only distributed file storage correspond to each other, thereby completing the entire data insertion transaction for the data insertion request.

[0125] In the embodiments of this specification, for each user data table, a system table can be added in the metadata (Catalog), and the naming rules are as follows: for AO tables (i.e., append-only tables), pg_aoseg.pg_aoseg_$tableoid is used; and for Parquet tables (i.e., column storage tables), pg_aoseg.pg_paqseg_$tableoid can be used. Figure 5 As shown in FIG. 1 , the AO table may include a secondary node field (segno), a file actual logical length field (eof), a file tuple number field (tupcount), a file variable length block number field (varblockcount), and a file logical length field before compression (eofuncompressed). Figure 6 As shown in the figure, the Parquet table can include the slave node field (segno), the actual logical length field of the file (eof), the number of file tuples field (tupcount), and the logical length field before file compression (eofuncompressed). When data is appended to a data file in a distributed file system, the corresponding actual logical length of the file, the number of file tuples, the number of file variable-length blocks, and the logical length before file compression will change. Therefore, when updating metadata, the relevant fields of the AO table or Parquet table need to be modified to maintain data consistency.

[0126] (3) When an append-only operation of a slave node fails, a rollback operation task is dispatched to the multiple slave nodes, so that the multiple slave nodes execute the rollback operation task based on the lane in the distributed file system.

[0127] When the append-only operation of any slave node fails, it indicates that the append-only operation for the data insertion request has failed as a whole. In this case, the local metadata cannot be updated (i.e., the local metadata remains unchanged). Therefore, rollback operation tasks need to be dispatched to multiple slave nodes to maintain data consistency.

[0128] In some embodiments, for each slave node, performing the rollback operation task may include:

[0129] (1) When a rollback operation task is received, determine a lane corresponding to the rollback operation task in the distributed file system.

[0130] (2) Open the data file corresponding to the lane.

[0131] (3) Clear the last data block of the data file corresponding to the lane. For example, in an exemplary embodiment, after the append, the data file corresponding to the lane has six data blocks from the first to the last: data block A, data block B, data block C, data block D, data block E, and data block F; among which, data block F is a newly added data block, and the rollback operation is to clear (Truncate) data block F.

[0132] (4) Close the data file corresponding to the lane to prevent handle leakage.

[0133] refer to Figure 8 As shown, in the embodiment of this specification, after confirming that the transaction is committed or rolled back, the master node can also return a data insertion response to the client to feedback the transaction processing result to the user.

[0134] refer to Figure 3 As shown, the embodiment of this specification provides a distributed transaction processing method based on append-only file storage, which can be applied to the master node side of a distributed database system. The method may include the following steps:

[0135] S301: Receive a data insertion request sent by a client.

[0136] S302. Dispatching an append-only operation task to a plurality of slave nodes according to the data insertion request, so that the plurality of slave nodes execute the append-only operation task based on lanes in a distributed file system.

[0137] S303: Receive the append-only operation results returned by the multiple slave nodes.

[0138] S304: Submit the append-only operation results returned by the multiple slave nodes as a transaction.

[0139] In the embodiments of this specification, since transaction submission is only performed on the master node, the slave node does not participate in the transaction submission, and the master node submits the append-only operation results returned by multiple slave nodes as a transaction, which is either successful or failed, and there will be no situation where some transactions are successfully submitted and some transactions fail to be submitted as in the prior art, thereby improving the data consistency of distributed transaction processing. In addition, the slave node performs the append-only operation task based on the lane in the distributed file system, and also realizes the concurrent append-only operation of data, which improves the processing efficiency.

[0140] exist Figure 3 In the illustrated embodiment, submitting the append-only operation results returned by the multiple slave nodes as a transaction may include:

[0141] Determining whether the append-only operations of the plurality of slave nodes are all successful according to the append-only operation results returned by the plurality of slave nodes;

[0142] When the append-only operations of the plurality of slave nodes are all successful, updating the local metadata;

[0143] When an append-only operation of a slave node is unsuccessful, a rollback operation task is dispatched to the multiple slave nodes, so that the multiple slave nodes execute the rollback operation task based on lanes in the distributed file system.

[0144] exist Figure 3 In the illustrated embodiment, an append-only table is provided in the metadata; the append-only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable-length block number field, and a file logical length field before compression.

[0145] exist Figure 3 In the illustrated embodiment, a column storage table is provided in the metadata; the column storage table includes a slave node field, a file actual logical length field, a file tuple number field, and a file logical length field before compression.

[0146] exist Figure 3 In the illustrated embodiment, each of the lanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

[0147] exist Figure 3 In the illustrated embodiment, the distributed database system includes an MPP distributed database system.

[0148] exist Figure 3 In the illustrated embodiment, the distributed file system includes a Hadoop distributed file system.

[0149] refer to Figure 7 As shown, the embodiment of this specification provides another distributed transaction processing method based on append-only file storage, which can be applied to the slave node side of a distributed database system. The method may include the following steps:

[0150] S701, receiving the append-only operation task dispatched by the master node.

[0151] S702: Execute the append-only operation task based on the lane in the distributed file system to obtain the append-only operation result.

[0152] S703: Return the append-only operation result to the master node, so that the master node submits the append-only operation results returned by multiple slave nodes as a transaction.

[0153] In the embodiments of this specification, since transaction submission is only performed on the master node, the slave node does not participate in the transaction submission, and the master node submits the append-only operation results returned by multiple slave nodes as a transaction, which is either successful or failed, and there will be no situation where some transactions are successfully submitted and some transactions fail to be submitted as in the prior art, thereby improving the data consistency of distributed transaction processing. In addition, the slave node performs the append-only operation task based on the lane in the distributed file system, and also realizes the concurrent append-only operation of data, which improves the processing efficiency.

[0154] exist Figure 7 In the illustrated embodiment, the performing of the append-only operation task based on a lane in a distributed file system includes:

[0155] Determine a swimming lane corresponding to the append-only operation task in the distributed file system;

[0156] Open the data file corresponding to the lane;

[0157] Inserting the data block specified in the append-only operation task at the end of the content of the data file corresponding to the lane;

[0158] Close the data file corresponding to the lane to prevent handle leakage.

[0159] exist Figure 7 In the illustrated embodiment, the master node submits the append-only operation results returned by the plurality of slave nodes as a transaction, including:

[0160] The master node determines whether the append-only operations of the multiple slave nodes are all successful according to the append-only operation results returned by the multiple slave nodes;

[0161] When the append-only operations of the plurality of slave nodes are all successful, the master node updates the local metadata;

[0162] When the append-only operation of a slave node is unsuccessful, the master node dispatches a rollback operation task to the multiple slave nodes.

[0163] exist Figure 7 In the illustrated embodiment, the method further comprises:

[0164] When a rollback operation task is received, determining a lane corresponding to the rollback operation task in the distributed file system;

[0165] Open the data file corresponding to the lane;

[0166] Clear the last data block of the data file corresponding to the lane;

[0167] Close the data file corresponding to the lane to prevent handle leakage.

[0168] exist Figure 7 In the illustrated embodiment, an append-only table is provided in the metadata; the append-only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable-length block number field, and a file logical length field before compression.

[0169] exist Figure 7 In the illustrated embodiment, a column storage table is provided in the metadata; the column storage table includes a slave node field, a file actual logical length field, a file tuple number field, and a file logical length field before compression.

[0170] exist Figure 7 In the illustrated embodiment, each of the lanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

[0171] exist Figure 7 In the illustrated embodiment, the distributed database system includes an MPP distributed database system.

[0172] exist Figure 7 In the illustrated embodiment, the distributed file system includes a Hadoop distributed file system. Although the process flow described above includes multiple operations that appear in a specific order, it should be clearly understood that these processes may include more or fewer operations, which may be performed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment).

[0173] With the above Figure 3 Corresponding to the distributed transaction processing method based on append-only file storage shown in FIG. 1 , the embodiment of this specification also provides a distributed transaction processing device based on append-only file storage, referring to FIG. Fig. 9 As shown, the distributed transaction processing device for appending only file storage may include: a request receiving module 91, a task dispatching module 92, a result receiving module 93 and a transaction submitting module 94. Among them:

[0174] The request receiving module 91 may be used to receive a data insertion request sent by a client.

[0175] The task dispatching module 92 may be used to dispatch append-only operation tasks to multiple slave nodes according to the data insertion request, so that the multiple slave nodes execute the append-only operation tasks based on lanes in the distributed file system.

[0176] The result receiving module 93 may be used to receive the append-only operation results returned by the plurality of slave nodes.

[0177] The transaction commit module 94 may be used to commit the append-only operation results returned by the multiple slave nodes as a transaction.

[0178] With the above Figure 7 Corresponding to the distributed transaction processing method based on append-only file storage shown in FIG. 1 , the embodiment of this specification also provides another distributed transaction processing device based on append-only file storage, referring to FIG. Fig.10 As shown, the distributed transaction processing device for appending only file storage may include: a task receiving module 11, a task executing module 12 and a result returning module 13. Among them:

[0179] The task receiving module 11 may be used to receive the append-only operation tasks dispatched by the master node.

[0180] The task execution module 12 may be used to execute the append-only operation task based on the lane in the distributed file system to obtain the append-only operation result.

[0181] The result returning module 13 may be used to return the append-only operation result to the master node, so that the master node can submit the append-only operation results returned by multiple slave nodes as a transaction.

[0182] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0183] The embodiments of this specification also provide a computer device. Fig.11As shown, in some embodiments of the present specification, the computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs) or graphics processing units (GPUs), and each processing unit may implement one or more hardware threads. The computer device 1102 may also include any memory 1106, which is used to store any kind of information such as code, settings, data, etc. In a specific embodiment, the computer program on the memory 1106 and can be run on the processor 1104, when the computer program is run by the processor 1104, the instructions of the distributed transaction processing method based on append-only file storage described in any of the above embodiments may be executed. Non-limitingly, for example, the memory 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1102. In one embodiment, when the processor 1104 executes the associated instructions stored in any memory or combination of memories, the computer device 1102 can perform any operation of the associated instructions. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0184] The computer device 1102 may also include an input / output interface 1110 (I / O) for receiving various inputs (via input devices 1112) and for providing various outputs (via output devices 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface 1118 (GUI). In other embodiments, the input / output interface 1110 (I / O), input device 1112, and output device 1114 may not be included, and the computer device 1102 may be used as a computer device in a network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0185] The communication link 1122 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products of some embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processor to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processor generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processor to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processor so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0190] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0191] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computer device. As defined in this specification, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0192] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0193] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processors connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0194] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0195] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the method and device embodiments, since they are basically similar to the system embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment.

[0196] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0197] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A distributed transaction processing method based on append-only file storage, applied to the master node side of a distributed database system, characterized in that: The method comprises: Receive data insertion request sent by the client; Dispatching an append-only operation task to a plurality of slave nodes according to the data insertion request, so that the plurality of slave nodes execute the append-only operation task based on lanes in a distributed file system; the executing the append-only operation task based on lanes in a distributed file system by the plurality of slave nodes comprises: determining, by the plurality of slave nodes, a lane corresponding to the append-only operation task in the distributed file system; opening, by the plurality of slave nodes, a data file corresponding to the lane; inserting, by the plurality of slave nodes, a data block of the appended data specified in the append-only operation task at the end of the content of the data file corresponding to the lane; and closing, by the plurality of slave nodes, the data file corresponding to the lane to prevent handle leakage; Receiving append-only operation results returned by the plurality of slave nodes; The append-only operation results returned by the multiple slave nodes are submitted as a transaction; the submitting the append-only operation results returned by the multiple slave nodes as a transaction includes: judging whether the append-only operations of the multiple slave nodes are all successful based on the append-only operation results returned by the multiple slave nodes; when the append-only operations of the multiple slave nodes are all successful, updating local metadata; when the append-only operation of a slave node is unsuccessful, dispatching a rollback operation task to the multiple slave nodes, so that the multiple slave nodes execute the rollback operation task based on lanes in the distributed file system.

2. The distributed transaction processing method based on append-only file storage according to claim 1, characterized in that: An append-only table is provided in the metadata; the append-only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field and a file logical length field before compression.

3. The distributed transaction processing method based on append-only file storage according to claim 1, characterized in that: A column storage table is provided in the metadata; the column storage table includes a slave node field, a file actual logical length field, a file tuple number field and a file logical length field before compression.

4. The distributed transaction processing method based on append-only file storage according to claim 1, characterized in that: Each of the swimming lanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

5. The distributed transaction processing method based on append-only file storage according to claim 1, characterized in that: The distributed database system includes an MPP distributed database system.

6. The distributed transaction processing method based on append-only file storage according to claim 1, characterized in that: The distributed file system includes the Hadoop distributed file system.

7. A distributed transaction processing device based on append-only file storage, characterized in that: include: A request receiving module, used for receiving a data insertion request sent by a client; A task dispatching module, used for dispatching an append-only operation task to a plurality of slave nodes according to the data insertion request, so that the plurality of slave nodes execute the append-only operation task based on a lane in a distributed file system; The multiple slave nodes executing the append-only operation task based on the lane in the distributed file system includes: the multiple slave nodes determining the lane corresponding to the append-only operation task in the distributed file system; the multiple slave nodes opening the data file corresponding to the lane; the multiple slave nodes inserting the data block specified in the append-only operation task at the end of the content of the data file corresponding to the lane; the multiple slave nodes closing the data file corresponding to the lane to prevent handle leakage; A transaction submission module is used to submit the append-only operation results returned by the multiple slave nodes as a transaction; submitting the append-only operation results returned by the multiple slave nodes as a transaction includes: judging whether the append-only operations of the multiple slave nodes are all successful based on the append-only operation results returned by the multiple slave nodes; when the append-only operations of the multiple slave nodes are all successful, updating local metadata; when the append-only operation of a slave node is unsuccessful, dispatching a rollback operation task to the multiple slave nodes, so that the multiple slave nodes execute the rollback operation task based on lanes in the distributed file system.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 1 to 6.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes instructions of the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product is executed by a processor of a computer device, the computer program product executes instructions of the method according to any one of claims 1 to 6.

11. A distributed transaction processing method based on append-only file storage, applied to the slave node side of a distributed database system, characterized in that: The method comprises: Receive append-only operation tasks dispatched by the master node; The append-only operation task is executed based on a lane in a distributed file system to obtain an append-only operation result; the append-only operation task is executed based on a lane in a distributed file system, including: determining a lane corresponding to the append-only operation task in the distributed file system; opening a data file corresponding to the lane; inserting a data block specified in the append-only operation task to the end of the content of the data file corresponding to the lane; and closing the data file corresponding to the lane to prevent handle leakage; Returning the append-only operation result to the master node, so that the master node submits the append-only operation results returned by multiple slave nodes as a transaction; the master node submits the append-only operation results returned by the multiple slave nodes as a transaction, including: the master node determines whether the append-only operations of the multiple slave nodes are all successful based on the append-only operation results returned by the multiple slave nodes; when the append-only operations of the multiple slave nodes are all successful, the master node updates local metadata; when the append-only operation of a slave node is unsuccessful, the master node dispatches a rollback operation task to the multiple slave nodes.

12. The distributed transaction processing method based on append-only file storage according to claim 11, characterized in that: The method further comprises: When a rollback operation task is received, determining a lane corresponding to the rollback operation task in the distributed file system; Open the data file corresponding to the lane; Clear the last data block of the data file corresponding to the lane; Close the data file corresponding to the lane to prevent handle leakage.

13. The distributed transaction processing method based on append-only file storage according to claim 11, characterized in that: An append-only table is provided in the metadata; the append-only table includes a slave node field, a file actual logical length field, a file tuple number field, a file variable length block number field and a file logical length field before compression.

14. The distributed transaction processing method based on append-only file storage according to claim 11, characterized in that: A column storage table is provided in the metadata; the column storage table includes a slave node field, a file actual logical length field, a file tuple number field and a file logical length field before compression.

15. The distributed transaction processing method based on append-only file storage according to claim 11, characterized in that: Each of the swimming lanes includes a slave node identifier, a file space identifier, a database identifier, a user data table identifier, and an append-only operation task identifier.

16. The distributed transaction processing method based on append-only file storage according to claim 11, characterized in that: The distributed database system includes an MPP distributed database system.

17. The distributed transaction processing method based on append-only file storage according to claim 11, characterized in that: The distributed file system includes the Hadoop distributed file system.

18. A distributed transaction processing device based on append-only file storage, characterized in that: include: The task receiving module is used to receive the append-only operation tasks dispatched by the master node; The performing of the append-only operation task based on the lane in the distributed file system includes: determining the lane corresponding to the append-only operation task in the distributed file system; opening the data file corresponding to the lane; inserting the data block specified in the append-only operation task at the end of the content of the data file corresponding to the lane; and closing the data file corresponding to the lane to prevent handle leakage. A task execution module, used for executing the append-only operation task based on the lane in the distributed file system to obtain the append-only operation result; The result returning module is used to return the append-only operation result to the master node, so that the master node can submit the append-only operation results returned by multiple slave nodes as a transaction; the master node submits the append-only operation results returned by the multiple slave nodes as a transaction, including: the master node determines whether the append-only operations of the multiple slave nodes are all successful based on the append-only operation results returned by the multiple slave nodes; when the append-only operations of the multiple slave nodes are all successful, the master node updates the local metadata; when the append-only operation of a slave node is unsuccessful, the master node dispatches a rollback operation task to the multiple slave nodes.

19. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 11 to 17.

20. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes the instructions of the method according to any one of claims 11-17.

21. A computer program product, characterized in that When the computer program product is executed by a processor of a computer device, the instructions of the method according to any one of claims 11 to 17 are executed.

Images