Method, device, apparatus and computer-readable medium for database transaction processing

Abstract

The present invention discloses a method, apparatus, device and computer-readable medium for database transaction processing, and relates to the field of computer technology. A specific implementation of the method includes: recording the data file name of the database transaction in the recovery file of the temporary directory of the distributed node; when the recovery file does not exist in the cloud platform, moving the data file of the database transaction from the temporary directory to the data file directory of the cloud platform, and moving the recovery file of the temporary directory to the data file directory of the cloud platform to submit the database transaction; based on the transaction sequence number of the database transaction, updating the symbolic link name of the data file in the visibility file; moving the recovery file of the cloud platform to the directory of the shared storage of the cloud platform to determine the successful submission of the database transaction. This implementation can improve the efficiency of database transaction processing in cloud native scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and computer-readable medium for database transaction processing. Background Art

[0002] Transactions are an important feature of relational databases. As databases evolve from single-machine centralized databases to distributed databases, single-machine database transactions also become distributed transactions. Distributed transactions are usually implemented using the two-phase commit protocol (2PC). The execution process of the two-phase commit is divided into two phases, the voting phase and the commit phase, as its name suggests.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: the main reason why distributed transactions use the two-phase commit protocol is that the two-phase commit is aimed at common distributed scenarios. For cloud-native scenarios that rely on basic services provided by cloud platforms, the efficiency of processing transactions is low. Summary of the invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and computer-readable medium for database transaction processing, which can improve the efficiency of database transaction processing in cloud-native scenarios.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for database transaction processing is provided, comprising:

[0006] Record the data file name of the database transaction in the recovery file of the temporary directory of the distributed node;

[0007] When the recovery file does not exist in the cloud platform, the data file of the database transaction is moved from the temporary directory to the data file directory of the cloud platform, and the recovery file of the temporary directory is moved to the data file directory of the cloud platform to submit the database transaction;

[0008] Based on the transaction sequence number of the database transaction, updating the symbolic link name of the data file in the visibility file, the visibility file being used for concurrency control of distributed nodes;

[0009] The recovery file of the cloud platform is moved to a directory of the cloud platform shared storage to confirm successful submission of the database transaction.

[0010] The database transaction includes an update transaction;

[0011] Before recording the data file name of the database transaction into the recovery file of the temporary directory of the distributed node, the method further includes:

[0012] A temporary data file is newly created in the distributed node, and the update transaction is executed in the temporary data file to update the database transaction.

[0013] The database transaction includes a non-update transaction;

[0014] Before recording the data file name of the database transaction into the recovery file of the temporary directory of the distributed node, it also includes

[0015] The data file of the database transaction is stored in the data file in the temporary directory.

[0016] When there is a recovery file in the cloud platform;

[0017] The method further comprises:

[0018] After deleting the visibility file of the data file in the cloud platform, the newly added file of the data file in the cloud platform is deleted, and the visibility file of the cloud platform is updated.

[0019] The method further comprises:

[0020] Traverse the directory of the shared storage of the cloud platform and determine the cleaning sequence number according to the reference count of the data file, where the reference count is used to record the number of times the data file is called;

[0021] Delete data files whose transaction sequence numbers are less than or equal to the cleanup sequence number.

[0022] The updating of the symbolic link name of the data file in the visibility file based on the transaction sequence number of the database transaction includes:

[0023] Associating the newly created symbolic link file in the data file with the data file corresponding to the transaction sequence number of the database transaction;

[0024] In the visibility file, the newly created symbolic link name of the data file is updated with the transaction sequence number of the database transaction.

[0025] Before submitting the database transaction, the method further includes: performing a locking operation;

[0026] After determining that the database transaction is successfully submitted, the method further includes: performing an unlocking operation.

[0027] According to a second aspect of an embodiment of the present invention, there is provided a device for database transaction processing, including:

[0028] A recording module is used to record the data file name of the database transaction into the recovery file of the temporary directory of the distributed node;

[0029] A submission module, configured to move the data file of the database transaction from the temporary directory to the data file directory of the cloud platform, and to move the recovery file of the temporary directory to the data file directory of the cloud platform, so as to submit the database transaction when the recovery file does not exist in the cloud platform;

[0030] An update module, configured to update the symbolic link name of the data file in a visibility file based on the transaction sequence number of the database transaction, wherein the visibility file is used for concurrent control of distributed nodes;

[0031] The moving module is used to move the recovery file of the cloud platform to a directory of the shared storage of the cloud platform to determine that the database transaction is successfully submitted.

[0032] According to a third aspect of an embodiment of the present invention, there is provided an electronic device for database transaction processing, comprising:

[0033] one or more processors;

[0034] a storage device for storing one or more programs,

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0036] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method as described above is implemented.

[0037] One embodiment of the above invention has the following advantages or beneficial effects: recording the data file name of the database transaction in the recovery file of the temporary directory of the distributed node; when the recovery file does not exist in the cloud platform, moving the data file of the database transaction from the temporary directory to the data file directory of the cloud platform, and moving the recovery file of the temporary directory to the data file directory of the cloud platform to submit the database transaction; based on the transaction sequence number of the database transaction, updating the symbolic link name of the data file in the visibility file, the visibility file is used for the concurrency control of the distributed nodes; moving the recovery file of the cloud platform to the directory of the shared storage of the cloud platform to determine the successful submission of the database transaction. The distributed node uploads the data file of the database transaction to the cloud platform to process the database transaction, and other distributed nodes can obtain the above data file, thereby improving the efficiency of database transaction processing in the cloud scenario.

[0038] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0040] Figure 1 It is a schematic diagram of the main flow of the method for database transaction processing according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of a distributed database architecture according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a framework for submitting a database transaction according to an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of a background cleaning process according to an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram of a process of updating a symbolic link name in a visibility file according to an embodiment of the present invention;

[0045] Figure 6 is a schematic diagram of the main structure of a device for database transaction processing according to an embodiment of the present invention;

[0046] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0047] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0049] The two-phase commit protocol of distributed transactions includes: voting phase and commit phase. In the voting phase, the coordinator sends a request to the transaction participants to perform operations and waits for responses from other participants. The participants will perform the corresponding transaction operations and return the results of the operations to the coordinator. When all participants return a confirmed result, the two-phase commit enters the commit phase. The confirmed results include approval or termination.

[0050] In the commit phase, the coordinator will send COMMIT or ABORT instructions to all participants according to the return results of the voting phase. When all participants in the transaction decide to COMMIT the transaction, the coordinator will send a COMMIT request to the participants. After completing the operation and releasing resources, the participants will return a completion message to the coordinator. When the coordinator receives the completion messages from all participants, it will end the entire transaction. On the contrary, when all participants decide to ABORT the current transaction, the coordinator will send an ABORT request to the participants of the transaction, and the participants will roll back and send a completion message to the coordinator.

[0051] The above two-phase submission is aimed at the common distributed scenario of MPP architecture. MPP architecture distributes tasks in parallel to multiple servers and nodes. After the calculation is completed on each node, the results of each part are aggregated to obtain the final result. The database using MPP architecture is called MPP database.

[0052] For cloud-native scenarios, the efficiency of processing transactions is low. Cloud-native scenarios refer to scenarios that rely on the basic services provided by the cloud platform. For example, scenarios where software runs on virtual machine services, cloud storage services, and cloud file services. Cloud-native scenarios are different from scenarios where traditional software is deployed on physical machines and relies on the hardware of physical machines to run. Cloud-native scenarios can also be called cloud scenarios. The characteristics of cloud scenarios are: computing nodes can be easily expanded, and the cluster size can be very large.

[0053] In the execution process of the above two-phase commit, there are the following specific disadvantages when applied to cloud scenarios:

[0054] All nodes involved in the transaction operation are in a blocked state. While waiting for responses from other participants, each participant will be unable to perform other task operations, resulting in very low performance.

[0055] In the two-stage process, a coordinator role is introduced. The coordinator role plays a very important role in the entire two-stage commit protocol. Once the coordinator fails, the second stage will not be able to operate. More seriously, if the coordinator fails in the second stage, other participants will be in a state of locking transaction resources and will not be able to continue to complete the transaction operation.

[0056] If a network failure or a failure of some participants occurs in the first phase, the coordinator will ABORT the transaction, reducing the probability of transaction success. This is especially true when the cluster is large.

[0057] If in the second phase, after the coordinator sends a commit request to the participants, a network anomaly occurs or the coordinator fails during the process of sending the commit request, this will result in only some of the participants receiving the commit request. After receiving the commit request, these participants will execute the commit operation. However, other machines that have not received the commit request cannot execute the transaction commit. As a result, the entire distributed system will experience data inconsistency.

[0058] In summary, for cloud scenarios that rely on basic services provided by cloud platforms, the efficiency of processing transactions is low.

[0059] In order to solve the problem of low efficiency in processing transactions in a cloud scenario, the following technical solution in the embodiments of the present invention may be adopted.

[0060] See also Figure 1 , Figure 1 1 is a schematic diagram of the main flow of the method for database transaction processing according to an embodiment of the present invention, in which a database transaction is submitted on a cloud platform through distributed nodes. Figure 1 As shown, the specific steps include:

[0061] S101. Record the data file name of the database transaction into a recovery file in the temporary directory of the distributed node.

[0062] The technical solution in the embodiment of the present invention is mainly applied to a cloud platform. The cloud platform provides cloud file services.

[0063] See also Figure 2 , Figure 2 Schematic diagram of a distributed database architecture according to an embodiment of the present invention. Figure 2 It includes two clients and multiple computing nodes. The clients enjoy cloud file services through the server nodes.

[0064] The cloud platform provides cloud file services, which belong to shared storage. On shared storage, when a computing node writes a file, other settlement nodes can see it in real time. The file is shared with all computing nodes that use the storage, so it is also called shared storage.

[0065] Clients can communicate with any computing node. A computing node is a computing unit of a cloud service, such as a virtual machine or container. A computing node performs calculations and accesses underlying storage data, and then provides query and other services to upper-level clients. A computing node can distribute computing tasks to other computing nodes to accelerate computing. Storage is specifically a storage unit of a cloud service. Storage units usually provide storage services for cloud file services or objects.

[0066] The solution in the embodiment of the present invention is to implement distributed transactions based on the functions provided by the cloud file service. The following describes the four basic characteristics of the cloud file service:

[0067] Feature 1: The cloud file service uses triple-copy redundant storage to provide super stability and reliability.

[0068] Feature 2: Supports standard NFSv4.0 and NFSv4.1 protocols, provides fully managed services, and can be implemented through standard file system mounting steps without modifying applications.

[0069] Feature 3: Multiple cloud hosts can simultaneously access the file storage created in the cloud file service through the NFS protocol, perform read and write operations on the files, and realize data sharing among multiple computing nodes.

[0070] Feature 4: All file operations are atomic, such as moving files, creating files, modifying file names, etc.

[0071] Because shared storage is used, all distributed nodes have the same data, and transactions do not need to be submitted in two stages. When each node commits, it records the submission information to the shared storage to ensure that other nodes have consistent data. By taking advantage of the atomicity of shared storage file operations, each node can have the same transaction status. It only needs to create a specific file to indicate transaction submission when submitting, so that all nodes can confirm transaction submission.

[0072] See also Figure 3 , Figure 3 1 is a schematic diagram of a framework for submitting a database transaction according to an embodiment of the present invention. In a shared storage, the framework for submitting a database transaction involves multiple directories and files.

[0073] from Figure 3 As can be seen from the figure, a table has multiple files under it. The following describes the files under the table respectively.

[0074] The N suffix indicates that there may be multiple files or directories. Lock files are used to perform locking operations, such as spin locks. The locking operation on the cloud file service is a distributed lock. Once a node is locked, other nodes cannot be locked.

[0075] Recover files are used for recovery, similar to the functions of redo logs and undo logs. Data files are files that actually store data on behalf of objects. Visibility files are mainly used to implement multi-version concurrency control technology (MVCC). The creation transaction number and deletion transaction number of the data file are recorded in the visibility file name in the format of Create_Drop.

[0076] The transaction sequence number file is a file generated when the transaction is committed. The file name is a monotonically increasing sequence (1, 2, 3...). When the file is generated successfully, it means that the transaction is successfully committed. The reference count file refers to the hard link of the transaction sequence number file currently being accessed, which is used to confirm whether there is a query currently accessing the data file generated by this transaction. Temporary data files are data files generated during the transaction and are not available before submission.

[0077] In an embodiment of the present invention, in order to process database transactions on a cloud platform, the data file name of the database transaction is recorded in a recovery file in a temporary directory of a distributed node.

[0078] from Figure 3 The NodeN_tmp directory in the above is the temporary directory of the distributed node. Figure 3 Recover file in the NodeN_tmp directory.

[0079] The recovery file in the temporary directory is similar to the functions of redolog and undolog and is used for recovery. The above writing of temporary files has two purposes.

[0080] Purpose 1: The write operation is not an atomic operation. The rename or move operation after writing is an atomic operation to ensure that the data file will not be damaged due to downtime.

[0081] Purpose 2: Writing file content takes a long time, which is not conducive to the submission stage. However, renaming is very fast. In cloud file services, moving operations are equivalent to renaming operations.

[0082] In one embodiment of the present invention, the database transaction includes an update transaction. In order not to affect the read operation, a temporary data file is newly created in the distributed node, and the update transaction is executed in the temporary data file to update the database transaction.

[0083] That is to say, instead of performing a local update, the data file to be modified is first copied to a new temporary file data, and then the temporary file data is modified. In this way, reading and writing are not the same file, and there will be no reading and writing conflicts.

[0084] In one embodiment of the present invention, the database transaction includes a non-update transaction. The data file of the database transaction is stored in a data file in a temporary directory of the distributed node.

[0085] That is, if the database transaction is not an update transaction, the data file of the database transaction is written to the data file in the temporary directory of each distributed node.

[0086] In the above steps, the data files and data file names of the database transactions are recorded in the temporary directory of the distributed nodes, laying the foundation for storing data in the cloud platform.

[0087] S102. When the recovery file does not exist in the cloud platform, the data file is moved from the temporary data directory to the data file directory of the cloud platform, and the recovery file in the temporary directory is moved to the data file directory of the cloud platform to submit the database transaction.

[0088] To ensure the atomicity of transactions, all rolled back or abnormally interrupted transactions must restore the modified data to the state before the transaction started. If an error occurs during the execution of a transaction, or constraints are not satisfied, and a rollback is required, all operations can be rolled back directly based on the information recorded in the memory. The purpose of the recovery file is to restore data.

[0089] In one embodiment of the present invention, when the recovery file does not exist in the cloud platform, it means that the last submitted database transaction was successful and no recovery is required. The recovery file submitted this time is directly moved from the temporary directory of the distributed node to the data file directory of the cloud platform. Then, the data file of the database transaction is moved from the temporary directory of the distributed node to the data file directory of the cloud platform.

[0090] In one embodiment of the present invention, when there is a recovery file in the cloud platform, it means that the last submitted database transaction failed and needs to be rolled back. The rollback operation includes deleting the visibility file of the data file in the cloud platform, deleting the newly added file of the data file in the cloud platform, and updating the visibility file of the cloud platform.

[0091] Specifically, delete the visibility file of the data file of the database transaction, and then delete the newly added file of the data file in the cloud platform. If the newly added file of the data file in the cloud platform does not exist, no processing is required. This part of the deletion operation can be repeated, that is, if an exception occurs in this process, it can be redone without additional processing.

[0092] In one embodiment of the present invention, in order to improve efficiency, the newly added files of the visibility files and data files are deleted. In specific implementation, the above files can be moved to a temporary directory and cleaned up by the background. The specific deletion scheme is as follows.

[0093] See also Figure 4 , Figure 4 This is a schematic diagram of the background cleaning process according to an embodiment of the present invention. Specifically, it includes the following steps:

[0094] S401, traverse the directory of the shared storage of the cloud platform, and determine the cleaning sequence number according to the reference count of the data file, where the reference count is used to record the number of times the data file is called.

[0095] In the embodiment of the present invention, in order to speed up the process of handling database transactions, the deletion operation is performed in the background. It can be understood that S401 and S402 are performed in the background. Specifically, if the deleted file is not found to be accessed, it will be cleaned up by the background.

[0096] Visibility files with transaction sequence numbers other than 0 in the visibility directory of shared storage in the cloud platform are usually not queried to be accessed, so the above visibility files are cleaned up by the background.

[0097] In the embodiment of the present invention, when querying the transaction sequence number, a hard link of the transaction sequence number file is established, which is called a reference count. The reference count is set in the temporary directory of the corresponding distributed node. Each time a data file is created, the reference count of the transaction sequence number increases by one. The reference count is used to record the number of times the data file is called. The smaller the reference count, the less frequently the file is used; the more the reference count, the more frequently the file is used.

[0098] In order to clean up files, it is necessary to determine the cleanup sequence number. Specifically, multiple transaction sequence number files are stored in the TIDS directory. Traverse the directories in the cloud platform, such as the TIDS directory, and use the largest transaction sequence number with a reference count greater than 1 as the cleanup identifier.

[0099] Specifically, traverse the TIDS directory in the cloud platform to find files with a reference count greater than 1. If there is only one transaction sequence number file, it means that there is no file that needs to be cleaned up; traverse the TIDS directory in the cloud platform to find files with a reference count greater than 1. If the reference counts of all transaction sequence number files are 1, the cleanup identifier is the maximum value among them; if the reference counts of all transaction sequence number files are not 1, the cleanup identifier is the first transaction sequence number whose reference count is not 1.

[0100] S402: Delete data files whose transaction numbers are less than or equal to the cleanup identifier.

[0101] Traverse all data files in the cloud platform and delete files with transaction numbers less than or equal to the cleanup identifier. Among them, all files with transaction numbers less than or equal to the cleanup identifier in all visibility files are files that can be deleted.

[0102] The cleanup identifier and previous transaction files can be deleted because no one has accessed them and they will not be accessed in the future. Finally, delete all files corresponding to transaction numbers less than the cleanup identifier.

[0103] In the embodiment of the present invention, the cleaning operation includes querying files and deleting files. The querying and deleting are reentrant, that is, if the target file is not found during deletion, it can be ignored.

[0104] exist Figure 4 In the embodiment, in order to increase the speed of processing database transactions, the operation of deleting files in the cloud platform can be implemented in the background.

[0105] S103. Based on the transaction sequence number of the database transaction, the symbolic link name of the data file in the visibility file is updated. The visibility file is used for concurrent control of distributed nodes.

[0106] The transaction sequence number of a database transaction is used to identify the database transaction. As an example, the transaction sequence number of a database transaction is the current maximum transaction sequence number of the database transaction plus 1. The transaction sequence number of a database transaction can be determined in the following manner.

[0107] Get the transaction number (ID) of the current database transaction, that is, traverse the TIDS directory, get the file name with the largest transaction number, and then add one to get the transaction ID of the current database transaction: X.

[0108] See also Figure 5 , Figure 5 The following is a flow chart of updating the symbolic link name in the visibility file according to an embodiment of the present invention. Specifically, the following steps are included:

[0109] S501. Associate the original symbolic link file in the data file with the data file corresponding to the transaction sequence number of the database transaction.

[0110] The symbolic link name of the data file in the visibility file in the cloud platform points to the data file of the database transaction. In order to obtain the above data file, it is necessary to update the symbolic link name of the data file in the visibility file based on the transaction sequence number of the database transaction.

[0111] Specifically, a symbolic link file with the transaction sequence number of a database transaction is created in the visibility directory, the data file of the database transaction is associated, and the symbolic link name of the original data file is modified.

[0112] As an example, the transaction sequence number of the database transaction is: X. Create a symbolic link file X_0 in the visibility directory to point to the data file of the database transaction, and modify the symbolic link name of the original data file from create_0 to create_X.

[0113] S502: In the visibility file, update the symbolic link name of the data file with the transaction sequence number of the database transaction.

[0114] In the visibility file, the symbolic link name of the data file is updated with the transaction sequence number of the database transaction. Then, the data file can be obtained based on the symbolic link name.

[0115] exist Figure 5 In the embodiment, the data file can be obtained based on the updated symbolic link name.

[0116] S104: Move the recovery file of the cloud platform to a directory of the shared storage of the cloud platform to confirm that the database transaction is successfully submitted.

[0117] In order to confirm that the database transaction is successfully submitted, the recovery file of the cloud platform needs to be moved to the shared storage directory of the cloud platform, such as the TIDS directory. Once the above move operation is successful, it means that the database transaction is successfully submitted. When querying data files, the latest version of the data file is determined by traversing the TIDS directory.

[0118] In the embodiment of the present invention, once the data file is named successfully, all data is actually stored on the cloud file server, the data is persistent, and there is no need to recover through redo logs.

[0119] It should be noted that in the process of reading data files, cache can be used to avoid reading data files from the cloud file service every time.

[0120] In the embodiment of the present invention, the disk data is stored on the shared storage, and the data seen by all distributed nodes is consistent, that is, the data read by each distributed node is the same, so as to ensure data consistency.

[0121] When a distributed node reads data, it first obtains the read number from the TIDS directory of the cloud platform, that is, it traverses the TIDS directory and obtains the largest transaction number. Then it reads the data file that can be read with the largest transaction number.

[0122] The judgment rule is: the file creation version number <= read number < file deletion number.

[0123] If database transaction X fails to commit, then there must be no file X in the TIDS directory, and the query transaction number can only be X-1. Then the file visibility file committed in database transaction X is X_0, or A_X. According to the above judgment rules, even if these files are not rolled back, they will not affect the reading, because the query itself will not read the file corresponding to X_0, but will read the A_X file. The effect is the same as rolling back transaction X.

[0124] That is to say, even for distributed queries, you only need to pass the number X obtained on the master node to other distributed nodes for query, and the queried data will be consistent.

[0125] In one embodiment of the present invention, in order to avoid concurrent operations of multiple distributed nodes, a locking operation may be performed before submitting a database transaction; and an unlocking operation may be performed after the database transaction is successfully submitted.

[0126] That is, first perform the lock operation, then submit the database transaction; after confirming that the database transaction is successfully submitted, perform the unlock operation. As an example, you can use external programs such as zookeeper to implement distributed locks.

[0127] The SQL standard defines four types of isolation levels, including some specific rules to limit which changes in transactions are visible and which are invisible. Low-level isolation levels generally support higher concurrency and have lower system overhead. They include read uncommitted, read committed, repeatable read, and serializable.

[0128] In the embodiment of the present invention, the read committed isolation level is supported. The read transaction does not conflict with other transactions at all, and it is a completely lock-free read. Insert transactions and update transactions will also conflict only at the commit node, but the locking process in the commit database transaction stage is very short, so the concurrency is very high.

[0129] Update transactions cannot be performed at the same time. Isolation is ensured by distributed locks, which requires external guarantees or distributed locks to be implemented on shared storage.

[0130] In the embodiment of the present invention, in order to ensure the atomicity of database transactions, all transactions that are rolled back or abnormally interrupted must restore the modified data to the state before the transaction started. If an error is found during the execution of a transaction, or a constraint is not satisfied, etc., and a rollback is required, all operations can be rolled back directly according to the information recorded in the memory. The main focus is on transaction rollbacks caused by downtime during the processing of database transactions.

[0131] The main abnormal situations include the following three:

[0132] Abnormal situation 1:

[0133] From the beginning of processing the database transaction to moving the recovery file in the temporary directory to the recovery file of the cloud platform. At this time, there is no recovery file in the cloud platform, and all files that need to be rolled back are in the temporary directory.

[0134] Therefore, the rollback operation is very simple, which is to clear the temporary directory of the corresponding distributed node. The specific implementation of this part can be divided into two parts. The first part is the node restart phase. When the distributed node is restarted, the corresponding temporary directory is cleared. The second part is handled by other distributed nodes in the distributed system. If other distributed nodes find that there is a distributed node failure, they will clear the temporary directory of the corresponding distributed node.

[0135] Abnormal situation 2:

[0136] After the recovery file is moved to the recovery file of the cloud platform, it is moved to a preset identification file. As an example, if the preset identification is X, it is moved to X file. At this stage, there is a recovery file, and the rollback operation is handed over to the next database transaction execution and submission process. If the next database transaction is not executed, there is no need to roll back, because it does not affect reading and consistency.

[0137] Abnormal situation 3:

[0138] The recovery file on the cloud platform is moved to the data file until the transaction ends. At this time, there is no recovery file and the transaction is considered committed without rollback.

[0139] In the above embodiment, the data file name of the database transaction is recorded in the recovery file of the temporary directory of the distributed node; when the recovery file does not exist in the cloud platform, the data file of the database transaction is moved from the temporary directory to the data file directory of the cloud platform, and the recovery file of the temporary directory is moved to the data file directory of the cloud platform to submit the database transaction; based on the transaction sequence number of the database transaction, the symbolic link name of the data file in the visibility file is updated, and the visibility file is used for concurrent control of distributed nodes; the recovery file of the cloud platform is moved to the directory of the shared storage of the cloud platform to determine the successful submission of the database transaction. The distributed node uploads the data file of the database transaction to the cloud platform to process the database transaction, and other distributed nodes can obtain the above data file, thereby improving the efficiency of database transaction processing in the cloud scenario.

[0140] See also Figure 6 , Figure 6 is a schematic diagram of the main structure of a database transaction processing device according to an embodiment of the present invention. The database transaction processing device can implement a database transaction processing method, such as Figure 6 As shown, the device for database transaction processing specifically includes:

[0141] The recording module 601 is used to record the data file name of the database transaction into the recovery file of the temporary directory of the distributed node;

[0142] A submission module 602 is used to move the data file of the database transaction from the temporary data directory to the data file directory of the cloud platform, and to move the recovery file of the temporary directory to the data file directory of the cloud platform, so as to submit the database transaction when the recovery file does not exist in the cloud platform;

[0143] An update module 603, used to update the symbolic link name of the data file in the visibility file based on the transaction sequence number of the database transaction, wherein the visibility file is used for concurrency control of distributed nodes;

[0144] The moving module 604 is used to move the recovery file of the cloud platform to the directory of the shared storage of the cloud platform to determine the successful submission of the database transaction.

[0145] In one embodiment of the present invention, the database transaction includes an update transaction;

[0146] The recording module 601 is further used to create a temporary data file in the distributed node, and execute the update transaction in the temporary data file to update the database transaction.

[0147] In one embodiment of the present invention, the database transaction includes a non-update transaction;

[0148] The recording module 601 is also used to store the data file of the database transaction into the data file in the temporary directory.

[0149] In one embodiment of the present invention, when there is a recovery file in the cloud platform;

[0150] The submission module 602 is also used to delete the visibility file of the data file in the cloud platform, then upload the new file of the data file in the cloud platform, and update the visibility file of the cloud platform.

[0151] In one embodiment of the present invention, the update module 603 is further used to traverse the directory of the shared storage of the cloud platform, and determine the cleaning sequence number according to the reference count of the data file, wherein the reference count is used to record the number of times the data file is called;

[0152] Delete data files whose transaction IDs are less than or equal to the cleanup sequence number.

[0153] In one embodiment of the present invention, the update module 603 is further used to associate the newly created symbolic link file in the data file with the data file corresponding to the transaction sequence number of the database transaction;

[0154] In the visibility file, the newly created symbolic link name of the data file is updated with the transaction sequence number of the database transaction.

[0155] In one embodiment of the present invention, the commit module 602 is further configured to, before committing the database transaction, further include: performing a locking operation.

[0156] The moving module 604 is also used to perform an unlocking operation.

[0157] Figure 7 An exemplary system architecture 700 is shown to which the method or apparatus for database transaction processing according to the embodiment of the present invention may be applied.

[0158] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0159] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0160] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0161] The server 705 may be a server that provides various services, such as a backend management server (only for example) that supports shopping websites browsed by users using the terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only for example) to the terminal device.

[0162] It should be noted that the method for database transaction processing provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the device for database transaction processing is generally set in the server 705 .

[0163] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0164] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0165] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0166] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0167] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0168] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0169] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a recording module, a submission module, an update module and a move module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the recording module may also be described as "a recovery file for recording the data file name of the database transaction in the temporary directory of the distributed node".

[0171] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0172] Record the data file name of the database transaction in the recovery file of the temporary directory of the distributed node;

[0173] When the recovery file does not exist in the cloud platform, the data file of the database transaction is moved from the temporary directory to the data file directory of the cloud platform, and the recovery file of the temporary directory is moved to the data file directory of the cloud platform to submit the database transaction;

[0174] Based on the transaction sequence number of the database transaction, updating the symbolic link name of the data file in the visibility file, the visibility file being used for concurrency control of distributed nodes;

[0175] The recovery file of the cloud platform is moved to a directory of the cloud platform shared storage to confirm successful submission of the database transaction.

[0176] According to the technical solution of the embodiment of the present invention, the data file name of the database transaction is recorded in the recovery file of the temporary directory of the distributed node; when the recovery file does not exist in the cloud platform, the data file of the database transaction is moved from the temporary directory to the data file directory of the cloud platform, and the recovery file of the temporary directory is moved to the data file directory of the cloud platform to submit the database transaction; based on the transaction sequence number of the database transaction, the symbolic link name of the data file in the visibility file is updated, and the visibility file is used for the concurrency control of the distributed nodes; the recovery file of the cloud platform is moved to the directory of the shared storage of the cloud platform to determine the successful submission of the database transaction. The distributed node uploads the data file of the database transaction to the cloud platform to process the database transaction, and other distributed nodes can obtain the above data files, thereby improving the efficiency of database transaction processing in the cloud scenario.

[0177] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for database transaction processing, characterized in that: include: Record the data file name of the database transaction in the recovery file of the temporary directory of the distributed node; When the recovery file does not exist in the cloud platform, the data file of the database transaction is moved from the temporary directory to the data file directory of the cloud platform, and the recovery file of the temporary directory is moved to the data file directory of the cloud platform to submit the database transaction; Based on the transaction sequence number of the database transaction, updating the symbolic link name of the data file in the visibility file, the visibility file being used for concurrency control of distributed nodes; Moving the recovery file of the cloud platform to a directory of the shared storage of the cloud platform to determine successful submission of the database transaction; The updating of the symbolic link name of the data file in the visibility file based on the transaction sequence number of the database transaction includes: Associating the newly created symbolic link file in the data file with the data file corresponding to the transaction sequence number of the database transaction; In the visibility file, the newly created symbolic link name of the data file is updated with the transaction sequence number of the database transaction.

2. The method for database transaction processing according to claim 1, characterized in that: The database transaction includes an update transaction; Before recording the data file name of the database transaction into the recovery file of the temporary directory of the distributed node, the method further includes: A temporary data file is newly created in the distributed node, and the update transaction is executed in the temporary data file to update the database transaction.

3. The method for database transaction processing according to claim 1, characterized in that: The database transaction includes a non-update transaction; Before recording the data file name of the database transaction into the recovery file of the temporary directory of the distributed node, it also includes The data file of the database transaction is stored in the data file in the temporary directory.

4. The method for database transaction processing according to claim 1, characterized in that: When there is a recovery file in the cloud platform; The method further comprises: After deleting the visibility file of the data file in the cloud platform, the newly added file of the data file in the cloud platform is deleted, and the visibility file of the cloud platform is updated.

5. The method for database transaction processing according to claim 1, characterized in that: The method further comprises: Traverse the directory of the shared storage of the cloud platform and determine the cleaning sequence number according to the reference count of the data file, where the reference count is used to record the number of times the data file is called; Delete data files whose transaction sequence numbers are less than or equal to the cleanup sequence number.

6. The method for database transaction processing according to claim 1, characterized in that: Before submitting the database transaction, the method further includes: performing a locking operation; After determining that the database transaction is successfully submitted, the method further includes: performing an unlocking operation.

7. A device for database transaction processing, characterized in that: include: A recording module is used to record the data file name of the database transaction into the recovery file of the temporary directory of the distributed node; A submission module, configured to move the data file of the database transaction from the temporary directory to the data file directory of the cloud platform, and to move the recovery file of the temporary directory to the data file directory of the cloud platform, so as to submit the database transaction when the recovery file does not exist in the cloud platform; An update module, configured to update the symbolic link name of the data file in the visibility file based on the transaction sequence number of the database transaction, the visibility file being used for concurrent control of distributed nodes; and to associate the newly created symbolic link file in the data file with the data file corresponding to the transaction sequence number of the database transaction; in the visibility file, update the newly created symbolic link name of the data file with the transaction sequence number of the database transaction; The moving module is used to move the recovery file of the cloud platform to a directory of the shared storage of the cloud platform to determine that the database transaction is successfully submitted.

8. An electronic device for database transaction processing, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images