File metadata optimization method and system based on 2PC transaction, equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a file metadata optimization method and system based on a 2PC transaction, equipment and a storage medium, and the system comprises a plurality of participants and a coordinator; multiple single-line transactions of the same file are optimized into one transaction; submission of multiple single-line transactions of the same file is optimized into one-time submission; and aggregating a plurality of transaction logs, and submitting the transaction logs together. Therefore, the performance of the file metadata is improved, and the write amplification problem of the transaction is reduced, so that the performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, for example to a file metadata optimization method and system based on a 2PC transaction, a device, and a storage medium. BACKGROUND

[0002] The establishment of the two-phase commit transaction is based on the following assumptions: in the distributed system, there is one node as a coordinator and other nodes as participants. Network communication can be carried out between the nodes. All nodes use a pre-write log, and the log is written and then kept on a reliable storage device, so that even if the node is damaged, the log data will not be lost. All nodes will not be permanently damaged, and can still be restored even after damage.

[0003] First phase - commit request phase: The coordinator node asks all participant nodes whether the commit operation can be performed, and starts to wait for the responses of the participant nodes; the participant nodes perform all transaction operations initiated by the inquiry, and write Undo information and Redo information into the log; each participant node responds to the inquiry initiated by the coordinator node. If the transaction operation of the participant node is actually successfully executed, a "yes" message is returned; if the transaction operation of the participant node is actually executed unsuccessfully, an "abort" message is returned Second phase - commit execution phase: The coordinator node sends a "formal commit" request to all participant nodes; the participant nodes formally complete the operation and release the resources occupied during the entire transaction; the participant nodes send a "complete" message to the coordinator node; the coordinator node completes the transaction after receiving the "complete" messages fed back by all participant nodes.

[0004] The existing technologies are all improvements on the above two basic implementation methods. A Chinese patent with application number 202311765490X discloses an improved two-phase commit transaction implementation method and system, device, and storage medium. The system includes multiple participants and one coordinator. In the preparation phase of the two-phase commit, the coordinator of the transaction initiates a lock request to all participants. After receiving the lock request, the participant will check whether the metadata it holds meets the transaction condition, and if it does, it will lock the metadata it holds and return the metadata to the coordinator. The coordinator updates the metadata of the participant in the memory and writes the updated metadata into the unique transaction log. The submission phase work of the transaction is executed asynchronously in the background, and the participant does not need to write a local transaction. In the preparation phase, as long as the coordinator successfully persists the transaction log, it means that the metadata update is successful. Therefore, the existing technology can greatly reduce the read-write times of the database in a distributed multi-row transaction, and reduce the transaction delay.

[0005] Although the above system and method can greatly reduce the read and write times of the database, reduce the transaction delay, there is still a problem of low running efficiency, mainly including: the coordinator waits until all participants vote before making a decision, which is time-consuming and affects performance; all nodes of the transaction use the pre-write log method, which causes the write amplification problem; all nodes of the transaction use the pre-write log method, which causes the write amplification problem. SUMMARY

[0006] The following presents a simplified summary of some aspects of the disclosed embodiments in order to provide a basic understanding of such embodiments. The summary is not an extensive overview of the disclosure and is not intended to identify key / critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some aspects of the disclosed embodiments in a simplified form as a prelude to the more detailed description that is presented later.

[0007] The embodiments of the present disclosure provide a 2PC transaction-based file metadata optimization method and system, device, and storage medium, which can greatly improve the running efficiency in distributed multi-line transactions.

[0008] In some embodiments, a 2PC transaction-based file metadata optimization method is provided, which is applied to a coordinator of a 2PC transaction-based file metadata optimization system, the system including a plurality of participants and a coordinator, and the method including: Preparation stage: When the front transaction is identified to be completed, all participants in the task-group are locked; After the locking is successful, the transaction log is persisted to the database.

[0009] All participants send an unlock request; Commit stage: Send a commit readiness judgment signal to all participants; After receiving the commit readiness signal from all participants, the task state is converted to a ready state; Merge multiple ready tasks into a unique task-group; Persist multiple records in the task-group to the database in a batch processing manner; Set the commit task state to completed, and notify all participants to remove the task from the commit task queue.

[0010] Preferably, persisting multiple records in the task-group to the database in a batch processing manner includes: After the number of tasks in the task-group reaches the upper limit or the TTL expires, the task-group is pushed to the group-commit queue, and the commit manager takes a task-group from the group-commit queue to perform batch operations on the kv database.

[0011] Preferably, the batch operation includes: converting a plurality of set and del operations into set-batch and delete-batch operations.

[0012] Preferably, before all the participants in the task-group are locked, the method further includes: Obtaining an executing pre-task; Obtaining a new request, checking whether there is a task-group, if there is a task-group, adding the request to the task-group, if there is no task-group, creating a task-group and adding the request to the task-group; Checking whether the pre-transaction is completed.

[0013] In some embodiments, a file metadata optimization method based on 2PC transaction is provided, which is applied to a participant in a file metadata optimization system based on 2PC transaction, the system including a plurality of participants and a coordinator, and the method including: Preparation stage: Receiving a lock request and performing locking; In the case of successful locking, sending a lock completion signal; Receiving an unlock request and performing unlocking; Commit stage: Receiving a commit readiness judgment signal and detecting whether all tasks in the commit task queue of the participant are completed; After completion, sending a commit readiness signal to the coordinator.

[0014] In some embodiments, a device is disclosed, including a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the program, the method for applying a participant in a file metadata optimization system based on 2PC transaction is executed.

[0015] In some embodiments, a computer readable storage medium is disclosed, which stores a computer program, and the program is executed by a processor to apply a participant in a file metadata optimization system based on 2PC transaction to a file metadata optimization method based on 2PC transaction.

[0016] In some embodiments, a device is disclosed, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the program, a 2PC transaction based file metadata optimization method applied to a coordinator in a 2PC transaction based file metadata optimization system is executed.

[0017] The 2PC transaction based file metadata optimization method and system, device, and storage medium provided by the embodiments of the present disclosure optimize the process of two-phase commit, optimize multiple single-row transactions of the same file into one transaction, optimize the commit of multiple single-row transactions of the same file into one commit, and aggregate multiple transaction logs and commit together. The embodiments of the present disclosure can achieve the following technical effects: The performance of file metadata is greatly improved, and the performance of metadata can be improved by more than 3 to 5 times The write amplification problem of transactions is greatly reduced, and the amplification coefficient is reduced from 3 to 4 to less than 1; The cpu load of the metadata server is greatly reduced, and the cpu of a single server can be reduced by more than 200%; The persistence load and cpu load of the database are greatly reduced; The network IO overhead of the database is greatly reduced.

[0018] The foregoing general description and the following description are only exemplary and explanatory, and are not used to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] One or more embodiments are exemplarily illustrated by corresponding drawings, which do not constitute a limitation on the embodiments, elements with the same reference numerals in the drawings are considered as similar elements, the drawings do not constitute a proportional limitation, and wherein: Figure 1 is a schematic diagram of an improved two-phase commit transaction implementation method provided by the embodiments of the present disclosure; Figure 2 is a schematic diagram of a 2PC transaction based file metadata optimization system provided by the embodiments of the present disclosure; Figure 3 is a schematic diagram of an architecture description provided by the embodiments of the present disclosure; Figure 4 is a schematic diagram of a first phase and second phase task chain representation provided by the embodiments of the present disclosure; Figure 5 is a schematic diagram of a 2PC transaction based file metadata optimization method provided by the embodiments of the present disclosure; Figure 6is a file metadata optimization method flowchart of a preparation phase provided by an embodiment of the present disclosure; Figure 7 is a commit processing flowchart after optimization provided by an embodiment of the present disclosure; Figure 8 is an interaction flowchart of a coordinator and a commit manager provided by an embodiment of the present disclosure; Figure 9 is a schematic diagram of optimizing a single transaction provided by an embodiment of the present disclosure; Figure 10 is a commit schematic diagram of optimizing a single transaction provided by an embodiment of the present disclosure; Figure 11 is a commit schematic diagram of aggregating multiple transaction logs provided by an embodiment of the present disclosure; Figure 12 is a schematic diagram of merging different transactions of a same partition into a transaction log provided by an embodiment of the present disclosure; Figure 13 is a schematic diagram of optimizing a partial pessimistic lock to an optimistic lock provided by an embodiment of the present disclosure; Figure 14 is a schematic diagram of a data structure required by an optimistic lock implementation provided by an embodiment of the present disclosure; Figure 15 is a preparation phase flowchart initiated by a transaction coordinator provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to enable a person skilled in the art to more fully understand the features and technical contents of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are used only for reference and do not limit the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, in order to simplify the drawings, well-known structures and systems can be simplified.

[0021] The following description and drawings are illustrative of specific embodiments of the application and are not intended to be limiting of the application. Other embodiments can include modifications and alterations of the specific embodiments. Specific embodiments are presented in terms of methods although methods have a wide range of applications. Various methods can have some similarity to others, and some methods can be more complex than others. Some reagents can be used in a similar manner, and therefore their description in the context of one method can also be applied to another where appropriate. Terms such as "element" or "component" encompass both reagents and hardware, unless otherwise indicated. Several embodiments of methods can be performed in conjunction with one another and can be mutually independent of one another. The steps of a method can be performed in any suitable order, and are not limited to the described orders. Several embodiments of hardware can be employed in conjunction with one another; components of one embodiment are interchangeable with other embodiments; and the steps of a method can be employed in conjunction with either of the hardware embodiments. The various embodiments can be implemented in hardware, software, or a combination thereof. The various embodiments can be implemented in one or more computer systems or other processing systems.

[0022] An improved two-phase commit transaction implementation method is disclosed in the prior art. See Figure 1 The improved two-phase commit transaction implementation method in the prior art is shown in the figure. The method includes: a preparation phase: The coordinator node asks all participant nodes whether the transaction can be executed, waits for the latest metadata of each participant node; after the coordinator receives the "agree" message of all participants, it persists a transaction log to the database; after the coordinator notifies all participants of the success of the transaction commit, the participants are unlocked.

[0023] A commit phase: The coordinator inquires all participants whether the transaction can be committed, waits for the state reports of the participants, and after receiving the 'agree' messages of all participants, updates the metadata of all participants to the database and deletes the transaction log. The coordinator reports the completion of the commit to all participants.

[0024] To solve the problems in the related art, the embodiments of the present disclosure disclose a file metadata optimization system based on 2PC transaction, as shown in Figure 2 The system comprises a plurality of participants and a coordinator.

[0025] The architecture in the embodiments of the present disclosure is described. The metadata service is divided into two layers in architecture. Database layer: responsible for the storage of metadata, distributed database tikv Metadata agent layer: provides an interface of file semantics to the outside, and converts the data of the file system namespace into records in the tikv database to the inside, and uses transactions to ensure the correctness of operations. As shown in Figure 3 It is a schematic diagram of an architecture.

[0026] When executing a transaction, as shown in Figure 4 It is a schematic diagram of the first and second phase task chains corresponding to the transaction. Each metadata has two task chain tables: Task chain table of the first phase of the transaction: Tasks(2pc-1) Task chain table of the second phase of the transaction: Tasks(2pc-2) It should be noted that Tasks(2pc-1): the grayed tasks represent the write transactions being executed, and the others are tasks waiting to be executed, and the tasks are ordered. Tasks(2pc-2): the grayed tasks represent the transactions being committed, and the others are tasks waiting to be committed. Figure 4 The Tasks(2pc-2) of the metadata C and D in the middle have no tasks, indicating that the commit operation has not been executed.

[0027] As shown in Figure 5 The embodiments of the present disclosure provide a file metadata optimization method based on 2PC transaction, which comprises the following steps: Preparation phase: S101, the coordinator obtains the pre-executed task.

[0028] S102, the coordinator obtains a new request, checks whether there is a task-group, if there is a task-group, adds the request to the task-group, and if there is no task-group, creates a task-group and adds the request to the task-group.

[0029] S103, the coordinator checks whether the precondition transaction is completed.

[0030] S104, the coordinator identifies the completion signal, and locks all participants in the task-group.

[0031] S105, the participant receives the lock request, and performs the lock.

[0032] S106, the participant sends the lock completion signal in case of successful lock.

[0033] S108, the coordinator successfully locks, and persists the transaction log in the database.

[0034] S109, the coordinator sends the unlock request to all participants.

[0035] S110, the participant receives the unlock request, and performs the unlock.

[0036] It should be understood that the write and setattr requests are submitted to the transaction coordinator. The transaction coordinator checks whether there is any precondition transaction not completed by the transaction participant. If not, the pre-optimization scheme is executed. If yes, the following scheme is executed. The coordinator creates a new task-group, and adds the task to the task-group. The subsequent continuous requests (write, setattr) are added to the task-group. The coordinator checks whether the precondition transaction is completed by the participant. If yes, the coordinator performs the lock process on the participant. After the lock is successful, the coordinator persists the transaction log in the database. The coordinator unlocks the participant, and returns the request result. See Figure 6 A schematic diagram of a file metadata optimization method in a preparation stage.

[0037] In the embodiments of the present disclosure, new data structures are introduced: task-group: a plurality of ready commit tasks are aggregated into a task group; commit queue: stores a plurality of task groups; commit manager: manages the commit queue, and batch-submits the metadata to the database.

[0038] See Figure 7 The embodiments of the present disclosure disclose an optimized commit processing method.

[0039] Commit processing flow stage: S201, the coordinator sends the commit ready signal to all participants; S202, after receiving the commit ready request, the participant checks whether there is a pre-commit task in the commit queue, if yes, the participant responds to the ready request, if no, the participant responds to the ready request; S203, the participant sends a commit ready signal to the coordinator after completing the task; S204, the coordinator converts the task state to ready after receiving the commit ready response from all participants; S205, the coordinator pushes the ready task to the task-group; S206, after the number of tasks in the task-group reaches the upper limit or TTL expires, the coordinator pushes the task-group to the group-commit queue, the commit manager pops a task-group from the group-commit queue, and performs batch processing operations on the kv database, including setbatch and delbatch operations; S207, the coordinator sets the commit task state to completed and notifies all participants to remove the task from the commit task queue.

[0040] It should be understood that the commit processing flow stage is also the commit phase. Commit is a core operation, which refers to the process of permanently taking effect all modifications to data in a transaction. See Figure 8 , the interaction process between the coordinator and the commit manager is shown. The task state of the commit task prepared for execution in Tasks(2pc-2) is converted to the ready state. The transaction coordinator initiates a ready instruction to all participants. After receiving the instruction, the participant queries whether the task can be ready and whether there is a pre-task, and reports the ready state to the coordinator. After receiving the ready response from all participants, the transaction coordinator converts the task state to the ready state. Multiple ready tasks are combined into a task-group, i.e., multiple set and del operations are converted into set-batch and delete-batch operations. The task-group is pushed to the group-commit queue, a task is popped from the group-commit queue, and submitted to the tikv database. After successful submission, the task state of the related multiple commit tasks in Tasks(2pc-2) is converted to completed, and the tasks are deleted from Tasks(2pc-2).

[0041] In the request phase, compared with the prior art, there are three optimizations: The first optimization can optimize single-row transactions.

[0042] See Figure 9In the prior art, the pessimistic lock transaction model, when multiple clients write a file at the same time in a concurrent file system, the execution of transactions needs to be queued and waited, resulting in limited performance. In the embodiment of the present disclosure, the tasks (write, setattr) of the waiting execution of continuous single-row transactions in Tasks (2pc-1) are aggregated into a task-group. Due to the processing of a single metadata, each task-group finally generates a transaction log, and there is only one update of metadata in the transaction log, and multiple update operations are combined into one update. In this way, the ability of concurrent file writing is greatly improved, the metadata performance is actually measured to be improved by 3 to 5 times, the write amplification problem of the transaction is reduced, the cpu load of the metadata server is reduced, the persistent load and cpu load of the database are reduced, and the network IO overhead of the database is reduced.

[0043] The second optimization optimizes the commit of single-row transactions.

[0044] Referring to Figure 10 In the prior art, the number of tasks waiting for commit is extremely large, which can accumulate to millions of tasks within a few minutes to preempt cpu scheduling, resulting in reduced transaction execution efficiency and extremely large memory occupation. In the embodiment of the present disclosure, the tasks (write, setattr) of the waiting execution of continuous single-row transactions in Tasks (2pc-2) are aggregated into a task-group. Due to the processing of a single metadata, a task-group can combine tens of thousands of task data, and finally only update one metadata to the database. In this way, the accumulated transaction data waiting for commit is sharply reduced, the cpu load of the metadata server is reduced, the write amplification problem of the transaction is reduced, the persistent load and cpu load of the database are reduced, and the network IO overhead of the database is reduced.

[0045] The third optimization aggregates and commits multiple transaction logs.

[0046] Referring to Figure 11 In the prior art, operations on directory items, such as create, mkdir, unlink, rename, etc., involve processing of multiple-row metadata, which cannot be aggregated and processed like single-row transactions, resulting in a very high accumulation of commit tasks. In the scenario of batch creation of files, the memory occupation is very high. The prior art converts the ready commit tasks into several set and delete instructions of tikv and issues them to tikv. In the embodiment of the present disclosure, the ready multiple commit tasks are converted into set-batch and delete-batch instructions of tikv and issued to tikv. In this way, the commit processing speed of the transaction is accelerated, the cpu load of the metadata server is reduced, and the network IO overhead of the database is reduced.

[0047] The fourth optimization is to combine different transactions of the same partition into one transaction log.

[0048] Referring to Figure 12 In the prior art, the write performance of tikv has problems. In multi-threaded tests, the set delay of tikv can reach 1 to 10 ms. In file semantics, one file operation needs to execute multiple set instructions, resulting in very high delay. In the embodiment of the present disclosure, in order to avoid the hotspot problem of the database, the metadata is logically partitioned. When writing a transaction log, transaction data generated by different metadata of the same partition at the same time is aggregated into one transaction log. In this way, the write amplification problem of the transaction is reduced, the cpu load of the metadata server is reduced, the persistence load and cpu load of the database are reduced, and the network IO overhead of the database is reduced.

[0049] The fifth optimization is to optimize part of the pessimistic lock to an optimistic lock.

[0050] Referring to Figure 13 In the prior art, the operation of the directory item (create, unlink, mkdir, etc.) involves the processing of three metadata, that is, the inode of the parent node needs to update the number of children and the mtime and ctime; create or delete the directory item; create or delete the inode of the child. When processing the transaction, the above three metadata together constitute the participants of the transaction; in the concurrent batch creation or deletion processing, the inode update of the parent node will first become a bottleneck; the inode update of the parent node only does addition and subtraction and update processing, and the order is not important, so this part of the transaction implementation can be changed from a pessimistic lock to an optimistic lock.

[0051] Referring to Figure 14 Schematic diagram of data structure required for optimistic lock implementation.

[0052] Parent inode metadata: the latest metadata that has been updated, which contains the current number of children.

[0053] Transaction task-group: group the continuous directory item tasks in Tasks (2pc-1) to be processed into a task-group.

[0054] Transaction id array: The contents of the elements in the array include: 1, transaction id; 2, whether the transaction is completed. The array contains the transaction ids of all tasks in the task-group, and the transaction ids in the array are in order. The purpose of the ordered id is to correctly replay all uncommitted transactions during transaction replay.

[0055] Referring to Figure 15 Fig. 2 is a schematic diagram of initiating a prepare phase process for a transaction coordinator.

[0056] It should be understood that the lock request is initiated to different transaction participants to request locking the corresponding resources. After the parent inode receives the request, it determines whether it is a directory entry operation. If not, it is processed according to the default pessimistic lock process, otherwise, the following optimistic lock process is executed.

[0057] The task is added to the task-group, and the transaction ID is added to the ID array and sorted, which is assigned by the transaction coordinator and is ordered and unique. The parent inode responds to the transaction coordinator that the lock is successful. The transaction coordinator waits for responses from all transaction participants. The transaction coordinator persists the transaction log, which only contains the incremental information of the children, i.e., the addition or deletion operation. After the transaction log is persisted successfully, the coordinator initiates an unlock request to all participants. After the parent inode receives the request, the task corresponding to the transaction ID in the ID array is set as completed. Whether there are still pre-task not completed in the ID array, if yes, it continues to wait; if not, the task is cleared from the task-group and the ID array, and the number of children in the metadata of the parent inode is updated. The latest metadata information of the parent inode is submitted to the commit queue, and a response is sent to the transaction coordinator.

[0058] In this way, when files are concurrently created or deleted under the same directory, the delay can be greatly reduced.

[0059] The embodiment of the disclosure also provides a file metadata optimization device based on a 2PC transaction, which comprises a processor and a memory. Optionally, the device can also comprise a communication interface and a bus. The processor, the communication interface and the memory can communicate with each other through the bus. The communication interface can be used for information transmission. The processor can call the logical instructions in the memory to execute the 2PC transaction-based file metadata optimization method applied to the 2PC transaction-based file metadata optimization system as described above.

[0060] The embodiment of the disclosure provides a storage medium, which stores computer executable instructions, and the computer executable instructions are configured to execute the 2PC transaction-based file metadata optimization method applied to the 2PC transaction-based file metadata optimization system as described above.

[0061] The embodiments of the present disclosure optimize the process of two-phase commit. In the preparation phase of two-phase commit, the coordinator of a transaction initiates a lock request to all participants. After receiving the lock request, the participant checks whether the metadata held by the participant satisfies the transaction condition. If yes, the participant locks the held metadata and returns the metadata to the coordinator. After receiving the information returned by all participants, the coordinator updates the metadata of the participants in the memory, writes the updated metadata into a transaction log, and immediately returns the result to the user. The commit phase of the transaction is executed asynchronously in the background, and the participants do not need to write local transactions. In the preparation phase, as long as the coordinator successfully persists the transaction log, it indicates that the metadata update is successful. The embodiments of the present disclosure can greatly reduce the read-write times of the database in a distributed multi-row transaction, and reduce the transaction delay.

[0062] The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable one skilled in the art to practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments represent only a few of the possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be changed. Parts and features of some embodiments can be included in or replace parts and features of other embodiments. Also, the words used in this application are only used to describe the embodiments and not to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or comprises" and the like mean the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, or device that includes the stated element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between embodiments can be referred to each other. For the methods, products, and the like disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, the relevant parts can be referred to the description of the method part.

[0063] Those skilled in the art can clearly understand the technical solutions of the present application from the description of the embodiments disclosed in the present application. The present application can be implemented by means of software functional modules and combined logic, and the present application can also be implemented by means of hardware. Based on the technical concepts of the present application, those skilled in the art can implement the present application by means of software and / or hardware using the technical means disclosed in the present application, and the technical solutions disclosed in the present application can be implemented by means of a combination of software and hardware. Based on the technical concepts of the present application, those skilled in the art can implement the present application by means of software and / or hardware using the technical means disclosed in the present application, and the technical solutions disclosed in the present application can be implemented by means of a combination of software and hardware.

[0064] The flow diagrams and the block diagrams in the drawings are presented to illustrate the possible implementation of the system, method and computer program product according to the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, a program segment or a portion of code, which comprises one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes can occur in a different order than that noted in the figures. For example, two consecutive boxes can actually be executed substantially in parallel, or they can be executed in reverse order, depending on the functionality involved. The operations or steps corresponding to the different boxes in the description of the flow diagrams and the block diagrams can also occur in a different order than that disclosed in the description, and sometimes there is no specific order between the different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can be executed in reverse order, depending on the functionality involved. Each box in the block diagrams and / or flow diagrams, and the combination of boxes in the block diagrams and / or flow diagrams, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or it can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A file metadata optimization method based on 2PC transactions, characterized in that: A coordinator is applied to a file metadata optimization system based on 2PC transactions, wherein the system includes multiple participants and a coordinator, and the method includes: Preparation stage: When the preceding transaction is identified as complete, all participants in the task-group are locked; After the lock is successfully acquired, the transaction log is persisted to the database; All participants send unlock requests; Submission stage: Send commit ready judgment signal to all participants; After receiving the commit ready signal from all participants, the task state is converted to the ready state; Merge multiple ready tasks into a single task-group; Persist multiple records in the task-group to the database in batch mode; Set the commit task status to completed and notify all participants to remove the task from the commit task queue.

2. The method according to claim 1, characterized in that Persist multiple records in the task-group to the database in batch mode, including: When the number of tasks in a task-group reaches the upper limit or the TTL expires, the task-group is pushed to the group-commit queue. The commit manager takes a task-group from the group-commit queue and performs batch operations on the kv database.

3. The method according to claim 1, characterized in that Batch operations, including: multiple set and del operations are converted into set-batch and delete-batch operations.

4. The method according to claim 1, wherein When the preceding transaction is identified as complete, before all participants in the task-group are locked, the following steps are also included: Get the predecessor task being executed; Get a new request and check whether there is a task-group. If there is a task-group, add the request to the task-group. If there is no task-group, create a task-group and add the request to the task-group. Check whether the preceding transaction is completed.

5. A file metadata optimization method based on 2PC transactions, characterized in that: A method for optimizing a file metadata system based on a 2PC transaction, wherein the system includes multiple participants and a coordinator, includes: Preparation stage: Receive lock request and lock; If the lock is successful, a lock completion signal is sent; Receive unlock request and unlock; Submission stage: Receive the commit ready judgment signal and check whether all tasks in its own commit task queue have been committed; After completion, a commit ready signal is sent to the coordinator.

6. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images