File metadata optimization method and system based on 2pc transaction, device, storage medium

Abstract

The application relates to the computer technical field and discloses a file metadata optimization method and system based on a 2PC transaction, equipment and a storage medium. The system comprises a plurality of participants and a coordinator. Multiple single-row transactions of the same file are optimized into one transaction. The submission of the multiple single-row transactions of the same file is optimized into one submission. Multiple transaction logs are aggregated and submitted together. Therefore, the application improves the performance of file metadata, reduces the write amplification problem of transactions, and improves the performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, for example, to a file metadata optimization method and system based on 2PC transaction, device and storage medium. BACKGROUND

[0002] The establishment of two-phase commit transaction is based on the following assumptions: in the distributed system, there is a node as a coordinator (Coordinator), and other nodes as participants (Participant). And the nodes can communicate with each other. All nodes use pre-written log, and the log is written and kept on reliable storage device, so that even if the node is damaged, the log data will not be lost. All nodes will not be permanently damaged, even if damaged, it can still be restored.

[0003] The first stage - the request for submission stage:

[0004] The coordinator node asks all participant nodes whether the commit operation can be executed, and starts to wait for the response of each participant node; the participant node executes all transaction operations from the initiation of the inquiry, and writes Undo information and Redo information into the log; each participant node responds to the inquiry initiated by the coordinator node. If the transaction operation of the participant node is actually executed successfully, a "agree" message is returned; if the transaction operation of the participant node is actually executed unsuccessfully, an "abort" message is returned

[0005] The second stage - the execution stage of submission:

[0006] The coordinator node sends a request for "formal submission" to all participant nodes; the participant node formally completes the operation, and releases the resources occupied during the entire transaction; the participant node sends a "completion" message to the coordinator node; the coordinator node receives the "completion" message fed back by all participant nodes, and completes the transaction.

[0007] The prior art is all improvements on the above two basic implementation modes. A Chinese patent with application number 202311765490X discloses an improved two-phase commit transaction implementation method and system, device, and storage medium. The system includes multiple participants and a coordinator. In the preparation phase of the two-phase commit, the coordinator of the transaction initiates a lock request to all participants. After receiving the lock request, the participant will retrieve whether the metadata it holds meets the transaction condition. If it meets, it will lock the held metadata and return the metadata to the coordinator. The coordinator updates the participant's metadata in memory and writes the updated metadata into the unique transaction log. The commit phase of the transaction is executed asynchronously in the background, and the participant does not need to write a local transaction. In the preparation phase, as long as the coordinator successfully persists the transaction log, it means that the metadata update is successful. Therefore, the prior art can greatly reduce the read-write times of the database in distributed multi-row transactions and reduce the transaction delay.

[0008] Although the above system and method can greatly reduce the read-write times of the database and reduce the transaction delay, there are still problems of low running efficiency, mainly including: the coordinator needs to wait for all participants to vote before making a decision, which is time-consuming and affects performance; all nodes of the transaction use the pre-write log method, which causes the write amplification problem; all nodes of the transaction use the pre-write log method, which causes the write amplification problem. SUMMARY

[0009] To have a basic understanding of some aspects of the disclosed embodiments, a brief overview is given below. The overview is not an extensive review of all of the features of the embodiments nor is it intended to identify key / critical elements of the embodiments or determine the scope of the embodiments. The sole purpose of the overview is to present some aspects of the embodiments in a simplified form as a prelude to the more detailed description that is presented later.

[0010] The embodiments of the present disclosure provide a file metadata optimization method and system based on 2PC transaction, device, and storage medium, which can greatly improve the running efficiency in distributed multi-row transactions.

[0011] In some embodiments, a file metadata optimization method based on 2PC transaction is provided, which is applied to a coordinator of a file metadata optimization system based on 2PC transaction. The system includes multiple participants and a coordinator. The method includes:

[0012] Preparation phase:

[0013] When the front transaction is identified to be completed, all participants in the task-group are locked;

[0014] After the locking is successful, the transaction log is persisted to the database.

[0015] All participants send a lock request;

[0016] Commit phase:

[0017] Send commit ready signal to all participants;

[0018] After receiving commit ready signal from all participants, change task status to ready status;

[0019] Merge multiple ready tasks into a unique task-group;

[0020] Persist multiple records in task-group to database in batch mode;

[0021] Set commit task status to completed, and notify all participants to remove task from commit task queue.

[0022] Preferably, persisting multiple records in task-group to database in batch mode comprises:

[0023] After the number of tasks in task-group reaches an upper limit or TTL expires, push task-group to group-commit queue, commit manager takes out a task-group from group-commit queue, and performs batch operation to kv database.

[0024] Preferably, batch operation comprises: multiple set and del operations are converted into set-batch and delete-batch operations.

[0025] Preferably, before locking all participants in task-group, the method further comprises:

[0026] Obtain the executing pre-task;

[0027] Obtain a new request, check whether there is a task-group, if there is a task-group, add the request to the task-group, if there is no task-group, create a task-group, and add the request to the task-group;

[0028] Check whether the pre-transaction is completed.

[0029] In some embodiments, a file metadata optimization method based on 2PC transaction is provided, which is applied to a participant of a file metadata optimization system based on 2PC transaction, the system comprising a plurality of participants and a coordinator, and the method comprising:

[0030] Preparation phase:

[0031] receiving the lock request and performing the lock;

[0032] sending a lock completion signal in case of successful lock;

[0033] receiving the unlock request and performing the unlock;

[0034] commit phase:

[0035] receiving a commit ready signal and detecting whether all tasks in the commit task queue of the participant are completed;

[0036] sending a commit ready signal to the coordinator after completion.

[0037] In some embodiments, a device is disclosed, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the method for optimizing file metadata based on 2PC transaction of the participant in the system for optimizing file metadata based on 2PC transaction.

[0038] In some embodiments, a computer readable storage medium is disclosed, which stores a computer program, wherein the program is executed by a processor to perform the method for optimizing file metadata based on 2PC transaction of the participant in the system for optimizing file metadata based on 2PC transaction.

[0039] In some embodiments, a device is disclosed, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the method for optimizing file metadata based on 2PC transaction of the coordinator in the system for optimizing file metadata based on 2PC transaction.

[0040] The method and system for optimizing file metadata based on 2PC transaction provided by the embodiments of the present disclosure optimize the process of two-phase commit, optimize multiple single-row transactions of the same file into one transaction, optimize the commit of multiple single-row transactions of the same file into one commit, and aggregate multiple transaction logs for commit. The embodiments of the present disclosure can achieve the following technical effects:

[0041] The performance of file metadata is greatly improved, and the performance of metadata can be improved by more than 3 to 5 times

[0042] The write amplification problem of transaction is greatly reduced, and the amplification coefficient is reduced from 3 to 4 to less than 1;

[0043] The cpu load of the metadata server is greatly reduced, and the cpu of a single server can be reduced by more than 200%.

[0044] The persistence load and CPU load of the database are greatly reduced;

[0045] The network IO overhead of the database is greatly reduced.

[0046] The foregoing general description and the following description are only exemplary and explanatory, and are not used to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0047] One or more embodiments are exemplarily illustrated by the corresponding drawings, which do not constitute a limitation on the embodiments, elements with the same reference numerals in the drawings are considered as similar elements, the drawings do not constitute a proportional limitation, and wherein:

[0048] Figure 1 is a schematic diagram of an improved two-phase commit transaction implementation method provided by an embodiment of the present disclosure;

[0049] Figure 2 is a schematic diagram of a file metadata optimization system based on a 2PC transaction provided by an embodiment of the present disclosure;

[0050] Figure 3 is a schematic diagram of an architecture provided by an embodiment of the present disclosure;

[0051] Figure 4 is a schematic diagram of a task chain representing a first phase and a second phase of a transaction provided by an embodiment of the present disclosure;

[0052] Figure 5 is a schematic diagram of a file metadata optimization method flow provided by an embodiment of the present disclosure based on a 2PC transaction;

[0053] Figure 6 is a flowchart of a file metadata optimization method in a preparation phase provided by an embodiment of the present disclosure;

[0054] Figure 7 is a schematic diagram of an optimized commit processing flow provided by an embodiment of the present disclosure;

[0055] Figure 8 is a schematic diagram of an interaction flow between a coordinator and a commit manager provided by an embodiment of the present disclosure;

[0056] Figure 9 is a schematic diagram of an optimized single-row transaction provided by an embodiment of the present disclosure;

[0057] Figure 10 is a commit schematic diagram of an optimized single-row transaction provided by an embodiment of the present disclosure;

[0058] Figure 11 is a commit diagram of aggregating multiple transaction logs provided by an embodiment of the present disclosure;

[0059] Figure 12 is a diagram of merging different transactions of a same partition into one transaction log provided by an embodiment of the present disclosure;

[0060] Figure 13 is a diagram of optimizing a partial pessimistic lock to an optimistic lock provided by an embodiment of the present disclosure;

[0061] Figure 14 is a diagram of a data structure required for implementing an optimistic lock provided by an embodiment of the present disclosure;

[0062] Figure 15 is a diagram of a flow of initiating a preparation phase by a transaction coordinator provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, multiple details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, in order to simplify the drawings, well-known structures and systems can be simplified.

[0064] The following description and accompanying drawings fully illustrate specific embodiments of the invention to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of embodiments of the invention includes the entire scope of the claims and all available equivalents thereof. In this document, each embodiment may be referred to individually or collectively with the term "invention," which is merely for convenience and is not intended to automatically limit the scope of application to any single invention or inventive concept if more than one invention is disclosed. Relational terms such as "first" and "second" are used herein only to distinguish one entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the methods, products, etc., disclosed in the embodiments, since they correspond to the method section disclosed in the embodiments, the descriptions are relatively simple; relevant details can be found in the method section description.

[0065] An improved two-phase commit transaction implementation method is disclosed in the prior art. See [link to previous document]. Figure 1 This is a schematic diagram of an improved two-phase commit transaction implementation method in the prior art. The method includes: a preparation phase;

[0066] The coordinator node queries all participant nodes to see if the transaction can be executed, and waits for the latest metadata from each participant node; after receiving "agreement" messages from all participants, the coordinator persists a transaction log to the database; after the coordinator notifies all participants that the transaction has been successfully committed, the participants unlock their accounts.

[0067] Submission phase:

[0068] The coordinator inquires all participants whether the transaction can be committed, waits for the state report of each participant, and after receiving the "agreement" message of all participants, updates the metadata of all participants to the database and then deletes the transaction log. The coordinator reports the completion of the commit to all participants.

[0069] To solve the problems in the related art, the embodiments of the present disclosure disclose a file metadata optimization system based on 2PC transaction, as shown in Figure 2 The system comprises a plurality of participants and a coordinator.

[0070] The architecture in the embodiments of the present disclosure is described. The metadata service is divided into two layers in architecture.

[0071] Database layer: responsible for the storage of metadata, distributed database tikv

[0072] Metadata agent layer: provides an interface of file semantics to the outside, and converts the data of the file system namespace into records in the tikv database to the inside, and uses transactions to ensure the correctness of operations. As shown in Figure 3 , it is a schematic diagram of an architecture.

[0073] When executing a transaction, as shown in Figure 4 , it is a schematic diagram of the first and second phase task chains corresponding to the transaction. Each metadata has two task chain tables:

[0074] Task chain table of the first phase of the transaction: Tasks(2pc-1)

[0075] Task chain table of the second phase of the transaction: Tasks(2pc-2)

[0076] It should be noted that Tasks(2pc-1): the grayed task indicates that the write transaction is being executed, and the others are tasks waiting to be executed, and the tasks are in order. Tasks(2pc-2): the grayed task indicates that the transaction is being committed, and the others are tasks waiting to be committed. Figure 4 The Tasks(2pc-2) of the metadata C and D in the middle have no tasks, indicating that the commit operation has not been executed.

[0077] In combination with Figure 5 , the embodiments of the present disclosure provide a file metadata optimization method based on 2PC transaction, which comprises the following steps:

[0078] Preparation phase:

[0079] S101, the coordinator acquires the pre-executed task.

[0080] S102, the coordinator gets a new request, checks if there is a task-group, if there is a task-group, adds the request to the task-group, if there is no task-group, creates a task-group, and adds the request to the task-group.

[0081] S103, the coordinator checks if the pre-transaction is completed.

[0082] S104, the coordinator identifies the completion signal, and locks all participants in the task-group.

[0083] S105, the participant receives the lock request, and performs the lock.

[0084] S106, the participant sends a lock completion signal in the case of successful locking.

[0085] S108, the coordinator successfully locks, and persists the transaction log in the database.

[0086] S109, the coordinator sends an unlock request to all participants.

[0087] S110, the participant receives the unlock request, and performs the unlock.

[0088] It should be understood that the write and setattr requests are submitted to the transaction coordinator. The transaction coordinator checks if there is a pre-transaction that has not been completed by the transaction participant. If not, the pre-transaction is executed according to the pre-optimization scheme. If yes, the following scheme is executed. The coordinator creates a new task-group, and adds the task to the task-group. Subsequent continuous requests (write, setattr) are added to the task-group. The coordinator checks if the participant has completed the pre-transaction. If yes, the coordinator performs the lock process on the participant. After the lock is successful, the coordinator persists the transaction log in the database. The coordinator unlocks the participant, and returns the request result. See Figure 6 A schematic diagram of a preparation phase file metadata optimization method.

[0089] A new data structure is introduced in the embodiments of the present disclosure:

[0090] task-group: a plurality of ready commit tasks are aggregated into a task group;

[0091] commit queue: stores a plurality of task groups;

[0092] commit manager: manages the commit queue, and batch-submits metadata to the database.

[0093] See Figure 7 This disclosure presents an optimized commit processing method.

[0094] Commit processing flow stages:

[0095] S201, the coordinator sends a commit ready signal to all participants;

[0096] S202. After receiving the commit ready request, the participant checks whether there are any unfinished previous commit tasks in its own commit queue. If they have been completed, the participant responds that it is ready.

[0097] S203, after the participant completes the process, they send a commit ready signal to the coordinator;

[0098] S204, after receiving the commit ready response from all participants, the coordinator changes the task status to ready;

[0099] S205, the coordinator pushes the ready tasks to the task-group;

[0100] S206, when the number of tasks in a task-group reaches the limit or the TTL expires, the coordinator will push the task-group to the group-commit queue. The commit manager will take a task-group from the group-commit queue and perform batch processing operations on the key-value database. The batch processing operations include setbatch and delbatch operations.

[0101] In S207, the coordinator sets the commit task status to completed and notifies all participants to remove the task from the commit task queue.

[0102] It should be understood that the commit processing phase is also the submission phase. Commit is a core operation that refers to the process of making all changes to data within a transaction permanent. See also... Figure 8This illustrates the interaction flow between the coordinator and the commit manager. The process begins by changing the task status of commit tasks ready to be executed in Tasks(2pc-2) to the ready state. The transaction coordinator sends a ready command to all participants. Upon receiving the command, participants query whether the task is ready, whether there are any preceding tasks, and report their ready status to the coordinator. After receiving the ready responses from all participants, the transaction coordinator changes the task status to the ready state. Multiple ready tasks are merged into a task-group, converting multiple set and del operations into set-batch and delete-batch operations. The task-group is pushed to the group-commit queue, a task is popped from the group-commit queue, and submitted to the TiKV database. After successful submission, the task status of the related commit tasks in Tasks(2pc-2) is changed to completed, and they are removed from Tasks(2pc-2).

[0103] During the request phase, compared with existing technologies, there are three optimizations:

[0104] The first optimization is able to optimize single-row transactions.

[0105] See Figure 9 In existing technologies, pessimistic locking transaction models in concurrent file systems require multiple clients to queue and wait when writing to the same file, resulting in performance limitations. However, in this embodiment, the tasks (write, setattr) of consecutive single-row transactions waiting to be executed in Tasks(2pc-1) are aggregated into task-groups. Since each task-group processes individual metadata, it ultimately generates a transaction log containing only one metadata update, merging multiple update operations into a single update. This significantly improves the ability to write concurrently to files, with actual tests showing a 3-5x improvement in metadata performance, reducing write amplification issues, lowering the CPU load on the metadata server, reducing the database's persistence and CPU load, and reducing network I / O overhead.

[0106] The second optimization is to optimize the commit of single-row transactions.

[0107] See Figure 10In existing technologies, the number of tasks waiting to be committed is extremely large, accumulating to tens of millions of tasks within minutes, competing for CPU scheduling, leading to decreased transaction execution efficiency and excessive memory consumption. However, in this embodiment, the tasks (write, setattr) of consecutive single-row transactions waiting to be executed in Tasks (2pc-2) are aggregated into task-groups. Because of the processing of individual metadata, a task-group can merge tens of thousands of task data entries, ultimately updating only one metadata entry to the database. This drastically reduces the accumulated transaction data awaiting commit, lowers the CPU load on the metadata server, mitigates write amplification issues, reduces the database's persistence load and CPU load, and reduces the database's network I / O overhead.

[0108] The third optimization is to aggregate and commit multiple transaction logs.

[0109] See Figure 11 In existing technologies, operations on directory entries, such as create, mkdir, unlink, and rename, involve processing multiple lines of metadata. This cannot be handled with the aggregation of single-line transactions, leading to a very high accumulation of commit tasks. In scenarios involving batch file creation, memory consumption is extremely high. Existing technologies convert ready commit tasks into several TiKV set and delete commands and send them to TiKV. However, in this embodiment, multiple ready commit tasks are converted into TiKV set-batch and delete-batch commands and sent to TiKV. This accelerates transaction commit processing, reduces the CPU load on the metadata server, and lowers the network I / O overhead of the database.

[0110] The fourth optimization is to merge different transactions within the same partition into a single transaction log.

[0111] See Figure 12 In existing technologies, TiKV's write performance has issues. In multi-threaded tests, TiKV's set latency can reach 1 to 10 ms. Furthermore, in file semantics, a single file operation requires executing multiple set commands, resulting in very high latency. In this embodiment, to avoid database hotspot issues, the metadata is logically partitioned. When writing the transaction log, transaction data generated by different metadata within the same partition at the same time is aggregated into a single transaction log. This reduces the write amplification problem, lowers the CPU load on the metadata server, reduces the database's persistence load and CPU load, and reduces the database's network I / O overhead.

[0112] The fifth optimization is to convert some pessimistic locking to optimistic locking.

[0113] See Figure 13 In existing technologies, directory entry operations (create, unlink, mkdir, etc.) involve the processing of three metadata items: updating the number of children inodes, mtime, and ctime in the parent node's inode; creating or deleting directory entries; and creating or deleting the inodes of children. During transaction processing, these three metadata items together constitute the participants in the transaction. In concurrent batch creation or deletion processing, updating the parent node's inode first becomes the bottleneck. Since the parent node's inode update only involves addition, subtraction, and update operations, the order is not important. Therefore, the transaction implementation for this part can be changed from pessimistic locking to optimistic locking.

[0114] See Figure 14 This is a diagram illustrating the data structure required for implementing optimistic locking.

[0115] Parent inode metadata: The latest updated metadata, which includes the current number of children.

[0116] Transaction task-group: Groups consecutive directory item tasks to be processed in Tasks(2pc-1) into a task-group.

[0117] Transaction ID array:

[0118] The elements in the array include: 1. Transaction ID; 2. Whether the transaction has been completed. The array contains the transaction IDs of all tasks in the task-group, and the transaction IDs in the array are ordered. The purpose of ordering the IDs is to correctly replay all uncommitted transactions during transaction replay.

[0119] See Figure 15 This is a flowchart illustrating the preparation phase process initiated by the coordinator of the transaction.

[0120] It should be understood that locking requests are sent to different transaction participants, requesting to lock the corresponding resources. After receiving the request, the parent inode determines whether it is a directory entry operation. If not, it is processed according to the default pessimistic locking process; otherwise, the optimistic locking process is executed as follows.

[0121] The task is added to the task-group, and the transaction ID is added to the ID array and sorted. The transaction ID is assigned by the transaction coordinator and is ordered and unique. The parent inode responds to the transaction coordinator that the lock acquisition was successful. The transaction coordinator waits for responses from all transaction participants. The transaction coordinator persists the transaction log, which only contains incremental information about the children, i.e., whether it is an addition or deletion operation. After the transaction log is successfully persisted, the coordinator sends an unlock request to all participants. Upon receiving the request, the parent inode checks if there are any preceding tasks in the ID array. If so, it continues to wait; otherwise, it removes the task from the task-group and the ID array, and updates the number of children in the parent inode's metadata. The latest metadata information of the parent inode is committed to the commit queue, and a response is sent to the transaction coordinator.

[0122] This significantly reduces latency when creating or deleting files concurrently in the same directory.

[0123] This disclosure also provides a file metadata optimization device based on 2PC transactions, including a processor and memory. Optionally, the device may further include a communication interface and a bus. The processor, communication interface, and memory can communicate with each other via the bus. The communication interface can be used for information transmission. The processor can invoke logical instructions in the memory to execute the 2PC transaction-based file metadata optimization method as described above in the 2PC transaction-based file metadata optimization system.

[0124] This disclosure provides a storage medium storing computer-executable instructions configured to execute the 2PC transaction-based file metadata optimization method as described above in a 2PC transaction-based file metadata optimization system.

[0125] This disclosure optimizes the two-phase commit process. During the preparation phase, the transaction coordinator sends a lock request to all participants. Upon receiving the lock request, each participant checks if its metadata meets the transaction conditions. If it does, it locks the metadata and returns it to the coordinator. After receiving the information from all participants, the coordinator updates the participants' metadata in memory and writes the updated metadata to a transaction log entry. It then immediately replies to the user, asynchronously executing the transaction commit phase in the background. Participants do not need to write to their local transactions. During the preparation phase, successful persistence of the transaction log by the coordinator signifies a successful metadata update. This disclosure significantly reduces the number of database read / write operations and transaction latency in distributed multi-row transactions.

[0126] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0127] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, units, and processes described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for optimizing file metadata based on 2PC transactions, characterized in that, A coordinator applied to a 2PC transaction-based file metadata optimization system, the system comprising multiple participants and a coordinator, the method comprising: Preparation phase: When the preceding transaction is identified as complete, all participants in the task-group are locked. The task-group is a group of multiple ready commit tasks, and the commit is an operation that makes all data modifications in the transaction permanent. After successfully acquiring the lock, persist the transaction log to the database. Send unlock request to all participants; Submission phase: Send a commit ready signal to all participants; After receiving commit ready signals from all participants, the task status is changed to ready. Merge multiple ready tasks into a single task-group; Persist multiple records in a task-group to the database in a batch manner; Set the commit task status to completed and notify all participants to remove the task from the commit task queue.

2. The method according to claim 1, characterized in that, Persist multiple records in a task-group to the database in a batch manner, including: When the number of tasks in a task-group reaches its limit or the TTL expires, the task-group is pushed to the group-commit queue. The commit manager retrieves a task-group from the group-commit queue and performs a batch processing operation on the key-value database. The group-commit queue stores the queue structure of multiple task-groups that have met the requirements for ready tasks.

3. The method according to claim 1, characterized in that, Batch processing operations include converting multiple set and del operations into set-batch and delete-batch operations.

4. The method according to claim 1, characterized in that, Before locking all participants in the task-group upon recognizing the completion of a preceding transaction, the following steps are also included: Get the currently executing prerequisite tasks; When a new request is received, check if a task-group exists. If a task-group already exists, add the request to the task-group. If no task-group exists, create the task-group and add the request to the task-group. Check whether the prerequisite tasks have been completed.

5. A method for optimizing file metadata based on 2PC transactions, characterized in that, A method for optimizing file metadata based on 2PC transactions, whereby the system comprises multiple participants and a coordinator, includes the following: Preparation phase: Receive a lock request and perform the lock operation; If the locking is successful, a locking completion signal is sent. Receive the unlock request and proceed with the unlocking. Submission phase: It receives a commit ready signal and checks whether all tasks in its own commit task queue have been committed. Commit is the operation that makes all data modifications in the transaction permanent. Once completed, send a commit ready signal to the coordinator.

6. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images