Metadata processing method, apparatus, device, storage medium, and product

Abstract

The application provides a metadata processing method, device, equipment, storage medium and product. The method comprises the following steps: in response to receiving an operation request for metadata in a distributed file system comprising a plurality of preset directory groups sent by a client device, determining a target directory group storing target metadata according to a storage path of the target metadata in the operation request; the target metadata comprises target directory access metadata and / or target file metadata under a directory thereof, and the target directory group is used for storing the target metadata and parent directory timestamp metadata thereof; for each preset directory group, the data in each preset directory group is stored in a corresponding non-uniform memory access node respectively, a plurality of non-uniform memory access nodes are located in corresponding metadata servers, and the data in each preset directory group comprises preset metadata and parent directory timestamp metadata thereof; performing an operation on the target metadata according to an operation type in the operation request, and updating the parent directory timestamp metadata of the target metadata.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a metadata processing method and device, equipment, storage medium and product. BACKGROUND

[0002] A distributed file system (DFS) connects multiple physical nodes together through a high-speed network to realize unified management of multi-node resources and provides a file system access interface and a unified file system namespace to users. The distributed file system is usually designed based on a client / server mode, the client provides a standard file system access interface for an application program, and the server stores all data and metadata in the file system. The distributed file system usually adopts a design architecture separating data and metadata, the data is stored on multiple data servers, and the metadata is stored on multiple metadata servers, and the metadata includes directory metadata and file metadata.

[0003] In the prior art, the metadata located in the same directory level is placed in the same non-uniform memory access node in units of directory levels of the distributed file system, that is, the metadata in different directory levels can be located in different non-uniform memory access nodes. When the metadata in a directory level is updated (for example, created, deleted, etc.), the parent directory metadata of the directory level needs to be updated correspondingly.

[0004] However, in the case that the metadata in different directory levels is located in different non-uniform memory access nodes, when the metadata in a directory level is updated (for example, created, deleted, etc.), network communication needs to be performed across non-uniform memory access nodes to complete the corresponding update of the parent directory metadata of the directory level, which can cause low performance of the metadata server. SUMMARY

[0005] The present application provides a metadata processing method, device, equipment, storage medium and product to solve the problem of low performance of the metadata server caused by the corresponding update of the parent directory metadata needing network communication across non-uniform memory access nodes.

[0006] In a first aspect, an embodiment of the present application provides a metadata processing method applied to a metadata processing device, and the metadata processing method comprises the following steps:

[0007] In response to receiving an operation request for metadata in a distributed file system sent by a client device, the operation request includes a storage path of target metadata and an operation type;

[0008] determining a target directory group storing the target metadata according to a storage path of the target metadata; the target metadata comprises target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system comprises a plurality of preset directory groups, each preset directory group is stored in a corresponding non-uniform memory access node, and a plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group comprises preset metadata and parent directory timestamp metadata of the preset metadata;

[0009] performing the operation on the target metadata according to the operation type, and updating the parent directory timestamp metadata of the target metadata in the target directory group.

[0010] Optionally, the performing the operation on the target metadata according to the operation type, and updating the parent directory timestamp metadata of the target metadata in the target directory group comprises:

[0011] in response to the operation type of the target metadata being target metadata creation, writing the target metadata into the target directory group;

[0012] determining a system timestamp when the target metadata is written as a creation time of the target metadata;

[0013] after the target metadata is written, updating the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata.

[0014] Optionally, after the performing the operation on the target metadata according to the operation type, and updating the parent directory timestamp metadata of the target metadata in the target directory group, the method further comprises:

[0015] in response to detecting that the distributed file system is abnormal and the creation time of the target metadata is later than a corresponding time in the parent directory timestamp metadata of the target metadata, updating the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata.

[0016] Optionally, the performing the operation on the target metadata according to the operation type, and updating the parent directory timestamp metadata of the target metadata in the target directory group comprises:

[0017] in response to the operation type being a deletion operation, marking the target metadata as invalid;

[0018] determining a system timestamp when the target metadata is marked as invalid as a deletion time of the target metadata;

[0019] updating parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata;

[0020] deleting the target metadata after updating the parent directory timestamp metadata of the target metadata.

[0021] Optionally, after the operation on the target metadata according to the operation type and the updating of the parent directory timestamp metadata of the target metadata in the target directory group, the method further comprises:

[0022] updating the parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata in response to the distributed file system exception and the deletion time of the target metadata being later than the corresponding time in the parent directory timestamp metadata of the target metadata;

[0023] deleting the target metadata in response to the distributed file system exception, the deletion time of the target metadata being the same as the corresponding time in the parent directory timestamp metadata of the target metadata, and the target metadata not being deleted.

[0024] Optionally, when the operation on the target metadata according to the operation type and the updating of the parent directory timestamp metadata of the target metadata in the target directory group, the method further comprises:

[0025] determining the operation type corresponding to the simultaneously triggered operations in response to the simultaneously triggered operations on the target metadata in the distributed file system including at least two, wherein the operation type corresponding to the simultaneously triggered operations includes at least one of a timestamp updating operation, a write operation and a read operation, the timestamp updating operation being an operation of updating the parent directory timestamp metadata of the target directory metadata in the creation or deletion process of the target metadata, the write operation being an operation of updating the target metadata except the timestamp updating operation, and the read operation being an operation of reading the target metadata;

[0026] adopting a preset read-write lock algorithm to perform concurrent control on the simultaneously triggered operations in response to the operation type corresponding to the simultaneously triggered operations including the write operation;

[0027] adopting a preset lock-free atomic updating algorithm to perform concurrent control on the multiple timestamp updating operations in response to the multiple timestamp updating operations being simultaneously triggered;

[0028] In response to the timestamp update operation being triggered at the same time as the read operation, a preset optimistic concurrency control algorithm is used to perform concurrency control on the timestamp update operation and the read operation.

[0029] Optionally, the operation types corresponding to the simultaneously triggered operations include at least one of the timestamp update operation and the read operation, and a write operation; and the concurrency control on the simultaneously triggered operations by using the preset read-write lock algorithm includes:

[0030] obtaining a current lock state in the preset read-write lock algorithm;

[0031] in response to the current lock state being a write lock held by the write operation, executing the write operation, and after the write operation is executed, configuring a read lock to be held by the timestamp update operation and / or the read operation, so as to execute the timestamp update operation and / or the read operation; or

[0032] in response to the current lock state being a read lock held by the timestamp update operation and / or the read operation, executing the timestamp update operation and / or the read operation, and after the timestamp update operation and / or the read operation are executed, configuring a write lock to be held by the write operation, so as to execute the write operation.

[0033] Optionally, the parent directory timestamp metadata of the target metadata includes a last data modification time and a last metadata modification time.

[0034] the concurrency control on the plurality of timestamp update operations by using the preset lock-free atomic update algorithm includes:

[0035] obtaining a to-be-updated time corresponding to each of the plurality of timestamp update operations, and a last data modification time and a last metadata modification time recorded in the parent directory timestamp metadata; wherein the to-be-updated time is a creation time of the target metadata or a deletion time of the target metadata;

[0036] updating the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata by using a maximum value among the to-be-updated time corresponding to each of the plurality of timestamp update operations, the last data modification time, and the last metadata modification time.

[0037] Optionally, the parent directory timestamp metadata includes a last metadata modification time.

[0038] the concurrency control on the timestamp update operation and the read operation by using the preset optimistic concurrency control algorithm includes:

[0039] executing the timestamp update operation;

[0040] reading the parent directory timestamp metadata to obtain a first last metadata modification time before performing the read operation;

[0041] reading the parent directory timestamp metadata to obtain a second last metadata modification time after performing the read operation;

[0042] re-performing the read operation in response to the first last metadata modification time being different from the second last metadata modification time until the first last metadata modification time is the same as the second last metadata modification time, and feeding back target metadata read by the read operation.

[0043] In a second aspect, an embodiment of the present application provides a metadata processing method, applied to a metadata processing device, and the method comprises:

[0044] splitting preset directory metadata in a distributed file system into preset directory timestamp metadata and preset directory access metadata; wherein the preset directory timestamp metadata comprises timestamp information of the preset directory metadata, and the preset directory access metadata comprises information of the preset directory metadata except the timestamp information;

[0045] storing at least one of the preset directory access metadata and preset file metadata under a directory where the preset directory access metadata is located and parent directory timestamp metadata of the preset directory access metadata in a preset directory group;

[0046] storing at least one of child file metadata of the preset directory access metadata and child directory access metadata of the preset directory access metadata and the preset directory timestamp metadata in another preset directory group;

[0047] storing metadata in the preset directory group in a corresponding non-uniform memory access node;

[0048] wherein a plurality of non-uniform memory access nodes are located in corresponding metadata servers, the preset directory group is used as a target directory group, the target directory group is determined according to a storage path of target metadata in a request for an operation on metadata in the distributed file system by a client, the target metadata comprises target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the operation on the target metadata and updating of parent directory timestamp metadata of target metadata in the target directory group are both performed in a non-uniform memory access node corresponding to the target directory group.

[0049] Optionally, the storing of the metadata in the preset directory group in a corresponding non-uniform memory access node comprises:

[0050] performing consistency hash calculation on the preset directory group to obtain a hash value of the preset directory group;

[0051] based on the hash value, storing metadata in the preset directory group in a corresponding non-uniform memory access node.

[0052] Optionally, the storing metadata in the preset directory group in a corresponding non-uniform memory access node based on the hash value comprises:

[0053] determining the number and the identity of the non-uniform memory access nodes of the metadata server;

[0054] respectively determining the remainders obtained by dividing the hash values of the respective preset directory groups by the number;

[0055] storing metadata in each of the preset directory groups in a corresponding non-uniform memory access node according to a pre-stored correspondence between the remainders and the identities.

[0056] Optionally, the storing metadata in the preset directory group in a corresponding non-uniform memory access node further comprises:

[0057] storing metadata in the preset directory group in a corresponding non-uniform memory access node in the form of a key-value pair.

[0058] Optionally, the parent directory timestamp metadata of the metadata of the preset directory comprises a directory identity.

[0059] The form of the key-value pair is as follows:

[0060] The key-value pair is composed of the directory identity as the key and the parent directory timestamp metadata of the metadata of the preset directory as the value.

[0061] The key-value pair is composed of the directory identity and a file naming string of file metadata under a directory where the metadata of the preset directory is located as the key, and the file metadata under the directory where the metadata of the preset directory is located as the value.

[0062] The key-value pair is composed of the directory identity and a directory naming string of the metadata of the preset directory as the key, and the metadata of the preset directory as the value.

[0063] In a third aspect, an embodiment of the present application provides a metadata processing method applied to a client device, and the method comprises:

[0064] sending an operation request for metadata in a distributed file system to a metadata processing device;

[0065] The operation request includes a storage path of target metadata and an operation type, the operation request is used to trigger the metadata processing device to determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system includes a plurality of preset directory groups, each preset directory group is stored in a corresponding non-uniform memory access node, a plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group includes preset metadata and parent directory timestamp metadata of the preset metadata; the operation request is also used to trigger the metadata processing device to perform an operation on the target metadata according to the operation type and update the parent directory timestamp metadata of the target metadata in the target directory group.

[0066] In a fourth aspect, an embodiment of the present application provides a metadata processing apparatus, applied to a metadata processing device, and the apparatus includes:

[0067] A response module is configured to respond to receiving an operation request for metadata in a distributed file system sent by a client device, the operation request including a storage path of target metadata and an operation type;

[0068] A determination module is configured to determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system includes a plurality of preset directory groups, each preset directory group is stored in a corresponding non-uniform memory access node, a plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group includes preset metadata and parent directory timestamp metadata of the preset metadata;

[0069] A processing module is configured to perform an operation on the target metadata according to the operation type and update the parent directory timestamp metadata of the target metadata in the target directory group.

[0070] In a fifth aspect, an embodiment of the present application provides a metadata processing device, and the metadata processing device includes a processor and a memory connected with the processor in communication;

[0071] The memory stores computer execution instructions;

[0072] The processor executes computer-executed instructions stored in the memory to implement the metadata processing method of any one of the first aspect embodiment and the second aspect embodiment.

[0073] In a sixth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executed instructions. The computer-executed instructions, when executed by a processor, are used to implement the metadata processing method of any one of the first aspect embodiment, the second aspect embodiment, and the third aspect embodiment.

[0074] In a seventh aspect, a computer program product is provided, and the computer program product includes a computer program. The computer program, when executed by a processor, implements the metadata processing method of any one of the first aspect embodiment, the second aspect embodiment, and the third aspect embodiment.

[0075] The metadata processing method, device, equipment, storage medium, and product provided in the present application include the following steps: in response to receiving an operation request for metadata in a distributed file system sent by a client device, the operation request including a storage path of target metadata and an operation type; determining a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata including target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the target directory group being used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system including a plurality of preset directory groups, and each preset directory group being stored in a corresponding non-uniform memory access node, a plurality of non-uniform memory access nodes being located in corresponding metadata servers, and data in each preset directory group including preset metadata and parent directory timestamp metadata of the preset metadata; performing an operation on the target metadata according to the operation type, and updating the parent directory timestamp metadata of the target metadata in the target directory group. Through the above scheme, the preset metadata and the parent directory timestamp metadata of the preset metadata are stored in the same preset directory group, and the metadata in the same preset directory group is stored in one non-uniform memory access node in units of preset directory groups, so that when the target metadata is operated according to the operation type, communication across non-uniform memory access nodes is not required, and the updating of the parent directory timestamp metadata of the target metadata can be completed in one non-uniform memory access node, thereby improving the access locality of the target metadata operation in the distributed file system, and improving the performance of the metadata server. BRIEF DESCRIPTION OF DRAWINGS

[0076] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles behind the present application.

[0077] Figure 1 A scene diagram of a preset directory group of the metadata processing method provided by the embodiment of the present application;

[0078] Figure 2 A scene diagram of a preset directory group of the metadata processing method provided by the embodiment of the present application;

[0079] Figure 3 A flow diagram of the metadata processing method provided by the embodiment of the present application;

[0080] Figure 4 Another flow diagram of the metadata processing method provided by the embodiment of the present application;

[0081] Figure 5 A flow diagram of creating target metadata of the metadata processing method provided by the embodiment of the present application;

[0082] Figure 6 A flow diagram of deleting target metadata of the metadata processing method provided by the embodiment of the present application;

[0083] Figure 7 A structure diagram of the metadata processing apparatus provided by the embodiment of the present application;

[0084] Figure 8 A structure diagram of the metadata processing device provided by the embodiment of the present application.

[0085] The above-described drawings have shown the specific embodiments of the present application, and the following will have more detailed description. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0086] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. The following description is presented with reference to the drawings, wherein the same reference numerals are used to refer to like or similar elements throughout various drawings and illustrative embodiments. The embodiments described in the following exemplary embodiments are not meant to represent all implementations in accordance with the present application. Rather, they are merely examples in accordance with some aspects of the present application as detailed in the appended claims.

[0087] First, the terms involved in the present application are explained:

[0088] Access locality is a term that depends on the phenomenon of frequent access to the same value or related storage location of the memory access mode. In computer science, access locality is also called the principle of locality.

[0089] Crash consistency refers to a feature that, when a system encounters power failure, crash or failure, etc., can restore the associated data to a consistent state.

[0090] Concurrency control refers to ensuring that, when multiple operations read and write the same data in a distributed file system at the same time, the isolation and consistency of the operations and the consistency of the distributed file system are not destroyed.

[0091] In order to clearly understand the technical solutions of the present application, the prior art solutions will be introduced in detail first.

[0092] In the prior art, a distributed file system is a hierarchical tree structure directory tree composed of directory metadata and file metadata, which organizes and manages files in the entire distributed file system. Further, the metadata under the same level directory is placed in the same non-uniform memory access node in units of directory levels of the distributed file system, that is, the metadata under different level directories may be located in different non-uniform memory access nodes. Different non-uniform memory access nodes may be located in different metadata servers. When the metadata under a certain directory level is updated (such as creation, deletion, etc.), the parent directory metadata of the directory level needs to be modified accordingly.

[0093] If the metadata under different directory levels is located in different non-uniform memory access nodes, when the metadata under a certain directory level is updated (such as creation, deletion, etc.), not only the metadata of the directory level needs to be modified in the non-uniform memory access node where the directory level is located, but also the parent directory metadata needs to be updated in another non-uniform memory access node where the parent directory is located, which results in the need to communicate with at least two non-uniform memory access nodes to complete the corresponding update of the parent directory metadata, and the update efficiency is low, and the access locality is poor, which further reduces the performance of a single metadata server. Even, different non-uniform memory access nodes may be located in different metadata servers, when the non-uniform memory access node where the directory level is located and the non-uniform memory access node where the parent directory of the directory level is located are located in different metadata servers, the access locality is worse, and the update efficiency is lower.

[0094] To solve the problem of low performance of the metadata server caused by network communication across non-uniform memory access nodes for corresponding update of the parent directory metadata, the inventors have found in research that when metadata under a certain directory level is updated (for example, created, deleted, etc.), the main information that needs to be updated is the timestamp information in the parent directory metadata of the directory level, and therefore, to solve the problem, the directory metadata in the distributed file system can be pre-split into directory timestamp metadata and directory access metadata, and at least one of the directory access metadata under the subdirectory of the directory metadata and the file metadata under the subdirectory of the directory metadata is stored in a directory group with the directory timestamp metadata, and then the metadata in the same directory group is stored in the same non-uniform memory access node, so that when metadata under a certain directory level is updated (for example, created, deleted, etc.), the update of the metadata under the directory level and the corresponding update of the timestamp information in the parent directory of the directory level can be completed in the same non-uniform memory access node, without network communication across multiple non-uniform memory access nodes, with better access locality and faster update efficiency, and the performance of the metadata server can be improved.

[0095] Based on the above creative findings, the inventors have proposed the technical solutions of the present application.

[0096] The application scenarios of the metadata processing method provided by the embodiments of the present application are introduced below.

[0097] As shown in Figure 1 The distributed file system includes a plurality of clients 100 and a plurality of metadata servers 200. The metadata server 200 is configured to store the metadata of the distributed file system. The client 100 is capable of communicating with the metadata server 200.

[0098] First, the metadata server 200 first splits the preset directory metadata in the distributed file system into preset directory timestamp metadata and preset directory access metadata. The preset directory timestamp metadata includes the timestamp information of the preset directory metadata, and the preset directory access metadata includes the information of the preset directory metadata except the timestamp information. Second, the metadata server 200 stores at least one of the preset directory access metadata and the preset file metadata under the directory of the preset directory access metadata in a preset directory group with the parent directory timestamp metadata of the preset directory access metadata, and stores at least one of the subfile metadata of the preset directory access metadata and the subdirectory access metadata of the preset directory access metadata in another preset directory group with the preset directory timestamp metadata. Third, the metadata server 200 stores the metadata in the preset directory group in a corresponding non-uniform memory access node.

[0099] Further, when a user triggers an operation request on metadata in the distributed file system through the client 100, the client 100 sends the operation request to the metadata server 200, the metadata server 200 determines a target directory group storing target metadata according to a storage path of the target metadata in the operation request, and then performs an operation on the target metadata and a corresponding update on parent directory timestamp metadata of the target metadata in the target directory group according to an operation type, and feeds back an operation result to the client 100. Thus, the update of the target metadata and the corresponding update of the parent directory timestamp metadata of the target metadata can be completed without crossing multiple non-uniform memory access nodes.

[0100] In the following, the technical solutions shown in the present application are described in detail through specific embodiments. Alternatively, the following embodiments can exist independently or in combination. For the same or similar content, the description is not repeated in different embodiments.

[0101] The metadata processing method provided by the embodiments of the present application is applied to a metadata processing device, and the metadata processing method comprises the following steps:

[0102] S61: splitting preset directory metadata in a distributed file system into preset directory timestamp metadata and preset directory access metadata; wherein the preset directory timestamp metadata comprises timestamp information of the preset directory metadata, and the preset directory access metadata comprises information other than the timestamp information of the preset directory metadata;

[0103] S62: storing at least one of the preset directory access metadata and preset file metadata under a directory where the preset directory access metadata is located, and parent directory timestamp metadata of the preset directory access metadata in a preset directory group;

[0104] S63: storing at least one of child file metadata of the preset directory access metadata and child directory access metadata of the preset directory access metadata, and the preset directory timestamp metadata in another preset directory group;

[0105] S64: storing metadata in the preset directory group in a corresponding non-uniform memory access node.

[0106] The plurality of non-uniform memory access nodes are located in corresponding metadata servers, the preset directory group is used as a target directory group, the target directory group is determined according to a storage path of target metadata in an operation request on metadata in the distributed file system by a client, the target metadata comprises target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the operation on the target metadata and the update of parent directory timestamp metadata of the target metadata in the target directory group are both performed in the non-uniform memory access node corresponding to the target directory group.

[0107] Specifically, the constituent components of the preset directory timestamp metadata are the same as those of the aforementioned target directory timestamp metadata, and the constituent components of the preset directory access metadata are the same as those of the aforementioned target directory access metadata. To avoid redundancy, details are not described herein.

[0108] It can be understood that the preset file metadata under the directory where the preset directory access metadata is located, i.e., the preset file metadata under the directory where the preset directory metadata before splitting is located. The parent directory timestamp metadata of the preset directory access metadata, i.e., the parent directory timestamp metadata of the preset directory metadata before splitting.

[0109] In some embodiments, based on the semantics of the distributed file system, the preset directory metadata in the distributed file system is split into preset directory timestamp metadata and preset directory access metadata.

[0110] For example, as shown in FIG. 1, the preset directory group 0 stores the preset directory access metadata 1, the preset directory group 1 stores the preset directory timestamp metadata 1, the preset directory access metadata 2, the preset directory access metadata 3, the preset file metadata 1 and the preset file metadata 2, the preset directory group 2 stores the preset directory timestamp metadata 2, the preset directory access metadata 4 and the preset file metadata 3, the preset directory group 3 stores the preset directory timestamp metadata 3, and the preset directory group 4 stores the preset directory timestamp metadata 4. Figure 2 It should be noted that, in the example of FIG. 1, the preset directory group 0 should also include at least the parent directory timestamp metadata of the preset directory access metadata 1. Due to the limitation of the length, the parent directory timestamp metadata of the preset directory access metadata 1 is not shown in FIG. 1.

[0111] Figure 2 Figure 2

[0112] ​​​The metadata processing method provided in the embodiments of the present application splits preset directory metadata in a distributed file system into preset directory timestamp metadata and preset directory access metadata; the preset directory timestamp metadata includes timestamp information of the preset directory metadata, and the preset directory access metadata includes information other than the timestamp information of the preset directory metadata; at least one of the preset directory access metadata and preset file metadata under a directory where the preset directory access metadata is located and parent directory timestamp metadata of the preset directory access metadata are stored in a preset directory group; at least one of child file metadata of the preset directory access metadata and child directory access metadata of the preset directory access metadata and the preset directory timestamp metadata are stored in another preset directory group; and the metadata in the preset directory group is stored in a corresponding non-uniform memory access node. The plurality of non-uniform memory access nodes are located in corresponding metadata servers, the preset directory group is used as a target directory group, the target directory group is determined according to a storage path of target metadata in a metadata operation request of a client to the distributed file system, the target metadata includes target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the operation on the target metadata and the update of parent directory timestamp metadata of the target metadata in the target directory group occur in the non-uniform memory access node corresponding to the target directory group. In this way, each parent directory timestamp metadata in the distributed file system and its child directory access metadata and child file metadata are stored in a directory group, which facilitates storing the parent directory timestamp metadata, the child directory access metadata and the child file metadata in the same directory group in the same non-uniform memory access node according to the directory group, and further facilitates completing the update of the parent directory timestamp metadata in the same non-uniform memory access node when the child directory access metadata and / or the child file metadata are updated, thereby improving the performance of the metadata server.

[0113] Optionally, step S64 comprises:

[0114] S641: performing consistency hash calculation on the preset directory group to obtain a hash value of the preset directory group;

[0115] S642: storing the metadata in the preset directory group in a corresponding non-uniform memory access node based on the hash value.

[0116] Specifically, consistency hash calculation is performed on each preset directory group respectively to obtain a hash value of each preset directory group, the hash values of different preset directory groups are different, and then the metadata in the plurality of preset directory groups is stored in a corresponding non-uniform memory access node based on the hash values.

[0117] The metadata processing method provided by the embodiments of the present application performs consistent hash calculation on a preset directory group to obtain a hash value of the preset directory group; and based on the hash value, stores metadata in the preset directory group in a corresponding non-uniform memory access node. In this way, the metadata in the same preset directory group is stored in the same non-uniform memory access node, and when the metadata of a subdirectory and / or the metadata of a subfile is updated, the update of the metadata of a parent directory can be completed in the same non-uniform memory access node, thereby improving the performance of the metadata server.

[0118] Optionally, step S642 comprises:

[0119] S6421: determining the number and the identity of the non-uniform memory access nodes of the metadata server;

[0120] S6422: respectively determining the remainders obtained by dividing the hash values of the preset directory groups by the number;

[0121] S6423: storing the metadata in each preset directory group in a corresponding non-uniform memory access node according to a pre-stored correspondence between the remainders and the identities.

[0122] Specifically, the identity can include a node ID and / or a node name.

[0123] In one example, the hash value of the first preset directory group is 1, the hash value of the second preset directory group is 2, the hash value of the third preset directory group is 3, the number of the non-uniform memory access nodes of the metadata server is 3, and the identities are NUMA node 0, NUMA node 1 and NUMA node 2 respectively, and the correspondence between the remainders and the identities is: remainder 1 corresponds to NUMA node 1; remainder 2 corresponds to NUMA node 2; and remainder 3 corresponds to NUMA node 0.

[0124] For the first preset directory group, since the remainder obtained by dividing 1 by 3 is 1, the metadata in the first preset directory group is stored in NUMA node 1 according to the correspondence between the remainders and the identities.

[0125] For the second preset directory group, since the remainder obtained by dividing 2 by 3 is 2, the metadata in the second preset directory group is stored in NUMA node 2 according to the correspondence between the remainders and the identities.

[0126] For the third preset directory group, since the remainder obtained by dividing 3 by 3 is 0, the metadata in the third preset directory group is stored in NUMA node 0 according to the correspondence between the remainders and the identities.

[0127] The metadata processing method provided in the embodiments of the present application determines the number and the identification of the non-uniform memory access nodes of the metadata server, respectively determines the remainders obtained by dividing the hash values of each preset directory group by the number, and stores the metadata in each preset directory group in a corresponding non-uniform memory access node according to the correspondence between the pre-stored remainders and the identifications. In this way, the multiple preset directory groups can be reasonably distributed and stored in the multiple non-uniform memory access nodes.

[0128] Optionally, step S64 further includes:

[0129] S643: storing the metadata in the preset directory group in a corresponding non-uniform memory access node in the form of a key-value pair.

[0130] The metadata processing method provided in the embodiments of the present application stores the metadata in the preset directory group in a corresponding non-uniform memory access node in the form of a key-value pair. In this way, the metadata in the preset directory group can be conveniently searched based on the key-value pair, and the searching efficiency is improved.

[0131] Optionally, the parent directory timestamp metadata of the preset directory access metadata includes a directory identification.

[0132] The form of the key-value pair is as follows:

[0133] The key-value pair is composed of the directory identification as the key and the parent directory timestamp metadata of the preset directory access metadata as the value.

[0134] The key-value pair is composed of the directory identification and the file naming string of the file metadata under the directory where the preset directory access metadata is located as the key, and the file metadata under the directory where the preset directory access metadata is located as the value.

[0135] The key-value pair is composed of the directory identification and the directory naming string of the preset directory access metadata as the key, and the preset directory access metadata as the value.

[0136] Specifically, the directory identification can be a directory ID.

[0137] Please refer to Figure 3 In one example, Figure 3 Part (a) of FIG. 1 shows part of a directory tree, where directory A has a subdirectory B and a subfile C, A is the directory naming string of directory A, B is the directory naming string of subdirectory B, C is the file naming string of subfile C, and the directory ID of directory A is 1.

[0138] Figure 3Part (b) shows how the directory groups corresponding to the directory tree are stored in the key-value store. Specifically, for directory A, the key of the directory timestamp metadata of directory A is its directory ID, which is 1; the key of the directory access metadata of subdirectory B of directory A consists of the directory ID of directory A and the directory name string of subdirectory B, which is 1 / B; the key of the directory file metadata of subfile C of directory A consists of the directory ID of parent directory A and the file name string of subfile C, which is 1 / C.

[0139] Figure 3 Part (c) shows the metadata of the directory entries corresponding to the parent directory, specifically, based on the preset identifier in the key of the key-value pair (in... Figure 3 The suffix " / " in the key determines the prefix and suffix. First, prefix matching is performed on the parent directory ID to determine the keys associated with the parent directory ID, namely 1, 1 / B, and 1 / C. Next, the d_name members in the metadata of the corresponding directory entry are generated by extracting the suffix, which are the directory name string B and the file name string C. Finally, the ID (d_ino member) and type (d_type member) of each d_name member in the metadata of the directory entry are generated by reading the value corresponding to the key. That is, the ID and type of the directory name string B in the metadata of the directory entry are generated by reading the value corresponding to 1 / B, and the ID and type of the file name string C in the metadata of the directory entry are generated by reading the value corresponding to 1 / C.

[0140] This application provides a metadata processing method that uses a directory identifier as the key and the parent directory timestamp metadata of the preset directory metadata as the value to form a key-value pair; it also uses the directory identifier and the named string of file metadata in the directory where the preset directory access metadata is located as the key, and the file metadata in the same directory as the preset directory access metadata as the value to form a key-value pair; and it further uses the directory identifier and the named string of the preset directory access metadata as the key, and the preset directory access metadata as the value to form a key-value pair. Thus, the preset directory access metadata, the file metadata in the directory where the preset directory access metadata is located, and the parent directory timestamp metadata of the preset directory access metadata are stored in a non-uniform memory access node in the form of key-value pairs, enabling quick retrieval of metadata in the preset directory group by searching the key in the key-value pair.

[0141] It should be noted that the specific values ​​mentioned above are only for illustrating the implementation of this application in detail, and should not be construed as limitations on this application. In other examples, implementation methods, or embodiments, other values ​​may be selected according to this application, and no specific limitations are made here.

[0142] Figure 4Another metadata processing method provided by the embodiments of the present application is shown in the flowchart of FIG. 6. The metadata processing method provided by the embodiments of the present application is applied to a metadata processing device, such as a metadata server, as shown in FIG. 6. The metadata processing method provided by the embodiments of the present application includes the following steps. Figure 4

[0143] S10: In response to receiving an operation request for metadata in the distributed file system sent by the client device, the operation request including a storage path of the target metadata and an operation type.

[0144] The execution subject of the metadata processing method provided by the embodiments of the present application can be a metadata processing device or a metadata processing apparatus provided in the metadata processing device. The metadata processing device can be a metadata server.

[0145] S20: Determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located; the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system includes a plurality of preset directory groups, each preset directory group is stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group includes preset metadata and parent directory timestamp metadata of the preset metadata.

[0146] Specifically, the storage path of the target metadata can include a parent directory ID and a parent directory naming string of the target metadata, and a directory ID and a directory naming string of each level directory between a root directory of the distributed file system and the parent directory of the target metadata. After obtaining the storage path of the target metadata, the metadata server verifies whether a user corresponding to the user ID in the operation request has operation permissions for each level directory and the parent directory between the root directory of the distributed file system and the parent directory of the target metadata according to the user ID directory by directory, and then performs an operation on the target metadata according to the operation type and updates the parent directory timestamp metadata of the target metadata in the target directory group when the user has the corresponding operation permissions.

[0147] ​It can be understood that in another metadata processing method of the embodiments of the present application, as long as there is directory timestamp metadata, a directory group corresponding to the directory timestamp metadata is generated regardless of whether there is subdirectory access metadata and subfile metadata corresponding to the directory timestamp metadata in the distributed file system. If there is subdirectory access metadata and / or subfile metadata corresponding to the directory timestamp metadata before the directory group is generated, after the directory group is generated, the directory group includes both the directory timestamp metadata and the subdirectory access metadata and / or subfile metadata corresponding to the directory timestamp metadata. If there is no subdirectory access metadata and subfile metadata corresponding to the directory timestamp metadata before the directory group is generated, after the directory group is generated, the directory group only includes the directory timestamp metadata, and the directory group includes the metadata created by the user and the directory timestamp metadata corresponding to the metadata created by the user only after the user creates the subdirectory access metadata and / or subfile metadata corresponding to the directory timestamp metadata. Therefore, the parent directory timestamp metadata of the target metadata can be determined according to the storage path of the target metadata, and the target directory group is further determined.

[0148] The distributed file system includes a plurality of preset directory groups, and a preset directory group is stored in a non-uniform memory access node. Different preset directory groups can be stored in the same non-uniform memory access node or different non-uniform memory access nodes, which is not limited herein. The non-uniform memory access node is located in the corresponding metadata server, and the number of non-uniform memory access nodes in each metadata server can be the same or different, which is not limited herein.

[0149] The preset metadata includes preset directory access metadata and / or preset file metadata under the directory where the preset directory access metadata is located. It can be understood that, relative to the parent directory timestamp metadata of the preset metadata, the preset directory access metadata is the subdirectory access metadata of the parent directory timestamp metadata, and the preset file metadata under the directory where the preset directory access metadata is located is the subfile metadata of the parent directory timestamp metadata. The relationship between the parent directory timestamp metadata of the target metadata and the target metadata is the same, and details are not described herein to avoid redundancy.

[0150] S30: performing an operation on the target metadata according to the operation type, and updating the parent directory timestamp metadata of the target metadata in the target directory group.

[0151] The operation type can include a creation operation and a deletion operation, wherein the creation operation can be understood as an operation for creating the target metadata in the distributed file system, and the deletion operation can be understood as an operation for deleting the target metadata in the distributed file system.

[0152] It is worth noting that when modifying the target metadata, the parent directory timestamp metadata of the target metadata does not need to be updated.

[0153] Another metadata processing method provided by the embodiment of the present application comprises the following steps: in response to receiving an operation request for metadata in a distributed file system sent by a client device, the operation request comprising a storage path of target metadata and an operation type; determining a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata comprising target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, the target directory group being used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system comprising a plurality of preset directory groups, each preset directory group being stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes being located in corresponding metadata servers, data in each preset directory group comprising preset metadata and parent directory timestamp metadata of the preset metadata; operating the target metadata according to the operation type and updating the parent directory timestamp metadata of the target metadata in the target directory group. Through the above scheme, the preset metadata and the parent directory timestamp metadata of the preset metadata are stored in the same preset directory group, and the metadata in the same preset directory group is stored in one non-uniform memory access node as a unit, so that when the target metadata is operated according to the operation type, communication across non-uniform memory access nodes is not required, and the updating of the parent directory timestamp metadata of the target metadata can be completed in one non-uniform memory access node, thereby improving the access locality of the target metadata operation in the distributed file system and improving the performance of the metadata server.

[0154] Optionally, step S30 comprises:

[0155] S31: in response to the operation type of the target metadata being target metadata creation, writing the target metadata into the target directory group;

[0156] S32: determining a system timestamp when the target metadata is written as the creation time of the target metadata;

[0157] S33: after the target metadata is written, updating the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata.

[0158] The target file metadata can include permission information of the target file metadata, timestamp information of the target file metadata, data information of the target file metadata, and information of the target file metadata for crash consistency guarantee, wherein the permission information of the target file metadata can include an owner ID of the target file metadata, a user group ID in which the owner is located, and access permissions (e.g., read-only, read-write, executable, etc.) of the target file metadata with respect to an arbitrary ID; the timestamp information of the target file metadata can include a last data modification time, a last access time, and a last metadata modification time corresponding to the target file metadata; the data information of the target file metadata can include an occupied memory size of a file described by the target file metadata and a mapping of a logical position of the described file to a physical position on a disk; and the information of the target file metadata for crash consistency guarantee can include a creation time of the target file metadata, a deletion time of the target file metadata, and a valid flag bit of the target file metadata. In one example, the valid flag bit of the target file metadata is 0, indicating that the target file metadata is invalid; and the valid flag bit of the target file metadata is 1, indicating that the target file metadata is valid.

[0159] The target directory access metadata can include a directory ID, permission information of the target directory access metadata, and information of the target directory access metadata for crash consistency guarantee, wherein the permission information of the target directory access metadata can include an owner ID of the target directory access metadata, a user group ID in which the owner is located, and access permissions (e.g., read-only, read-write, executable, etc.) of the target directory access metadata with respect to an arbitrary ID; and the information of the target directory access metadata for crash consistency guarantee can include a creation time of the target directory access metadata, a deletion time of the target directory access metadata, and a valid flag bit of the target directory access metadata. In one example, the valid flag bit of the target directory access metadata is 0, indicating that the target directory access metadata is invalid; and the valid flag bit of the target directory access metadata is 1, indicating that the target directory access metadata is valid.

[0160] The parent directory timestamp metadata can include timestamp information of the parent directory timestamp metadata, and the timestamp information of the parent directory timestamp metadata can include a last data modification time (English: mtime), a last access time (English: atime), and a last metadata modification time (English: ctime) corresponding to the parent directory timestamp metadata.

[0161] In some embodiments, while writing the target metadata into the target directory group, a creation time of the target metadata is set as a system timestamp at the time of writing the target metadata.

[0162] After the target metadata is written, the content of the last data modification time field and the content of the last metadata modification time field in the parent directory timestamp metadata of the target metadata are both modified to the system timestamp at the time of writing the target metadata.

[0163] Please refer to Figure 5 In one example, the last data modification time in the parent directory timestamp metadata of the parent directory A is 3, the last metadata modification time is 4, and the subdirectory B is written under the parent directory A at time t = 5, then the creation time (English: btime) of the subdirectory B is 5, and the last data modification time in the parent directory timestamp metadata of the parent directory A is updated to 5, and the last metadata modification time is updated to 5. Figure 3 The slash " / " in indicates the root directory, that is, the directory A is located under the root directory.

[0164] The metadata processing method provided by the embodiment of the present application, in response to the operation type of the target metadata being target metadata creation, groups the target directory of the target metadata; determines the system timestamp at the time of writing the target metadata as the creation time of the target metadata; after writing the target metadata, updates the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata. In this way, the target metadata is first written into the target directory group, and then the parent directory timestamp metadata of the target metadata is updated according to the system timestamp at the time of writing the target metadata, and the target metadata is created in this order, which can effectively guarantee the crash consistency of the distributed file system, reduce the crash consistency overhead of the distributed file system, and further improve the performance of the metadata server.

[0165] Optionally, after step S30, the method further comprises:

[0166] S41: in response to detecting that the distributed file system is abnormal and the creation time of the target metadata is later than the corresponding time in the parent directory timestamp metadata of the target metadata, updating the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata.

[0167] Specifically, the distributed file system abnormality can be understood as an unexpected power failure, crash or failure of the distributed file system.

[0168] In some embodiments, after the distributed file system is abnormal, the distributed file system will automatically restart, and after restarting, the metadata in the distributed file system is checked directory by directory, and when it is detected that the creation time of the target metadata is later than the last data modification time and the last metadata modification time in the parent directory timestamp metadata of the target metadata, the content of the last data modification time field and the content of the last metadata modification time field in the parent directory timestamp metadata of the target metadata are modified to the creation time of the target metadata, thereby guaranteeing the crash consistency of the distributed file system.

[0169] It can be understood that when an exception occurs in the distributed file system after the step S32 is performed and before the step S33 is performed, the content of the last data modification time field and the content of the last metadata modification time field in the parent directory timestamp metadata of the target metadata fail to be updated to the creation time of the target metadata, and thus a case that the creation time of the target metadata is later than the corresponding time in the parent directory timestamp metadata of the target metadata occurs.

[0170] Another metadata processing method provided by the embodiment of the present application comprises the following steps of: in response to detecting that an exception occurs in a distributed file system and that the creation time of target metadata is later than the corresponding time in the parent directory timestamp metadata of the target metadata, updating the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata. Thus, even if an exception occurs in the distributed file system during the creation of the target metadata, the associated data can be automatically recovered after restart, thereby ensuring the crash consistency of the distributed file system.

[0171] Optionally, the step S30 comprises the following steps of:

[0172] S33: in response to the operation type being a deletion operation, marking the target metadata as invalid;

[0173] S34: determining the system timestamp when the target metadata is marked as invalid as the deletion time of the target metadata;

[0174] S35: updating the parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata;

[0175] S36: after the parent directory timestamp metadata of the target metadata is updated, deleting the target metadata.

[0176] In some embodiments, the deletion time of the target metadata is set as the system timestamp when the target metadata is marked as invalid at the same time when the target metadata is marked as invalid.

[0177] After the target metadata is marked as invalid, the content of the last data modification time field and the content of the last metadata modification time field in the parent directory timestamp metadata of the target metadata are modified as the system timestamp when the target metadata is marked as invalid, and after the parent directory timestamp metadata of the target metadata is modified, the target metadata is deleted.

[0178] Please refer to Figure 6In one example, the last data modification time in the parent directory timestamp metadata of the parent directory A is 5, the last metadata modification time is 5, the subdirectory B under the parent directory A is marked as invalid at time t = 6, the deletion time (dtime) of the subdirectory B is determined as 6, the last data modification time in the parent directory timestamp metadata of the parent directory A is updated as 6, the last metadata modification time is updated as 6, and finally the subdirectory B is deleted. Figure 4 The slash " / " in the path indicates the root directory, i.e., the directory A is located under the root directory.

[0179] Another metadata processing method provided by the embodiment of the present application comprises the following steps: in response to an operation type being a deletion operation, marking target metadata as invalid; determining a system timestamp when the target metadata is marked as invalid as a deletion time of the target metadata; updating parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata; and deleting the target metadata after updating the parent directory timestamp metadata of the target metadata. In this way, the target metadata is first marked as invalid, then the parent directory timestamp metadata of the target metadata is updated according to the system timestamp when the target metadata is marked as invalid, and finally the target metadata is deleted. In this order, the deletion of the target metadata can effectively guarantee the crash consistency of the distributed file system, reduce the crash consistency overhead of the distributed file system, and further improve the performance of the metadata server.

[0180] Optionally, after step S30, the method further comprises the following steps:

[0181] S42: in response to an abnormality of the distributed file system and the deletion time of the target metadata being later than a corresponding time in the parent directory timestamp metadata of the target metadata, updating the parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata.

[0182] S43: in response to an abnormality of the distributed file system, the deletion time of the target metadata being the same as the corresponding time in the parent directory timestamp metadata of the target metadata, and the target metadata not being deleted, deleting the target metadata.

[0183] In some embodiments, after the distributed file system is abnormal, the distributed file system is automatically restarted, and after the restart, the metadata in the distributed file system is checked directory by directory, when it is detected that there is target metadata marked as invalid in the system and the deletion time of the target metadata is later than the corresponding time in the parent directory timestamp metadata of the target metadata, the content of the last metadata modification time field in the parent directory timestamp metadata of the target metadata is modified as the deletion time of the target metadata, and after the modification of the parent directory timestamp metadata of the target metadata is completed, the target metadata is deleted, so as to ensure the crash consistency of the distributed file system; when it is detected that there is target metadata marked as invalid in the system and the deletion time of the target metadata is the same as the corresponding time in the parent directory timestamp metadata of the target metadata, the target metadata is immediately deleted, so as to ensure the crash consistency of the distributed file system.

[0184] It can be understood that when the distributed file system is abnormal after the execution of the above step S34 and before the execution of the above step S35 and the above step S36, the content of the last metadata modification time field in the parent directory timestamp metadata of the target metadata fails to be updated as the system timestamp when the target metadata is marked as invalid, so that the deletion time of the target metadata is later than the corresponding time in the parent directory timestamp metadata of the target metadata. When the distributed file system is abnormal after the execution of the above step S35 and before the execution of the above step S36, the target metadata marked as invalid has not been deleted, so that the deletion time of the target metadata is the same as the corresponding time in the parent directory timestamp metadata of the target metadata and the target metadata marked as invalid still exists.

[0185] Another metadata processing method provided by the embodiments of the present application responds to the abnormality of the distributed file system and the deletion time of the target metadata being later than the corresponding time in the parent directory timestamp metadata of the target metadata, and updates the parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata. The method responds to the abnormality of the distributed file system, the deletion time of the target metadata being the same as the corresponding time in the parent directory timestamp metadata of the target metadata, and the target metadata not being deleted, and deletes the target metadata. Thus, even if the distributed file system is abnormal during the deletion of the target metadata, the associated data can be automatically recovered and deleted after the restart, so as to ensure the crash consistency of the distributed file system.

[0186] Optionally, at step S30, the method further comprises:

[0187] S51: in response to the simultaneous triggering operations on the target metadata in the distributed file system including at least two, determining the operation types corresponding to the simultaneous triggering operations; wherein the operation types corresponding to the simultaneous triggering operations include at least one of a timestamp update operation, a write operation and a read operation, the timestamp update operation is an operation of updating the parent directory timestamp metadata of the target directory metadata in the creation or deletion process of the target metadata, the write operation is an operation of updating the target metadata except the timestamp update operation, and the read operation is an operation of reading the target metadata;

[0188] S52: in response to the operation types corresponding to the simultaneous triggering operations including the write operation, using a preset read-write lock algorithm to perform concurrent control on the simultaneous triggering operations;

[0189] S53: in response to the simultaneous triggering of multiple timestamp update operations, using a preset lock-free atomic update algorithm to perform concurrent control on the multiple timestamp update operations;

[0190] S54: in response to the simultaneous triggering of the timestamp update operation and the read operation, using a preset optimistic concurrency control algorithm to perform concurrent control on the timestamp update operation and the read operation.

[0191] Specifically, the simultaneous triggering of multiple write operations, the simultaneous triggering of at least one write operation and at least one timestamp update operation, the simultaneous triggering of at least one write operation and at least one read operation, the simultaneous triggering of at least one write operation, at least one timestamp update operation and at least one read operation, the simultaneous triggering of multiple timestamp update operations, the simultaneous triggering of at least one timestamp update operation and at least one read operation, or the simultaneous triggering of multiple read operations can occur, which is not limited herein.

[0192] Please refer to Table 1, which is a concurrent control table of the metadata processing method of the embodiment. Since the simultaneous triggering of multiple read operations does not affect the isolation and uniformity of the operations, the embodiment does not add corresponding concurrent control for the scenario of the simultaneous triggering of multiple read operations.

[0193] Table 1

[0194] Operation type Write operation Timestamp update operation Read operation Write operation Prescribed read-write lock algorithm Prescribed read-write lock algorithm Prescribed read-write lock algorithm Timestamp update operation Prescribed read-write lock algorithm Prescribed lock-free atomic update algorithm Prescribed optimistic concurrency control algorithm Read operation Prescribed read-write lock algorithm Prescribed optimistic concurrency control algorithm --

[0195] For the scenarios of the simultaneous triggering of multiple write operations, the simultaneous triggering of at least one write operation and at least one timestamp update operation, the simultaneous triggering of at least one write operation and at least one read operation, and the simultaneous triggering of at least one write operation, at least one timestamp update operation and at least one read operation, since the operation types all include the write operation, the preset read-write lock algorithm is used to perform concurrent control on the above scenarios including the write operation, so as to ensure the isolation and uniformity of the operations.

[0196] For the scenario that multiple timestamp update operations are triggered simultaneously, a preset lock-free atomic update algorithm is adopted to perform concurrent control on the multiple timestamp update operations, so as to ensure the isolation and unity of the operations.

[0197] For the scenario that at least one timestamp update operation and at least one read operation are triggered simultaneously, a preset optimistic concurrency control algorithm is adopted to perform concurrent control on the timestamp update operation and the read operation, so as to ensure the isolation and unity of the operations.

[0198] It should be noted that in the present application, "multiple" should be understood as two, three or more than three, which is not limited herein.

[0199] Another metadata processing method provided by the embodiment of the present application, in response to the operation triggered simultaneously on the target metadata in the distributed file system including at least two, determining the operation type corresponding to the operation triggered simultaneously; wherein the operation type corresponding to the operation triggered simultaneously includes at least one of the timestamp update operation, the write operation and the read operation, the timestamp update operation is an operation of updating the parent directory timestamp metadata of the target directory metadata in the creation or deletion process of the target metadata, the write operation is an operation of updating the target metadata except the timestamp update operation, and the read operation is an operation of reading the target metadata; in response to the operation type corresponding to the operation triggered simultaneously including the write operation, a preset read-write lock algorithm is adopted to perform concurrent control on the operation triggered simultaneously; in response to the timestamp update operation and the timestamp update operation triggered simultaneously, a preset lock-free atomic update algorithm is adopted to perform concurrent control on the timestamp update operation and the timestamp update operation; in response to the timestamp update operation and the read operation triggered simultaneously, a preset optimistic concurrency control algorithm is adopted to perform concurrent control on the timestamp update operation and the read operation. Thus, for various scenarios, various concurrency control strategies are adopted, which can improve the concurrency degree of the distributed file system under the shared directory and further improve the performance of the metadata server.

[0200] Optionally, the operation type corresponding to the operation triggered simultaneously includes at least one of the timestamp update operation, the read operation and the write operation. Step S52 includes:

[0201] S521: obtaining the current lock state in the preset read-write lock algorithm;

[0202] S522: in response to the current lock state being that the write operation holds the write lock, executing the write operation, and after the write operation is executed, configuring the timestamp update operation and / or the read operation to hold the read lock, so as to execute the timestamp update operation and / or the read operation; or

[0203] S523: in response to the current lock state being the read lock held by the timestamp update operation and / or the read operation, performing the timestamp update operation and / or the read operation, and after the timestamp update operation and / or the read operation is performed, configuring the write operation to hold the write lock so as to perform the write operation.

[0204] Specifically, the write lock and the read lock are mutually exclusive, that is, when the write lock is held, no operation can hold the read lock; and when the read lock is held, no operation can hold the write lock. In the preset read-write lock algorithm, an operation that holds a lock first is executed before an operation that holds a lock later.

[0205] It can be understood that, since the write lock can be held by only one operation at the same time, and when an operation holds the write lock, other operations (even if they are operations of the same type) cannot hold the write lock, when multiple write operations are triggered at the same time, only one of the write operations is executed at the same time, and after the execution of the write operation, the next write operation holds the write lock to be executed, thereby ensuring the isolation of multiple write operations. Since the read lock can be held by multiple operations at the same time, when multiple operations holding the read lock are executed, further concurrent control needs to be performed in combination with the preset lock-free atomic update algorithm and / or the preset optimistic concurrency control algorithm.

[0206] The other metadata processing method provided by the embodiment of the present application comprises the following steps: obtaining a current lock state in a preset read-write lock algorithm; in response to the current lock state being the write lock held by the write operation, performing the write operation, and after the write operation is performed, configuring the timestamp update operation and / or the read operation to hold the read lock so as to perform the timestamp update operation and / or the read operation; or in response to the current lock state being the read lock held by the timestamp update operation and / or the read operation, performing the timestamp update operation and / or the read operation, and after the timestamp update operation and / or the read operation is performed, configuring the write operation to hold the write lock so as to perform the write operation. Thus, when the operation types corresponding to the operations triggered at the same time comprise at least one of the timestamp update operation, the read operation and the write operation, the preset read-write lock algorithm can be used to isolate at least one of the timestamp update operation and the read operation from the write operation, thereby ensuring the isolation and uniformity of the operations.

[0207] Optionally, the parent directory timestamp metadata of the target metadata comprises a last data modification time and a last metadata modification time. Step S53 comprises:

[0208] S531: obtaining a to-be-updated time corresponding to the multiple timestamp update operations, and a last data modification time and a last metadata modification time recorded in the parent directory timestamp metadata; wherein the to-be-updated time is a creation time of the target metadata or a deletion time of the target metadata;

[0209] S532: updating the maximum value among the to-be-updated time, the last data modification time and the last metadata modification time in the parent directory timestamp metadata of the target metadata by using the plurality of timestamp update operations.

[0210] Since the parent directory timestamp metadata of the target metadata needs to be modified according to the creation time of the target metadata or the deletion time of the target metadata every time the target metadata is created or deleted, when a plurality of users create and / or delete different target metadata under the same parent directory, there may be a case that a plurality of to-be-updated times want to modify the parent directory timestamp metadata of the target metadata at the same time, that is, a plurality of timestamp update operations are triggered at the same time.

[0211] In one example, when a plurality of timestamp update operations are triggered at the same time, the to-be-updated time corresponding to each timestamp update operation and the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata are obtained respectively, the to-be-updated time corresponding to one of the timestamp update operations is compared with the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata, the maximum value among the three is determined, and the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata are updated according to the maximum value; then, the to-be-updated time corresponding to another of the timestamp update operations is compared with the last data modification time and the last metadata modification time recorded in the updated parent directory timestamp metadata, the maximum value among the three is determined, and the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata are updated again according to the maximum value; and so on, until the to-be-updated time corresponding to the last of the timestamp update operations is compared with the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata updated last time, the maximum value among the three is determined, and the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata are updated again according to the maximum value, so as to complete the plurality of timestamp update operations. It can be understood that when a plurality of times are compared, the latest time is the maximum value, for example, 23:55 of the previous day, 23:59 of the previous day and 00:00 of the next day are compared, and the maximum value is 00:00 of the next day.

[0212] It can be understood that when multiple timestamp update operations are triggered at the same time, if the latest to-be-updated time corresponding to the multiple timestamp update operations is later than the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata, the content of the last data modification time field and the content of the last metadata modification time field recorded in the parent directory timestamp metadata are both modified to the latest to-be-updated time corresponding to the multiple timestamp update operations; if the latest to-be-updated time corresponding to the multiple timestamp update operations is earlier than or equal to the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata, the content of the last data modification time field and the content of the last metadata modification time field recorded in the parent directory timestamp metadata remain unchanged.

[0213] Another metadata processing method provided by the embodiment of the present application comprises the following steps: obtaining a to-be-updated time corresponding to one of the timestamp update operations, a to-be-updated time corresponding to another of the timestamp update operations, and a last metadata modification time recorded in parent directory timestamp metadata; wherein the to-be-updated time is a creation time of target metadata or a deletion time of the target metadata; and updating the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata by using the maximum value of the to-be-updated time corresponding to one of the timestamp update operations, the to-be-updated time corresponding to another of the timestamp update operations, and the last metadata modification time. Thus, for the scenario in which multiple timestamp update operations are triggered at the same time, the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata is updated according to the maximum value of the to-be-updated time corresponding to the multiple timestamp update operations and the last metadata modification time, which can improve the concurrency of the distributed file system and thus improve the performance of the metadata server.

[0214] Optionally, the parent directory timestamp metadata comprises a last metadata modification time. Step S54 comprises:

[0215] S541: performing a timestamp update operation;

[0216] S542: reading the parent directory timestamp metadata before performing a read operation to obtain a first last metadata modification time;

[0217] S543: reading the parent directory timestamp metadata after performing the read operation to obtain a second last metadata modification time;

[0218] S544: in response to the first last metadata modification time being different from the second last metadata modification time, re-executing the read operation until the first last metadata modification time is the same as the second last metadata modification time, and feeding back target metadata read by the read operation.

[0219] Specifically, when the timestamp update operation and the read operation are triggered at the same time, the timestamp update operation and the read operation can be executed at the same time, the timestamp update operation can be executed normally by using an atomic update instruction, and the last metadata modification time recorded in the parent directory timestamp metadata can be regarded as the version number of the target metadata to be read for the read operation, that is, the last metadata modification time recorded in the parent directory timestamp metadata is acquired once before and after reading the target metadata, if the last metadata modification times acquired twice are the same, it is determined that the target metadata is not updated during the reading process, and the target metadata read can be used, and then the target metadata read can be fed back to the client; if the last metadata modification times acquired twice are different, it is determined that the target metadata is updated during the reading process, and the target metadata read is not available, and then the read operation needs to be executed again.

[0220] Another metadata processing method provided by the embodiment of the present application comprises the following steps: performing a timestamp update operation; reading parent directory timestamp metadata before performing a read operation to obtain a first last metadata modification time; reading the parent directory timestamp metadata after performing the read operation to obtain a second last metadata modification time; and in response to the first last metadata modification time being different from the second last metadata modification time, re-executing the read operation until the first last metadata modification time is the same as the second last metadata modification time, and feeding back target metadata read by the read operation. Thus, for the scenario that the timestamp update operation and the read operation are triggered at the same time, the preset optimistic concurrency control algorithm is used to perform concurrency control on the timestamp update operation and the read operation, so as to ensure the isolation and unity of the operations.

[0221] Another metadata processing method provided by the embodiment of the present application comprises the following steps: performing a timestamp update operation; reading parent directory timestamp metadata before performing a read operation to obtain a first last metadata modification time; reading the parent directory timestamp metadata after performing the read operation to obtain a second last metadata modification time; and in response to the first last metadata modification time being different from the second last metadata modification time, re-executing the read operation until the first last metadata modification time is the same as the second last metadata modification time, and feeding back target metadata read by the read operation. Thus, for the scenario that the timestamp update operation and the read operation are triggered at the same time, the preset optimistic concurrency control algorithm is used to perform concurrency control on the timestamp update operation and the read operation, so as to ensure the isolation and unity of the operations.

[0222] S71: sending an operation request for metadata in a distributed file system to a metadata processing device.

[0223] The operation request includes a storage path of target metadata and an operation type, the operation request is used to trigger the metadata processing device to determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system includes a plurality of preset directory groups, each of which is stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group includes preset metadata and parent directory timestamp metadata of the preset metadata; and the operation request is further used to trigger the metadata processing device to perform an operation on the target metadata according to the operation type and update the parent directory timestamp metadata of the target metadata in the target directory group.

[0224] In another metadata processing method provided by the embodiment of the present application, an operation request for metadata in a distributed file system is sent to a metadata processing device. The operation request includes a storage path of target metadata and an operation type, the operation request is used to trigger the metadata processing device to determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system includes a plurality of preset directory groups, each of which is stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group includes preset metadata and parent directory timestamp metadata of the preset metadata; and the operation request is further used to trigger the metadata processing device to perform an operation on the target metadata according to the operation type and update the parent directory timestamp metadata of the target metadata in the target directory group. In this way, the preset metadata and the parent directory timestamp metadata of the preset metadata are stored in the same preset directory group, and the metadata in the same preset directory group is stored in one non-uniform memory access node, so that when the operation on the target metadata according to the operation type is performed, communication across non-uniform memory access nodes is not needed, and the update of the parent directory timestamp metadata of the target metadata can be completed in one non-uniform memory access node, thereby improving the access locality of the operation on the target metadata in the distributed file system and improving the performance of the metadata server.

[0225] The metadata processing device provided by the embodiment of the present application includes a splitting module, a first storage module and a second storage module.

[0226] The splitting module is configured to split preset directory metadata in the distributed file system into preset directory timestamp metadata and preset directory access metadata; the preset directory timestamp metadata includes timestamp information of the preset directory metadata, and the preset directory access metadata includes information other than the timestamp information of the preset directory metadata.

[0227] The first storage module is configured to store at least one of the preset directory access metadata and preset file metadata under a directory of the preset directory access metadata and parent directory timestamp metadata of the preset directory access metadata in a preset directory group; and store at least one of child file metadata of the preset directory access metadata and child directory access metadata of the preset directory access metadata and the preset directory timestamp metadata in another preset directory group.

[0228] The second storage module is configured to store metadata in the preset directory group in a corresponding non-uniform memory access node. The plurality of non-uniform memory access nodes are located in corresponding metadata servers. The preset directory group is used as a target directory group. The target directory group is determined according to a storage path of target metadata in a metadata operation request of a client to the distributed file system. The target metadata includes target directory access metadata and / or target file metadata under a directory of the target directory access metadata. The operation on the target metadata and the update of parent directory timestamp metadata of the target metadata in the target directory group are both performed in the non-uniform memory access node corresponding to the target directory group.

[0229] Optionally, the second storage module is specifically configured to:

[0230] perform consistent hashing calculation on the preset directory group to obtain a hash value of the preset directory group;

[0231] store the metadata in the preset directory group in a corresponding non-uniform memory access node based on the hash value.

[0232] Optionally, the second storage module is specifically configured to:

[0233] determine a number and an identifier of the non-uniform memory access nodes of the metadata server;

[0234] determine remainders obtained by dividing the hash values of the preset directory groups by the number, respectively;

[0235] store the metadata in each of the preset directory groups in a corresponding non-uniform memory access node according to a pre-stored correspondence between the remainders and the identifiers.

[0236] Optionally, the second storage module is specifically configured to:

[0237] The metadata in the preset directory group is stored in a corresponding non-uniform memory access node in the form of a key-value pair.

[0238] Optionally, the parent directory timestamp metadata of the preset directory access metadata comprises a directory identifier, and the key-value pair is in the following form:

[0239] The directory identifier is taken as a key, and the parent directory timestamp metadata of the preset directory access metadata is taken as a value to form a key-value pair.

[0240] The directory identifier and a file naming string of the file metadata under the directory where the preset directory access metadata is located are taken as keys, and the file metadata under the directory where the preset directory access metadata is located is taken as a value to form a key-value pair.

[0241] The directory identifier and a directory naming string of the preset directory access metadata are taken as keys, and the preset directory access metadata is taken as a value to form a key-value pair.

[0242] The metadata processing apparatus provided in the embodiments of the present application can implement the technical solutions shown in the metadata processing method embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0243] Figure 7 The structure diagram of another metadata processing apparatus provided in the embodiments of the present application is provided, and the metadata processing apparatus provided in the embodiments of the present application is applied to a metadata processing device, as shown in the following Figure 7 The metadata processing apparatus 400 provided in the embodiments of the present application comprises a response module 401, a determination module 402 and a processing module 403.

[0244] The response module 401 is configured to respond to the operation request for the metadata in the distributed file system sent by the client device, and the operation request comprises a storage path of the target metadata and an operation type.

[0245] The determination module 402 is configured to determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata comprises target directory access metadata and / or target file metadata under the directory where the target directory access metadata is located, the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system comprises a plurality of preset directory groups, each preset directory group is stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes are located in corresponding metadata servers, and the data in each preset directory group comprises preset metadata and parent directory timestamp metadata of the preset metadata.

[0246] The processing module 403 is configured to perform the operation on the target metadata according to the operation type, and update the parent directory timestamp metadata of the target metadata in the target directory group.

[0247] Optionally, the processing module 403 is specifically configured to:

[0248] In response to the operation type of the target metadata being a creation operation, write the target metadata into the target directory group; and determine a system timestamp when the target metadata is written as a creation time of the target metadata.

[0249] After the target metadata is written, update the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata.

[0250] Optionally, the metadata processing apparatus 400 further comprises a first recovery module.

[0251] The first recovery module is configured to, in response to detecting that the distributed file system is abnormal and the creation time of the target metadata is later than a corresponding time in the parent directory timestamp metadata of the target metadata, update the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata.

[0252] Optionally, the processing module 403 is specifically configured to:

[0253] In response to the operation type being a deletion operation, mark the target metadata as invalid.

[0254] Determine a system timestamp when the target metadata is marked as invalid as a deletion time of the target metadata.

[0255] Update the parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata; and after the parent directory timestamp metadata of the target metadata is updated, delete the target metadata.

[0256] Optionally, the metadata processing apparatus 400 further comprises a second recovery module, and the second recovery module is configured to:

[0257] In response to the distributed file system being abnormal, the deletion time of the target metadata being later than a corresponding time in the parent directory timestamp metadata of the target metadata, update the parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata.

[0258] In response to the distributed file system being abnormal, the deletion time of the target metadata being the same as the corresponding time in the parent directory timestamp metadata of the target metadata, and the target metadata not being deleted, delete the target metadata.

[0259] Optionally, the metadata processing apparatus 400 further comprises a concurrency control module, and the concurrency control module is configured to:

[0260] In response to the simultaneous triggering of the operations on the target metadata in the distributed file system including at least two, the operation types corresponding to the simultaneously triggered operations are determined; wherein the operation types corresponding to the simultaneously triggered operations include at least one of a timestamp update operation, a write operation and a read operation, the timestamp update operation is an operation of updating the parent directory timestamp metadata of the target directory metadata in the creation or deletion process of the target metadata, the write operation is an operation of updating the target metadata except the timestamp update operation, and the read operation is an operation of reading the target metadata;

[0261] In response to the operation types corresponding to the simultaneously triggered operations including the write operation, a preset read-write lock algorithm is used to perform concurrent control on the simultaneously triggered operations;

[0262] In response to the simultaneous triggering of the multiple timestamp update operations, a preset lock-free atomic update algorithm is used to perform concurrent control on the multiple timestamp update operations;

[0263] In response to the simultaneous triggering of the timestamp update operation and the read operation, a preset optimistic concurrent control algorithm is used to perform concurrent control on the timestamp update operation and the read operation.

[0264] Optionally, the operation types corresponding to the simultaneously triggered operations include at least one of the timestamp update operation and the read operation and the write operation. The concurrent control module is specifically configured to:

[0265] acquire a current lock state in the preset read-write lock algorithm;

[0266] In response to the current lock state being that the write operation holds a write lock, the write operation is executed, and after the execution of the write operation is completed, the timestamp update operation and / or the read operation is configured to hold a read lock, so as to execute the timestamp update operation and / or the read operation;

[0267] or in response to the current lock state being that the timestamp update operation and / or the read operation holds a read lock, the timestamp update operation and / or the read operation is executed, and after the execution of the timestamp update operation and / or the read operation is completed, the write operation is configured to hold a write lock, so as to execute the write operation.

[0268] Optionally, the parent directory timestamp metadata of the target metadata includes a last data modification time and a last metadata modification time. The concurrent control module is specifically configured to:

[0269] acquire a to-be-updated time corresponding to the multiple timestamp update operations and the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata; wherein the to-be-updated time is a creation time of the target metadata or a deletion time of the target metadata;

[0270] The maximum value of the to-be-updated time, the last data modification time and the last metadata modification time corresponding to the plurality of timestamp update operations is used to update the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata.

[0271] Optionally, the parent directory timestamp metadata comprises the last metadata modification time. The concurrency control module is specifically configured to:

[0272] perform a timestamp update operation;

[0273] read the parent directory timestamp metadata before performing the read operation to obtain a first last metadata modification time;

[0274] read the parent directory timestamp metadata after performing the read operation to obtain a second last metadata modification time;

[0275] in response to the first last metadata modification time being different from the second last metadata modification time, re-perform the read operation until the first last metadata modification time is the same as the second last metadata modification time, and feed back the target metadata read by the read operation.

[0276] The metadata processing apparatus 400 provided by the embodiment of the present application can execute the technical solutions shown in the above another metadata processing method embodiment, and has similar implementation principles and technical effects, which will not be described here again.

[0277] The another metadata processing apparatus provided by the embodiment of the present application is applied to a client device, and comprises a sending module.

[0278] The sending module is configured to send an operation request for metadata in a distributed file system to a metadata processing device.

[0279] The operation request comprises a storage path of the target metadata and an operation type, and is configured to trigger the metadata processing device to determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata comprises target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the target directory group is configured to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system comprises a plurality of preset directory groups, each of which is stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group comprises preset metadata and parent directory timestamp metadata of the preset metadata; and the operation request is further configured to trigger the metadata processing device to perform an operation on the target metadata according to the operation type, and to update the parent directory timestamp metadata of the target metadata in the target directory group.

[0280] The metadata processing apparatus provided in the embodiments of the present application can execute the technical solutions shown in the above-mentioned another metadata processing method embodiment, and the implementation principles and technical effects are similar, which will not be described here again.

[0281] Figure 8 The structural schematic diagram of the metadata processing device provided in the embodiments of the present application is shown in FIG. 5, which comprises a processor 501 and a memory 502 connected with the processor 501; the memory 502 stores computer execution instructions; the processor 501 executes the computer execution instructions stored in the memory 502 to realize the above-mentioned one metadata processing method and another metadata processing method. Figure 8

[0282] For example, when the processor 501 executes the computer execution instructions stored in the memory 502, the following steps of the metadata processing method are realized:

[0283] S10: In response to receiving the operation request for the metadata in the distributed file system sent by the client device, the operation request comprises the storage path of the target metadata and the operation type;

[0284] S20: Determine the target directory group storing the target metadata according to the storage path of the target metadata; the target metadata comprises the target directory access metadata and / or the target file metadata under the directory where the target directory access metadata is located, and the target directory group is used to store the target metadata and the parent directory timestamp metadata of the target metadata; the distributed file system comprises a plurality of preset directory groups, each preset directory group is stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes are located in corresponding metadata servers, and the data in each preset directory group comprises preset metadata and the parent directory timestamp metadata of the preset metadata;

[0285] S30: Perform the operation on the target metadata according to the operation type, and update the parent directory timestamp metadata of the target metadata in the target directory group.

[0286] In Figure 8 In the corresponding embodiments, the program can comprise program codes, and the program codes comprise computer execution instructions. The memory 502 can comprise a high-speed RAM memory, and can also comprise a non-volatile memory such as at least one disk memory.

[0287] ​The memory 502 and the processor 501 are connected through a bus. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 8 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0288] The metadata processing device 500 provided by the embodiments of the present application can execute the technical solutions shown in the above-mentioned one metadata processing method embodiment and another metadata processing method embodiment, and the implementation principles and technical effects are similar, and will not be repeated here.

[0289] The client device provided by the embodiments of the present application includes a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory to implement the above-mentioned another metadata processing method.

[0290] For example, when the processor executes the computer execution instructions stored in the memory, the following steps of the metadata processing method are implemented: S71: sending an operation request for metadata in a distributed file system to a metadata processing device.

[0291] The operation request includes a storage path of target metadata and an operation type, and the operation request is used to trigger the metadata processing device to determine a target directory group storing the target metadata according to the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata under a directory where the target directory access metadata is located, and the target directory group is used to store the target metadata and parent directory timestamp metadata of the target metadata; the distributed file system includes a plurality of preset directory groups, each preset directory group is stored in a corresponding non-uniform memory access node, the plurality of non-uniform memory access nodes are located in corresponding metadata servers, and data in each preset directory group includes preset metadata and parent directory timestamp metadata of the preset metadata; and the operation request is further used to trigger the metadata processing device to operate the target metadata according to the operation type and update the parent directory timestamp metadata of the target metadata in the target directory group.

[0292] The client device provided by the embodiments of the present application can execute the technical solutions shown in the above-mentioned another metadata processing method embodiment, and the implementation principles and technical effects are similar, and will not be repeated here.

[0293] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used for implementing the metadata processing method of any of the foregoing embodiments.

[0294] The embodiment of the present application provides a computer program product, and the computer program product comprises a computer program. When the computer program is executed by a processor, the computer program implements the metadata processing method of any of the foregoing embodiments.

[0295] It should be noted that, for the foregoing method embodiments, in order to simply describe, the foregoing method embodiments are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0296] It should be further noted that, although each step in the flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated in this paper, the execution of these steps has no strict order limitation, and these steps can be executed in other order. Moreover, at least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0297] It should be understood that the above-mentioned device embodiments are only schematic, and the device of the present application can also be realized by other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and another division mode can be used in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0298] In addition, each functional unit / module in each embodiment of the present application can be integrated in one unit / module, or each unit / module can exist physically, or two or more units / modules can be integrated together. The integrated unit / module can be realized in the form of hardware or in the form of software program module.

[0299] If the integrated units / modules are implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processor can be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit can be any appropriate magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0300] If the integrated units / modules are implemented in the form of software program modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0301] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. The technical features of the above embodiments can be combined arbitrarily, and in order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

Claims

1. A metadata processing method, applied to a metadata processing device, characterized in that, The method includes: In response to receiving a request from a client device to operate on metadata in a distributed file system, the request including the storage path of the target metadata and the operation type; The target directory group for storing the target metadata is determined based on the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata in the directory where the target directory access metadata is located; the target directory group is used to store the target metadata and the parent directory timestamp metadata of the target metadata; the distributed file system includes multiple preset directory groups, and for each preset directory group, it is stored in a corresponding non-unified memory access node, and the multiple non-unified memory access nodes are located in the corresponding metadata server; the data in each preset directory group includes preset metadata and the parent directory timestamp metadata of the preset metadata. Perform operations on the target metadata according to the operation type, and update the parent directory timestamp metadata of the target metadata in the target directory group; When performing operations on the target metadata according to the operation type and updating the parent directory timestamp metadata of the target metadata in the target directory group, the method further includes: in response to the fact that there are at least two operations that are triggered simultaneously on the target metadata in the distributed file system, determining the operation type corresponding to the simultaneous triggered operations; wherein, the operation type corresponding to the simultaneous triggered operations includes at least one of a timestamp update operation, a write operation, and a read operation, the timestamp update operation is an operation that updates the parent directory timestamp metadata of the target metadata during the creation or deletion of the target metadata, the write operation is an operation that updates the target metadata other than the timestamp update operation, and the read operation is an operation that reads the target metadata; In response to the operation type corresponding to the simultaneous triggered operation, which includes a write operation, a preset read-write lock algorithm is used to perform concurrency control on the simultaneous triggered operation; In response to the simultaneous triggering of multiple timestamp update operations, a preset lock-free atomic update algorithm is used to perform concurrent control on the multiple timestamp update operations; In response to the simultaneous triggering of the timestamp update operation and the read operation, a preset optimistic concurrency control algorithm is used to perform concurrency control on the timestamp update operation and the read operation.

2. The method according to claim 1, characterized in that, The step of operating on the target metadata according to the operation type and updating the parent directory timestamp metadata of the target metadata in the target directory group includes: In response to the target metadata operation type being target metadata creation, the target metadata is written to the target directory group; The system timestamp at which the target metadata was written is determined as the creation time of the target metadata; After writing the target metadata, update the parent directory timestamp metadata of the target metadata according to the creation time of the target metadata.

3. The method according to claim 2, characterized in that, After performing operations on the target metadata according to the operation type and updating the parent directory timestamp metadata of the target metadata in the target directory group, the method further includes: In response to the detection of an anomaly in the distributed file system and the creation time of the target metadata being later than the corresponding time in the parent directory timestamp metadata of the target metadata, the parent directory timestamp metadata of the target metadata is updated according to the creation time of the target metadata.

4. The method according to claim 1, characterized in that, The step of operating on the target metadata according to the operation type and updating the parent directory timestamp metadata of the target metadata in the target directory group includes: In response to the operation type being a deletion operation, the target metadata is marked as invalid; The system timestamp when the target metadata was marked as invalid is determined as the deletion time of the target metadata; Update the parent directory timestamp metadata of the target metadata according to the deletion time of the target metadata; After updating the parent directory timestamp metadata of the target metadata, delete the target metadata.

5. The method according to claim 4, characterized in that, After performing operations on the target metadata according to the operation type and updating the parent directory timestamp metadata of the target metadata in the target directory group, the method further includes: In response to an abnormality in the distributed file system, and the deletion time of the target metadata is later than the corresponding time in the timestamp metadata of the parent directory of the target metadata, the timestamp metadata of the parent directory of the target metadata is updated according to the deletion time of the target metadata; In response to an abnormality in the distributed file system, and the deletion time of the target metadata is the same as the corresponding time in the timestamp metadata of the parent directory of the target metadata, and the target metadata has not been deleted, the target metadata is deleted.

6. The method according to claim 1, characterized in that, The operation types triggered simultaneously include timestamp update operation, at least one of the read operations, and write operation; The method of using a preset read-write lock algorithm to control the concurrency of the simultaneously triggered operations includes: Get the current lock state in the preset read-write lock algorithm; In response to the current lock state being a write operation holding a write lock, the write operation is executed, and after the write operation is completed, the timestamp update operation and / or the read operation is configured to hold a read lock so that the timestamp update operation and / or the read operation can be executed; or In response to the current lock state being that the timestamp update operation and / or the read operation holds a read lock, the timestamp update operation and / or the read operation are executed, and after the timestamp update operation and / or the read operation are completed, the write operation is configured to hold a write lock so as to execute the write operation.

7. The method according to claim 1, characterized in that, The parent directory timestamp metadata of the target metadata includes the last data modification time and the last metadata modification time; The concurrent control of multiple timestamp update operations using a preset lock-free atomic update algorithm includes: Obtain the pending update time corresponding to multiple timestamp update operations, the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata; wherein, the pending update time is the creation time of the target metadata or the deletion time of the target metadata; The maximum value among the pending update time, the last data modification time, and the last metadata modification time corresponding to multiple timestamp update operations is used to update the last data modification time and the last metadata modification time recorded in the parent directory timestamp metadata of the target metadata.

8. The method according to claim 1, characterized in that, The parent directory timestamp metadata includes the last metadata modification time; The method of using a preset optimistic concurrency control algorithm to control the concurrency of the timestamp update operation and the read operation includes: Perform the timestamp update operation; Before performing the read operation, read the parent directory timestamp metadata to obtain the first last metadata modification time; After performing the read operation, the parent directory timestamp metadata is read to obtain the second last metadata modification time; In response to the difference between the first last metadata modification time and the second last metadata modification time, the read operation is re-executed until the first last metadata modification time and the second last metadata modification time are the same, and then the target metadata read by the read operation is fed back.

9. A metadata processing method, applied to a metadata processing device, characterized in that, The method includes: The metadata of a preset directory in the distributed file system is split into preset directory timestamp metadata and preset directory access metadata; wherein, the preset directory timestamp metadata includes the timestamp information of the preset directory metadata, and the preset directory access metadata includes the information of the preset directory metadata other than the timestamp information; Store at least one of the preset directory access metadata and the preset file metadata in the directory where the preset directory access metadata is located, along with the parent directory timestamp metadata of the preset directory access metadata, in a preset directory group. At least one of the subfile metadata of the preset directory access metadata and the subdirectory access metadata of the preset directory access metadata is stored together with the preset directory timestamp metadata in another preset directory group; The metadata in the preset directory group is stored in a corresponding non-uniform memory access node; In this system, multiple non-uniform memory access nodes are located in corresponding metadata servers. The preset directory group is used as the target directory group. The target directory group is determined according to the storage path of the target metadata in the client's operation request for metadata in the distributed file system. The target metadata includes target directory access metadata and / or target file metadata in the directory where the target directory access metadata is located. The operation on the target metadata and the update of the parent directory timestamp metadata of the target metadata in the target directory group both occur in the non-uniform memory access node corresponding to the target directory group. When performing operations on the target metadata and updating the parent directory timestamp metadata of the target metadata in the target directory group, the method further includes: in response to the simultaneous triggering of at least two operations on the target metadata in the distributed file system, determining the operation type corresponding to the simultaneous triggering operations; wherein the operation type corresponding to the simultaneous triggering operations includes at least one of timestamp update operation, write operation, and read operation, the timestamp update operation is an operation to update the parent directory timestamp metadata of the target metadata during the creation or deletion of the target metadata, the write operation is an operation to update the target metadata other than the timestamp update operation, and the read operation is an operation to read the target metadata; In response to the operation type corresponding to the simultaneous triggered operation, which includes a write operation, a preset read-write lock algorithm is used to perform concurrency control on the simultaneous triggered operation; In response to the simultaneous triggering of multiple timestamp update operations, a preset lock-free atomic update algorithm is used to perform concurrent control on the multiple timestamp update operations; In response to the simultaneous triggering of the timestamp update operation and the read operation, a preset optimistic concurrency control algorithm is used to perform concurrency control on the timestamp update operation and the read operation.

10. The method according to claim 9, characterized in that, The step of storing the metadata in the preset directory group in a corresponding non-uniform memory access node includes: Perform a consistent hash calculation on the preset directory group to obtain the hash value of the preset directory group; Based on the hash value, the metadata in the preset directory group is stored in a corresponding non-uniform memory access node.

11. The method according to claim 10, characterized in that, The step of storing the metadata in the preset directory group in a corresponding non-uniform memory access node based on the hash value includes: Determine the number and identifiers of the non-uniform memory access nodes of the metadata server; Determine the remainder obtained by dividing the hash value of each preset directory group by the number of groups; Based on the pre-stored correspondence between remainders and identifiers, the metadata in each of the preset directory groups is stored in a corresponding non-uniform memory access node.

12. The method according to claim 9, characterized in that, The step of storing the metadata in the preset directory group in a corresponding non-uniform memory access node further includes: The metadata in the preset directory group is stored in the form of key-value pairs in a corresponding non-uniform memory access node.

13. The method according to claim 12, characterized in that, The parent directory timestamp metadata of the preset directory access metadata includes the directory identifier; The key-value pairs are in the following form: A key-value pair is formed by using the directory identifier as the key and the parent directory timestamp metadata of the preset directory access metadata as the value. The key-value pair consists of the file name string of the directory identifier and the file metadata of the file in the directory where the preset directory access metadata is located, as the key, and the file metadata of the file in the directory where the preset directory access metadata is located as the value. The directory identifier and the directory name string of the preset directory access metadata are used as keys, and the preset directory access metadata is used as the value to form a key-value pair.

14. A metadata processing method, applied to a client device, characterized in that, The method includes: Send a request to the metadata processing device to operate on the metadata in the distributed file system; The operation request includes the storage path and operation type of the target metadata. The operation request triggers the metadata processing device to determine the target directory group for storing the target metadata based on the storage path. The target metadata includes target directory access metadata and / or target file metadata in the directory containing the target directory access metadata. The target directory group stores the target metadata and its parent directory timestamp metadata. The distributed file system includes multiple preset directory groups. For each preset directory group, data is stored in a corresponding non-uniform memory access node. These multiple non-uniform memory access nodes are located in corresponding metadata servers. The data in each preset directory group includes preset metadata and its parent directory timestamp metadata. The operation request also triggers the metadata processing device to operate on the target metadata according to the operation type and update the parent directory timestamp metadata of the target metadata in the target directory group. When the metadata processing device operates on the target metadata according to the operation type and updates the parent directory timestamp metadata of the target metadata in the target directory group, the method further includes: the metadata processing device, in response to at least two operations simultaneously triggered on the target metadata in the distributed file system, determines the operation type corresponding to the simultaneously triggered operations; wherein, the operation type corresponding to the simultaneously triggered operations includes at least one of timestamp update operation, write operation, and read operation; the timestamp update operation is an operation to update the parent directory timestamp metadata of the target metadata during the creation or deletion of the target metadata; the write operation is an operation to update the target metadata other than the timestamp update operation; and the read operation is an operation to read the target metadata; in response to the operation type corresponding to the simultaneously triggered operations including a write operation, a preset read-write lock algorithm is used to perform concurrency control on the simultaneously triggered operations; in response to multiple timestamp update operations being triggered simultaneously, a preset lock-free atomic update algorithm is used to perform concurrency control on the multiple timestamp update operations; and in response to the timestamp update operation and the read operation being triggered simultaneously, a preset optimistic concurrency control algorithm is used to perform concurrency control on the timestamp update operation and the read operation.

15. A metadata processing apparatus, applied to a metadata processing device, characterized in that, The apparatus is used to implement the metadata processing method according to any one of claims 1-8, the apparatus comprising: The response module is used to respond to a request from a client device to operate on metadata in a distributed file system, wherein the operation request includes the storage path of the target metadata and the operation type. The determination module is used to determine the target directory group for storing the target metadata based on the storage path of the target metadata; the target metadata includes target directory access metadata and / or target file metadata in the directory where the target directory access metadata is located; the target directory group is used to store the target metadata and the parent directory timestamp metadata of the target metadata; the distributed file system includes multiple preset directory groups, and for each preset directory group, it is stored in a corresponding non-unified memory access node, and the multiple non-unified memory access nodes are located in the corresponding metadata server; the data in each preset directory group includes preset metadata and the parent directory timestamp metadata of the preset metadata. The processing module is used to operate on the target metadata according to the operation type and update the parent directory timestamp metadata of the target metadata in the target directory group.

16. A metadata processing device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the metadata processing method as described in any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the metadata processing method as described in any one of claims 1-14.

18. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the metadata processing method according to any one of claims 1-14.

Images