Distributed file system metadata management method, device, equipment and storage medium

Abstract

The present application relates to the field of cloud storage, and specifically discloses a distributed file system metadata management method, apparatus, device and storage medium. The distributed file system includes at least one metadata node, and the method includes: obtaining metadata to be stored and the data type of the metadata to be stored, and determining index entries and directory entries of the metadata to be stored; determining a target metadata node where a parent index entry of a parent directory of the metadata to be stored is located from at least one of the metadata nodes, and storing the directory entry in the target metadata node; determining an index storage node in at least one of the metadata nodes according to the data type of the metadata to be stored, and storing the index entry in the index storage node.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a distributed file system metadata management method, device, equipment and storage medium. Background Art

[0002] Distributed file systems are the most complex among distributed storage systems because they need to provide various metadata information and management interfaces such as permissions, creation and update time, file size, and file locks. Most of the functions required by these system calls need to be implemented by metadata, which also makes metadata a hot spot for file system access. Therefore, whether metadata is well implemented is crucial to the success of the system. Summary of the invention

[0003] The present application provides a distributed file system metadata management method, apparatus, device and storage medium to improve access convenience.

[0004] In a first aspect, the present application provides a distributed file system metadata management method, wherein the distributed file system includes at least one metadata node, and the method includes:

[0005] Acquire metadata to be stored and a data type of the metadata to be stored, and determine an index entry and a directory entry of the metadata to be stored;

[0006] Determine a target metadata node where a parent index entry of a parent directory of the metadata to be stored is located from at least one of the metadata nodes, and store the directory entry in the target metadata node;

[0007] An index storage node is determined in at least one of the metadata nodes according to the data type of the metadata to be stored, and the index entry is stored in the index storage node.

[0008] In a second aspect, the present application further provides a distributed file system metadata management device, the distributed file system comprising at least one metadata node, the device comprising:

[0009] An entry determination module, used to obtain metadata to be stored and a data type of the metadata to be stored, and determine an index entry and a directory entry of the metadata to be stored;

[0010] A directory saving module, used to determine a target metadata node where a parent index entry of a parent directory of the metadata to be stored is located from at least one of the metadata nodes, and save the directory entry in the target metadata node;

[0011] The index storage module is used to determine an index storage node in at least one of the metadata nodes according to the data type of the metadata to be stored, and store the index entry in the index storage node.

[0012] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the distributed file system metadata management method as described above when executing the computer program.

[0013] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the distributed file system metadata management method as described above.

[0014] The present application discloses a distributed file system metadata management method, device, equipment and storage medium, which obtains the metadata to be stored and the data type of the metadata to be stored, and determines the index entry and directory entry of the metadata to be stored; then determines the target metadata node where the parent index entry of the parent directory of the metadata to be stored is located from at least one metadata node, and stores the directory entry in the target metadata node; then determines the index storage node in at least one metadata node according to the data type of the metadata to be stored, and stores the index entry in the index storage node. The directory entry of the metadata to be stored is stored on the same metadata node, while the index entry of the data to be stored can be distributed to other metadata nodes for storage, thereby forming a horizontal expansion based on the directory, which is convenient for traversal operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 is a schematic flow chart of a distributed file system metadata management method provided in an embodiment of the present application;

[0017] Figure 2 It is a schematic diagram of a process for locating a metadata node where a directory is located provided in an embodiment of the present application;

[0018] Figure 3 It is a schematic flow chart of the steps of determining an index storage node according to the data type of metadata to be stored provided by an embodiment of the present application;

[0019] Figure 4 It is a schematic diagram of the separation of namespaces within a data instance provided in an embodiment of the present application;

[0020] Figure 5 It is a schematic diagram of a process for locating a metadata node where a file is located provided in an embodiment of the present application;

[0021] Figure 6 A schematic block diagram of a distributed file system metadata management device provided in an embodiment of the present application;

[0022] Figure 7 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0025] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0026] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] The embodiments of the present application provide a distributed file system metadata management method, device, computer equipment and storage medium. The distributed file system metadata management method uses a KV database as an underlying storage engine to build a metadata service cluster of a parallel file system, thereby effectively avoiding the restrictions of the local file system on the metadata operations of the distributed file system in terms of function and performance. At the same time, the distributed file system metadata management method also ensures a balance between the scalability of the distributed file system and the performance of local metadata operations.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] See also Figure 1 , Figure 1 This is a schematic flow chart of a distributed file system metadata management method provided in an embodiment of the present application. The distributed file system metadata management method splits and sorts the system by directory, thereby storing metadata in the distributed file system separately, giving full play to the scalability of the distributed system, while also taking into account the locality of operations such as traversal.

[0030] In one embodiment, the distributed file system includes at least one metadata node. In the specific implementation process, three processes can be used to build the entire distributed file system, including MON (monitor), MDS (metadata service) and CSS (Chunk storage service). Among them, MON is the registration center and coordinator of the entire cluster. It needs to ensure high availability, and all processes need to be interconnected with MON for heartbeat. MDS is the metadata service process, and CSS is the data sharding storage process of the file.

[0031] Among them, the metadata managed by MDS is stored in the local KV database and is persisted in the form of Key-Value. In addition, a client kernel module is provided to mount the file system on the head server. Users can directly operate it as a local file system. The kernel module interacts with the backend cluster through the network.

[0032] When the MDS starts, it will actively send a registration request to the MON. After receiving the registration request from the MDS, the MON will save the MDS status information. When the cluster is started for the first time, the first registered MDS is recorded as the metadata node where the metadata of the root directory ' / ' of the distributed file system is located.

[0033] like Figure 1 As shown, the distributed file system metadata management method specifically includes: step S101 to step S103.

[0034] S101: Obtain metadata to be stored and the data type of the metadata to be stored, and determine index entries and directory entries of the metadata to be stored.

[0035] First, for the metadata to be stored, the data type of the metadata to be stored can be determined, and the data type includes directory data and file data. The directory data refers to the file directory in the distributed file system, such as the root directory, parent directory, subdirectory, etc. The file data refers to the file data in each directory in the distributed file system.

[0036] The metadata node MDS divides the metadata information of the metadata to be stored into two categories, index entries and directory entries. The directory entries and index entries of the metadata to be stored need to be stored in different namespaces for storage. The index entry can be a dir-entry, and the index entry can be an inode.

[0037] The index entry includes data such as the attributes, permissions, and last update time of the file or directory, and the directory entry is used to record the location information of the file or directory and to index the location of the index entry of the file or directory.

[0038] S102: Determine a target metadata node where a parent index entry of a parent directory of the metadata to be stored is located from at least one of the metadata nodes, and save the directory entry in the target metadata node.

[0039] When storing the metadata to be stored, the location of the parent directory entry of the parent directory of the metadata to be stored is determined based on the metadata to be stored, and then a query is performed at the location of the parent directory entry until the metadata node where the parent index entry is located is determined. The metadata node is the target metadata node.

[0040] For example Figure 2 As shown in Figure 2 It is a flowchart for locating the metadata node where the directory is located.

[0041] like Figure 2 As shown, the location of the root directory ' / ' of the mounted file system can be first obtained through MON, that is, the metadata node where the root directory entry of the root directory is located, that is, MDS1. Since the index entries of the files and directories under the directory are on the same metadata node as the directory entry of the parent directory, a query request can be sent to MDS1 to query the location of the parent index entry of the parent directory under the root directory, and then the metadata node where the directory entry of the parent directory is located is obtained through the parent index entry, that is, MDS2. After that, MDS2 can be used as the target metadata node.

[0042] After determining the target metadata node where the parent index entry of the parent directory of the metadata to be stored is located from at least one metadata node, the directory entry can be stored in the target metadata node.

[0043] In addition, before saving the directory entry for storing data, the directory entry for storing metadata can also be named. When naming, it can be named according to the preset naming rules. For example, the naming rule for the key of the directory entry can be:

[0044] The index entry of the parent directory to which the metadata to be stored belongs plus the actual file name or directory name.

[0045] Among them, using the index entry of the parent directory to which the metadata to be stored belongs during naming can facilitate prefix traversal to implement functions such as readdir. In the specific implementation process, the index entry can be a snow-flake type uuid.

[0046] S103: Determine an index storage node in at least one of the metadata nodes according to the data type of the metadata to be stored, and store the index entry in the index storage node.

[0047] Before saving the index entry for storing metadata, the index entry for storing metadata can also be named. When naming, it can be named according to the preset naming rules. For example, the naming rule for the key of the index entry can be:

[0048] If the data type of the metadata to be stored is directory data, then the key of the index entry of the directory data is directly the index entry of the directory data; if the data type of the metadata to be stored is file data, then the key of the index entry of the directory data is the index entry of the parent directory plus the index entry of the file data.

[0049] The index storage node is determined according to the data type of the metadata to be stored. The determined index storage node can be either a target metadata node storing the directory entry of the metadata to be stored or other metadata nodes in the distributed file system.

[0050] In one embodiment, if Figure 3 As shown, the step of determining the index storage node according to the data type of the metadata to be stored may include: S1031. If the data type of the metadata to be stored is directory data, determining the index storage node based on the node surplus of at least one of the metadata nodes; S1032. If the data type of the metadata to be stored is file data, determining the target metadata node as the index storage node.

[0051] If the data type of the metadata to be stored is directory data, the index storage node can be determined based on the node remaining capacity of each metadata node in the distributed file system, that is, the node margin. In the specific implementation process, according to the different remaining capacities of each metadata node, a metadata node with more remaining capacity can be randomly selected as an index storage node.

[0052] For example, MON can receive capacity information of all metadata nodes through heartbeats. Each metadata node periodically downloads the capacity information to the local. According to the remaining capacity, a metadata node with more remaining capacity is randomly selected as an index storage node.

[0053] If the data type of the metadata to be stored is file data, the index entry of the metadata to be stored may be directly stored in the determined target metadata node.

[0054] For file data, the index entries and directory entries of the file data are stored on the same metadata node as the parent directory of the metadata to be stored. The index entries of the directory may be allocated to other metadata nodes for storage. That is, the index entries of a directory and all the files stored under it are on the same metadata node, and the index entries of other subdirectories under the directory may be allocated to other metadata nodes for storage, thus forming a horizontal expansion based on directories.

[0055] In one embodiment, when creating a new directory, the client directly sends a message to the metadata node where the parent directory is located, and then the metadata node selects the metadata node where the new directory index entry is located (MON receives the capacity information of all MDSs through heartbeats, and each metadata node periodically downloads the capacity information to the local area, and randomly selects one of the metadata nodes with more remaining capacity according to the remaining capacity), and then sends a request to create an index entry to the selected metadata node. After the creation is successful, the metadata node where the parent directory is located receives the response and then creates a directory entry for the new directory locally. After the creation is completed, it returns a success message to the client.

[0056] In one embodiment, a metadata node may include multiple data instances. When an index storage node includes multiple data instances, the step of storing the index entry in the index storage node may include: performing a hash calculation according to a preset rule based on the data type of the metadata to be stored, determining a storage instance of the index entry from the multiple data instances, and storing the index entry based on the storage instance.

[0057] You can configure multiple data instances, or db instances, to be enabled simultaneously on a single metadata node to improve the concurrency of write operations. When saving the index entries of the metadata to be stored, you also need to select a save location from multiple data instances. In the specific implementation process, you can perform hash selection based on different rules according to the data type of the data to be stored, so as to determine the save instance, and then save the index entries of the metadata to be stored in the save instance.

[0058] In one embodiment, the step of performing a hash calculation according to a preset rule based on the data type of the metadata to be stored may include: if the data type of the metadata to be stored is directory data, performing a hash calculation according to the index entry of the metadata to be stored; if the data type of the metadata to be stored is file data, performing a hash calculation according to the index entry of the parent directory of the metadata to be stored.

[0059] If the data type of the metadata to be stored is directory data, hashing can be performed based on the index entry of the metadata to be stored, thereby selecting which data instance to save it on. If the data type of the metadata to be stored is file data, hashing can be performed based on the index entry of the parent directory of the metadata to be stored, thereby selecting which instance to save it on. This ensures that all directory entries under the same parent directory fall on the same instance, facilitating subsequent traversal operations.

[0060] In one embodiment, the distributed file system metadata management method may further include: obtaining an extended attribute entry of the data to be stored, and storing the extended attribute entry in the index storage node.

[0061] For each extended attribute of file data or directory data, an extended attribute entry is added to the data instance and saved in the index saving node. When the file data or directory data is deleted, the corresponding extended attribute entry needs to be deleted together.

[0062] Among them, index entries and directory entries can be stored in isolation by different namespaces when they are saved. In order to realize the extended attribute feature of the POSIX file system, several more namespaces can be set separately for the index entries of file data and the index entries of directory data to store the extended attribute entries of directory data and the extended attribute entries of file data respectively.

[0063] like Figure 4The figure shows a schematic diagram of the separation of the namespace within a data instance, where dir-inodes represents the index entry of the directory data, dentries represents the directory entry of the directory data, file-inodes represents the index entry of the file data, dir-inode-xattrs represents the extended attribute entry of the directory data, and filer-inodes-xattrs represents the extended attribute entry of the file data.

[0064] Before the extended attribute entry of the data to be stored is saved, it can be named according to a certain naming rule. For example, the naming rule may be: the index entry of the parent directory to which the metadata to be stored belongs plus the extended attribute name.

[0065] After the metadata to be stored is partitioned and stored, when you need to read the file data, such as Figure 5 The figure shows a flowchart of locating the metadata node where the file is located.

[0066] First, the client obtains the location of the root directory ' / ' of the mounted file system through MON, that is, the metadata node where the index entry of the root directory is located, that is, MDS1. Since the directory entry of the parent directory and the index entry of the root directory are on the same MDS, a query request is sent to MDS1 to query the location of the directory entry of the root directory, that is, MDS2. Then, a query request is sent to MDS2 to find the directory entry of the subdirectory on MDS2, and it is learned that the location information of the index entry of the subdirectory is on MDS3. Since the index entries of a directory and all the files stored under it are on the same metadata node, the metadata information of the target file under the subdirectory can be obtained by communicating with MDS3.

[0067] When performing an operation to traverse a directory, such as executing ls on the command line or executing the readdir function in a program, the client will find the MDS where the target directory is located and send a traversal request to the MDS. Since the directory entries of all files and directories under a directory are on the same data instance of the same MDS as the parent directory, prefix traversal is performed directly on the data instance.

[0068] To improve traversal efficiency, a certain number of entries are pre-read during traversal. After returning to the client, they are returned upward according to the user's use. After they are used up, they are requested to continue traversal from MDS. Each time the MDS pre-reads, it records the offset (auto-incremental uuid) of the last entry. The next time it traverses, it carries the offset returned during the last traversal. MDS locates the end of the last traversal based on the offset, and then continues to traverse backwards.

[0069] In one embodiment, the distributed file system metadata management method may also include the following steps: dividing at least one of the metadata nodes into multiple redundant groups, and determining the master node in each of the redundant groups; and forwarding data to other metadata nodes in the redundant group based on the master node for data processing.

[0070] At least one metadata node can be divided into multiple redundancy groups by manual setting or automatic setting, and a metadata node is designated as a master node in each redundancy group, and the remaining metadata nodes in the redundancy group are slave nodes. In the specific implementation process, the redundancy groups do not cross, that is, each metadata node can only be in one redundancy group, and each redundancy group can include two metadata nodes, one metadata node is a master node, and the other metadata node is a slave node.

[0071] Then, based on the master node, the data is forwarded to other metadata nodes in the redundancy group for data processing, where data processing includes receiving and updating data.

[0072] In the specific implementation process, the client can download node status information from MON at regular intervals, and communicate with the master node during communication. When the update message reaches the master node, the master node forwards the update message to the slave node. When the slave node completes the processing of the update message, the master node processes the message locally. When the master node loses heartbeat connection with MON, MON can update the status of the master node to offline and replace the master node of the redundant group. The client then synchronizes the status and changes the interactive peer.

[0073] When a metadata node in a redundant group is offline, MON will downgrade the data security of the redundant group. When the offline metadata node comes online again, the currently online master node needs to be responsible for data synchronization with the newly online slave node. When all MDSs in a redundant group are offline, MON will record the last online MDS and prohibit the group from continuing to provide services before the last online MDS comes online, so as to avoid the loss of authoritative data caused by the offline of all redundant group members.

[0074] In one embodiment, the step of forwarding data to other metadata nodes in the redundant group for data processing based on the master node may include: obtaining an update cache operation and a snapshot file; sending the snapshot file to other metadata nodes in the redundant group based on the master node, so that the other metadata nodes in the redundant group load the snapshot file; sending the update cache operation to other metadata nodes in the redundant group based on the master node, so that the update cache operation is synchronized to the other metadata nodes in the redundant group.

[0075] When data synchronization is started, an update operation buffer queue is created to cache all update operations initiated by all businesses after the synchronization is started. In addition, a snapshot is generated for the data instance at the current time. After the snapshot is generated, the snapshot file is directly sent to the slave node. The slave node is responsible for loading the snapshot locally. After the slave node loads the snapshot, the master node suspends the business and synchronizes the update business operations in the update buffer queue to the slave node. After the synchronization is completed, the queue is deleted, the business is resumed, and the full data synchronization of the entire slave node is completed.

[0076] The distributed file system metadata management method provided by the above embodiment obtains the metadata to be stored and the data type of the metadata to be stored, and determines the index entry and directory entry of the metadata to be stored; then determines the target metadata node where the parent index entry of the parent directory of the metadata to be stored is located from at least one metadata node, and saves the directory entry in the target metadata node; then determines the index storage node in at least one metadata node according to the data type of the metadata to be stored, and saves the index entry in the index storage node. The directory entry of the metadata to be stored is stored on the same metadata node, while the index entry of the data to be stored can be distributed to other metadata nodes for storage, thereby forming a horizontal expansion based on directories, which is convenient for traversal operations.

[0077] See also Figure 6 , Figure 6 The embodiment of the present application also provides a schematic block diagram of a distributed file system metadata management device, which is used to execute the above-mentioned distributed file system metadata management method. The distributed file system metadata management device can be configured in a server or a terminal.

[0078] The server may be an independent server or a server cluster. The terminal may be an electronic device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, a wearable device, etc.

[0079] like Figure 6 As shown, the distributed file system metadata management device 200 includes: an entry determination module 201, a directory storage module 202 and an index storage module 203.

[0080] The entry determination module 201 is used to obtain metadata to be stored and the data type of the metadata to be stored, and determine index entries and directory entries of the metadata to be stored.

[0081] The directory saving module 202 is used to determine the target metadata node where the parent index entry of the parent directory of the metadata to be stored is located from at least one of the metadata nodes, and save the directory entry in the target metadata node.

[0082] The index storage module 203 is used to determine an index storage node in at least one of the metadata nodes according to the data type of the metadata to be stored, and store the index entry in the index storage node.

[0083] In one embodiment, the index storage module includes a first determining submodule and a second determining submodule.

[0084] Among them, the first determination submodule is used to determine the index storage node based on the node margin of at least one of the metadata nodes if the data type of the metadata to be stored is directory data; the second determination submodule is used to determine the target metadata node as the index storage node if the data type of the metadata to be stored is file data.

[0085] In one embodiment, the metadata node includes multiple data instances, and the index storage module includes an instance determination submodule.

[0086] Among them, the instance determination submodule is used to perform hash calculation according to preset rules based on the data type of the metadata to be stored, determine the saved instance of the index entry from multiple data instances, and save the index entry based on the saved instance.

[0087] In one embodiment, the instance determination submodule includes a first determination submodule and a second determination submodule.

[0088] Among them, the first determination submodule is used to perform hash calculation according to the index entry of the metadata to be stored if the data type of the metadata to be stored is directory data; the second determination submodule is used to perform hash calculation according to the index entry of the parent directory of the metadata to be stored if the data type of the metadata to be stored is file data.

[0089] In one embodiment, the distributed file system metadata management device further includes a redundant grouping module and a data processing module.

[0090] Among them, the redundant grouping module is used to divide at least one of the metadata nodes into multiple redundant groups and determine the master node in each of the redundant groups; the data processing module is used to forward data to other metadata nodes in the redundant group for data processing based on the master node.

[0091] In one embodiment, the data grouping module includes a file acquisition submodule, a file forwarding submodule and an operation forwarding submodule.

[0092] Among them, the file acquisition submodule is used to obtain update cache operations and snapshot files; the file forwarding submodule is used to send the snapshot file to other metadata nodes in the redundant group based on the master node, so that the other metadata nodes in the redundant group load the snapshot file; the operation forwarding submodule is used to send the update cache operation to other metadata nodes in the redundant group based on the master node, so that the update cache operation is synchronized to other metadata nodes in the redundant group.

[0093] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the distributed file system metadata management device and each module described above can refer to the corresponding process in the aforementioned distributed file system metadata management method embodiment, and will not be repeated here.

[0094] The above-mentioned distributed file system metadata management device can be implemented in the form of a computer program. The computer program can be used in Figure 7 Runs on the computer device shown.

[0095] See also Figure 7 , Figure 7 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.

[0096] See also Figure 7 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0097] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any distributed file system metadata management method.

[0098] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0099] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any distributed file system metadata management method.

[0100] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0101] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0102] In one embodiment, the distributed file system includes at least one metadata node, and the processor is used to run a computer program stored in the memory to implement the following steps:

[0103] Acquire metadata to be stored and a data type of the metadata to be stored, and determine an index entry and a directory entry of the metadata to be stored;

[0104] Determine a target metadata node where a parent index entry of a parent directory of the metadata to be stored is located from at least one of the metadata nodes, and store the directory entry in the target metadata node;

[0105] An index storage node is determined in at least one of the metadata nodes according to the data type of the metadata to be stored, and the index entry is stored in the index storage node.

[0106] In one embodiment, when the processor implements determining the index storage node in at least one of the metadata nodes according to the data type of the metadata to be stored, the processor is used to implement:

[0107] If the data type of the metadata to be stored is directory data, determining an index storage node based on a node margin of at least one metadata node;

[0108] If the data type of the metadata to be stored is file data, the target metadata node is determined to be an index storage node.

[0109] In one embodiment, the index storage node includes multiple data instances, and the processor, when implementing the storing of the index entry in the index storage node, is configured to implement:

[0110] A hash calculation according to a preset rule is performed based on the data type of the metadata to be stored, a saved instance of the index entry is determined from a plurality of the data instances, and the index entry is saved based on the saved instance.

[0111] In one embodiment, when implementing the hash calculation based on the data type of the metadata to be stored according to the preset rule, the processor is used to implement:

[0112] If the data type of the metadata to be stored is directory data, performing a hash calculation according to the index entry of the metadata to be stored;

[0113] If the data type of the metadata to be stored is file data, a hash calculation is performed according to the index entry of the parent directory of the metadata to be stored.

[0114] In one embodiment, the processor is further configured to implement:

[0115] The extended attribute entry of the data to be stored is obtained, and the extended attribute entry is stored in the index storage node.

[0116] In one embodiment, the processor is configured to implement:

[0117] Divide at least one of the metadata nodes into a plurality of redundancy groups, and determine a master node in each of the redundancy groups;

[0118] Data processing is performed based on forwarding data from the master node to other metadata nodes in the redundant group.

[0119] In one embodiment, when the processor implements the data processing based on forwarding data from the master node to other metadata nodes in the redundancy group, it is used to implement:

[0120] Get update cache operations and snapshot files;

[0121] Sending the snapshot file to other metadata nodes in the redundancy group based on the master node, so that the other metadata nodes in the redundancy group load the snapshot file;

[0122] Based on the master node, the update cache operation is sent to other metadata nodes in the redundant group, so that the update cache operation is synchronized to other metadata nodes in the redundant group.

[0123] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any distributed file system metadata management method provided in the embodiment of the present application.

[0124] The computer-readable storage medium may be an internal storage unit of the computer device described in the above embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0125] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A distributed file system metadata management method, characterized in that: The distributed file system includes at least one metadata node, and the method includes: Acquire metadata to be stored and a data type of the metadata to be stored, and determine an index entry and a directory entry of the metadata to be stored; Determine a target metadata node where a parent index entry of a parent directory of the metadata to be stored is located from at least one of the metadata nodes, and store the directory entry in the target metadata node; Determining an index storage node in at least one of the metadata nodes according to the data type of the metadata to be stored, including: if the data type of the metadata to be stored is directory data, determining the index storage node based on the node margin of at least one of the metadata nodes; if the data type of the metadata to be stored is file data, determining the target metadata node as the index storage node; and storing the index entry in the index storage node, the index storage node including multiple data instances, and storing the index entry in the index storage node, including: performing hash calculation according to preset rules based on the data type of the metadata to be stored, including: if the data type of the metadata to be stored is directory data, performing hash calculation according to the index entry of the metadata to be stored; if the data type of the metadata to be stored is file data, performing hash calculation according to the index entry of the parent directory of the metadata to be stored; determining the storage instance of the index entry from the multiple data instances, and saving the index entry based on the storage instance; wherein, when saving the index entry of the metadata to be stored, it is necessary to perform hash selection according to different rules according to the data type of the metadata to be stored to determine the storage instance, and save the index entry of the metadata to be stored in the storage instance.

2. The distributed file system metadata management method according to claim 1, characterized in that: The method further comprises: The extended attribute entry of the data to be stored is obtained, and the extended attribute entry is stored in the index storage node.

3. The distributed file system metadata management method according to claim 1, characterized in that: The method comprises: Divide at least one of the metadata nodes into a plurality of redundancy groups, and determine a master node in each of the redundancy groups; Data processing is performed based on forwarding data from the master node to other metadata nodes in the redundant group.

4. The distributed file system metadata management method according to claim 3, characterized in that: The data processing based on forwarding data from the master node to other metadata nodes in the redundancy group includes: Get update cache operations and snapshot files; Sending the snapshot file to other metadata nodes in the redundancy group based on the master node, so that the other metadata nodes in the redundancy group load the snapshot file; Based on the master node, the update cache operation is sent to other metadata nodes in the redundant group, so that the update cache operation is synchronized to other metadata nodes in the redundant group.

5. A distributed file system metadata management device, characterized in that: The distributed file system includes at least one metadata node, and the device includes: An entry determination module, used to obtain metadata to be stored and a data type of the metadata to be stored, and determine an index entry and a directory entry of the metadata to be stored; A directory saving module, used to determine a target metadata node where a parent index entry of a parent directory of the metadata to be stored is located from at least one of the metadata nodes, and save the directory entry in the target metadata node; The index saving module is used to determine an index saving node in at least one of the metadata nodes according to the data type of the metadata to be stored, including: if the data type of the metadata to be stored is directory data, then determine the index saving node based on the node margin of at least one of the metadata nodes; if the data type of the metadata to be stored is file data, then determine the target metadata node as the index saving node; and save the index entry in the index saving node, the index saving node includes multiple data instances, and the saving of the index entry in the index saving node includes: performing a hash calculation according to a preset rule based on the data type of the metadata to be stored, including: if the data type of the metadata to be stored is directory data, then perform a hash calculation according to the index entry of the metadata to be stored; if the data type of the metadata to be stored is file data, then perform a hash calculation according to the index entry of the parent directory of the metadata to be stored; determine the saving instance of the index entry from the multiple data instances, and save the index entry based on the saving instance; wherein, when saving the index entry of the metadata to be stored, it is necessary to perform a hash selection according to different rules according to the data type of the metadata to be stored to determine the saving instance, and save the index entry of the metadata to be stored in the saving instance.

6. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the distributed file system metadata management method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the distributed file system metadata management method according to any one of claims 1 to 4.

Images