Metadata updating method and device, node equipment and computer readable storage medium

Abstract

The invention relates to the technical field of distributed file systems, and provides a metadata updating method and device, node equipment and a computer readable storage medium, the metadata updating method and device, the node equipment and the computer readable storage medium are applied to each node equipment in a plurality of node equipment in a distributed file system, and the method comprises the steps of obtaining a file index node identifier of a target file needing to create a hard link; updating metadata of the target file according to the file index node identifier, and updating metadata of an associated directory related to a hard link of the target file; the updated metadata of the target file does not have an index node identifier of the associated catalog and the target catalog, or the metadata of the target file, the metadata related to the target file in the associated catalog and the metadata related to the target file in the target catalog are stored in the same node device in a plurality of node devices; the target directory is a directory specified when the target file is created. According to the method, cross-node transactions can be reduced while hard links are supported, and the system efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed file systems, and in particular to a metadata updating method, apparatus, node device and computer-readable storage medium. Background Art

[0002] The metadata of a high-performance distributed file system is usually organized in the form of Key-Value pairs, and the metadata is also stored in the KV database in the form of Key-Value pairs. In order to ensure the reliability of metadata modification, the database transaction model is usually used to ensure that metadata modification meets ACID, namely Atomicity, Consistency, Isolation, and Durability.

[0003] The metadata of a distributed file system is usually stored in the form of Key-Value (KV pairs) in a distributed manner in the KV databases of multiple nodes. When updating files in a distributed file system, it usually involves the simultaneous modification of the corresponding metadata in multiple KV databases. At this time, cross-node distributed transactions are required to ensure that the modification of the corresponding metadata in multiple KV databases meets ACID. However, cross-node distributed transactions will affect the overall performance of the distributed file system.

[0004] In order to avoid the impact of cross-node distributed transactions on overall performance, the existing implementation method is to bind the file and the directory to which it belongs, and determine the node device that stores the file metadata according to the directory to which it belongs, thereby avoiding the modification of metadata across nodes. However, this implementation method requires that a file can only belong to one directory. When creating a hard link for a file, a file may belong to multiple directories. Therefore, this method cannot support hard links. Summary of the invention

[0005] The purpose of the present invention is to provide a metadata updating method, apparatus, node device and computer-readable storage medium, which can not only support hard links, but also avoid modifying metadata through cross-node transactions when creating hard links, ultimately achieving both support for hard links and reducing the impact of cross-node transactions on the overall performance of the distributed file system.

[0006] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a metadata updating method, which is applied to each node device of a plurality of node devices in a distributed file system, and the method comprises: Get the file index node identifier of the target file for which a hard link needs to be created; updating the metadata of the target file according to the file index node identifier, and updating the metadata of the associated directory related to the hard link of the target file; The index node identifiers of the associated directory and the target directory do not exist in the metadata of the updated target file, or the metadata of the target file, the metadata of the associated directory related to the target file, and the metadata of the target directory related to the target file are stored in the same node device among the multiple node devices, and the target directory is the directory specified when the target file is created.

[0007] In an optional implementation, the step of updating the metadata of the target file according to the file index node identifier includes: Querying metadata of the target file according to the file index node identifier, where the metadata of the target file includes a directory index node identifier of the target directory; Updating the directory index node identifier to an invalid value; The metadata of the target file is migrated to a preset node device among the multiple node devices.

[0008] In an optional implementation manner, the node device caches a mapping relationship between the file index node identifier and the directory index node identifier, and the method further includes: Based on a file query operation, obtaining the directory index node identifier according to the mapping relationship, the file query operation is used to query target file attribute information according to the file index node identifier; Determine a target node from the plurality of node devices according to the directory index node identifier; Using the file index node identifier as the Key value, searching the target node for a Value value corresponding to the Key value; If the Value value is an invalid value, the target file attribute information is obtained from the preset node device; otherwise, the target file attribute information is obtained from the target node according to the file index node identifier.

[0009] In an optional embodiment, the method further comprises: Based on a directory query operation for querying all directory items of a specified directory, determining a node to be queried for each directory item from the plurality of node devices; The metadata of the corresponding directory item is queried from the to-be-queried node of each directory item, and the directory query operation is responded to according to the metadata of all the queried directory items.

[0010] In an optional implementation manner, the step of determining the node to be queried for each directory entry from the multiple node devices includes: Obtaining an index node identifier of the specified directory, and determining a first node to be queried from the multiple node devices according to the index node identifier of the specified directory; Query the type and directory entry index node identifier of each directory entry in the specified directory from the first node to be queried; For any directory entry, if the type of the directory entry is a directory type, a second node to be queried is determined from the multiple node devices according to the directory entry index node identifier, and the first node to be queried and the second node to be queried are used as two nodes to be queried for the directory entry, wherein the first node to be queried stores first metadata of the directory entry, the second node to be queried stores second metadata of the directory entry, and the modification frequency of the first metadata is less than that of the second metadata; If the type of the directory entry is a file type, and the directory entry index node identifier is not an invalid value, taking the first node to be queried as the node to be queried of the directory entry; If the type of the directory entry is a file type and the index node identifier of the directory entry is an invalid value, the preset node device is used as the node to be queried for the directory entry.

[0011] In an optional embodiment, the method further comprises: When the target file does not have a hard link, obtaining the directory index node identifier; Update the directory index node identifier to the metadata of the target file; The metadata of the target file is migrated from the preset node device to the node device determined by the directory index node identifier.

[0012] In an optional implementation manner, the step of obtaining the file index node identifier of the target file for which a hard link needs to be created includes: Based on the creation operation of creating the target file in the target directory, obtaining a file index node identifier of the target file; Determine a node to be updated from the plurality of node devices according to the file index node identifier; A first transaction is created at the node to be updated, and metadata of the target file and metadata of the target directory are updated in the first transaction.

[0013] In an optional implementation, the step of updating the metadata of the target file according to the file index node identifier, and updating the metadata of the associated directory related to the hard link of the target file includes: A second transaction is created at the node to be updated, and in the second transaction, the index node identifier of the associated directory is updated into the metadata of the target file and the metadata of the associated directory is updated.

[0014] In an optional embodiment, the method further comprises: Based on the file query operation of querying the attribute information of the target file, obtaining the file index node identifier; Determine a target node from the plurality of node devices according to the file index node identifier; The target file attribute information is obtained from the target node.

[0015] In a second aspect, the present invention provides a metadata updating device, which is applied to each node device of a plurality of node devices in a distributed file system, and the device comprises: An acquisition module is used to obtain a file index node identifier of a target file for which a hard link needs to be created; An update module, used to update the metadata of the target file according to the file index node identifier, and update the metadata of the associated directory related to the hard link of the target file; The index node identifiers of the associated directory and the target directory do not exist in the metadata of the updated target file, or the metadata of the target file, the metadata of the associated directory related to the target file, and the metadata of the target directory related to the target file are stored in the same node device among the multiple node devices, and the target directory is the directory specified when the target file is created.

[0016] In a third aspect, the present invention provides a node device, including a processor and a memory, wherein the memory is used to store a program, and the processor is used to implement the metadata updating method described in any one of the aforementioned implementation modes when executing the program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the metadata updating method described in any one of the aforementioned embodiments.

[0018] Compared with the prior art, the present invention has the following beneficial effects: When creating a hard link to a target file, the present invention uses a file index node identifier to update the metadata of the target file and the metadata of an associated directory related to the hard link to the target file, either by not having the index node identifiers of the associated directory and the target directory in the metadata of the updated target file, so that the metadata of the target file and the directory to which it belongs are unbound, so that updates of the two are independent of each other and do not need to be restricted to the same transaction, thereby avoiding modification of metadata through cross-node transactions; or by storing the metadata of the target file, the metadata related to the target file in the associated directory, and the metadata related to the target file in the target directory in the same node device, so that updates of the three can be completed in the same node device, thereby avoiding modification of metadata through cross-node transactions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is an example diagram of an application scenario provided by this embodiment.

[0021] Figure 2 This is a block diagram of an example node device provided in this embodiment.

[0022] Figure 3 This is an example diagram of a file hard link provided in this embodiment.

[0023] Figure 4 This is an example flowchart of the metadata updating method provided in this embodiment.

[0024] Figure 5 The embodiment provided Figure 3 Example of metadata after creating a hard link in Figure 1 .

[0025] Figure 6 The embodiment provided Figure 3 Example of metadata after creating a hard link in Figure 2 .

[0026] Figure 7 This is a block diagram of an example of a metadata updating device provided in this embodiment.

[0027] Icons: 10 - node device; 11 - processor; 12 - memory; 13 - bus; 20 - client; 100 - metadata update device; 110 - acquisition module; 120 - update module; 130 - query module; 140 - creation module. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0031] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0032] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0033] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0034] Please refer to Figure 1 , Figure 1 This is an example diagram of an application scenario provided in this embodiment. Figure 1In the distributed file system, a plurality of node devices 10 are connected to the distributed file system and the client 20 is in communication. The client 20 creates a file in the distributed file system and writes data to the created file. The data written by the client 20 in the file is stored in a distributed form in each node device 10. In addition to the data written by the client 20, the distributed file system also needs to manage the files or directories created by the client 20. The data for managing files or directories is called metadata, which can also be stored in a distributed form in each node device 10. In a specific implementation, the storage of data and metadata can be independent of each other and stored in different multiple node devices. Data and metadata can also be distributed on each node device at the same time.

[0035] A stand-alone database, such as RocksDB, can be deployed on a single node device 10, or a distributed cluster database, such as TiKV, can be deployed on multiple node devices 10. Metadata is distributedly stored in the database deployed on the node device 10 in the form of Key-Value.

[0036] The node device 10 may be a physical computer device or a virtual device having the same function as the physical computer device. The node device 10 may be a server, a storage array, and the like.

[0037] based on Figure 1 This embodiment also provides a block diagram of a node device 10. The node device 10 implements the metadata update method of the aforementioned embodiment. Please refer to Figure 2 , Figure 2 This is a block diagram of a node device 10 provided in this embodiment. The node device 10 includes a processor 11, a memory 12 and a bus 13. The processor 11 and the memory 12 are connected via the bus 13.

[0038] The processor 11 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the metadata update method of the above embodiment may be completed by an integrated logic circuit of hardware in the processor 11 or by instructions in the form of software. The above processor 11 may be a general-purpose processor, including a CPU (Central Processing Unit), a NP (Network Processor), etc.; it may also be a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Logic Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0039] The memory 12 is used to store a program for implementing the metadata updating method. The program may be a software function module stored in the memory 12 in the form of software or firmware or solidified in the OS (Operating System) of the node device 10 .

[0040] After receiving the execution instruction, the processor 11 executes the program to implement the metadata updating method of the above embodiment.

[0041] Before introducing the metadata updating method provided by this embodiment, several concepts involved in this embodiment are first explained.

[0042] Distributed file system: A system that allows files to be stored and managed across multiple devices (such as servers and storage devices), allowing multiple users or clients to access these files simultaneously, and providing high availability, reliability, and scalability. For example, Windows' NTFS (New Technology File System), Linux's ext4, xfs, etc.

[0043] Directory: Used to organize and store other files and subdirectories.

[0044] Directory Entry: An entry in a directory that contains information such as the name of a file or subdirectory.

[0045] File: A collection of data stored on a storage device, which can contain different types of data such as text, images, audio, video, etc.

[0046] Hard Link: A file link method that allows multiple directory entries to point to the same file in the same distributed file system. Hard links do not take up additional storage space. Figure 3 , Figure 3 is an example diagram of a file hard link provided in this embodiment. Figure 3 In the figure, squares represent directories, such as D1, D2, and D3, and round boxes represent files, such as F1, F2, F3, and F4. Before creating a hard link for F3, D3 is an empty directory and F3 only belongs to directory D2, that is, F3 has only one parent directory D2. After creating a hard link for F3, F3 belongs to both directories D2 and D3, that is, F3 has two parent directories D2 and D3.

[0047] Inode: is a data structure used to store metadata of a file or directory (such as file size, permissions, creation time, modification time, etc.), as well as pointers to file data blocks. A directory or a file is an inode.

[0048] Inode_id: A number that uniquely identifies an inode. Each file or directory has a unique inode_id, which is used to find the corresponding inode in the distributed file system. The inode_id is generally represented by a 64-bit unsigned number.

[0049] Key-Value: A basic model for data storage and retrieval, in which each data item consists of a unique key (Key) and a corresponding value (Value). Key: A key is a unique identifier used to identify and retrieve a specific data item. The key is usually a string or other immutable data type to ensure its uniqueness. Value: A value is data of any type that represents the actual stored content. Values ​​can be simple data types (such as integers, strings) or complex data structures (such as lists, dictionaries). In this embodiment, metadata is organized according to Key-Value.

[0050] Since some operations will trigger the update of multiple metadata, and different metadata may be distributed on different node devices, in order to ensure that the metadata update meets the atomicity, consistency, isolation and persistence, a transaction mechanism is usually used. When the metadata to be updated is distributed on multiple node devices, it is necessary to create a cross-node transaction mechanism.

[0051] Taking the file creation operation as an example, at least three operations are involved: (1) modifying the timestamp of the file's parent directory; (2) adding a directory entry to the parent directory; and (3) adding metadata attribute information to the newly created file. These three operations need to be completed in the same database transaction. If it is a distributed transaction of the entire database cluster, it may involve coordination between multiple database nodes, resulting in high performance overhead, which affects the overall performance of the distributed file system.

[0052] In order to reduce the impact of cross-node transaction mechanisms on performance, the distributed file system InfiniFS divides directory attributes into two parts: (1) the immutable part, which includes directory access permissions, owner, etc.; and (2) the mutable part, which includes timestamps and directory entries. The attributes of the mutable part and its sub-items are all hashed using the inode_id of the directory to which they belong, ensuring that they belong to the same node device, making the entire transaction a local transaction rather than a distributed transaction across nodes. This has a significant theoretical improvement in file creation performance.

[0053] The distributed file system SingularFS handles this by storing file attributes in the KV pair of the directory entry. The key addressing method is the parent directory inode_id + file name, and the value stores the file inode_id and file attribute information. By binding the file to its parent directory, it is ensured that they belong to the same node device, making the entire transaction a local transaction instead of a distributed transaction across nodes.

[0054] Although both of the above methods reduce the impact of the transaction mechanism of cross-node devices on performance, neither of them can support hard links. After an in-depth analysis of their implementation principles, the inventors found that the reason why these two methods do not support hard links is that: first, the addressing of the file Key has a certain parent directory inode_id, and the file can only support a unique parent directory; second, the storage rule of the file KV pair is bound to the parent directory inode_id, which also limits the file to only support a unique parent directory. Hard links mean that the same file belongs to two different parent directories at the same time, which limits the support for hard links.

[0055] In view of this, the present embodiment provides a metadata update method, apparatus, node device and computer-readable storage medium, which can not only support hard links, but also avoid modifying metadata through cross-node transactions when creating hard links. Ultimately, it achieves both support for hard links and reduction of the impact of cross-node transactions on the overall performance of the distributed file system. It will be described in detail below.

[0056] Please refer to Figure 4 , Figure 4This is a flowchart of a metadata updating method provided in this embodiment. The method can be applied to Figure 1 and Figure 2 The node device in the method comprises the following steps: Step S101, obtaining a file index node identifier of a target file for which a hard link needs to be created.

[0057] In this embodiment, the target file is a file for which a hard link needs to be created, the file index node identifier is the inode_id of the target file, and the distributed file system allocates an inode_id to the target file when creating the target file.

[0058] Step S102, updating metadata of the target file according to the file index node identifier, and updating metadata of the associated directory related to the hard link of the target file; The metadata of the updated target file does not contain the index node identifiers of the associated directory and the target directory, or the metadata of the target file, the metadata of the associated directory related to the target file, and the metadata of the target directory related to the target file are stored in the same node device among multiple node devices, and the target directory is the directory specified when the target file is created.

[0059] In this embodiment, the creation of a hard link to a target file triggers changes in the metadata of the target file and the directory to which the target file belongs. The associated directory is a directory related to the hard link to the target file, and the target directory is a directory specified when the target file is created. For example, when creating a target file F, it is specified to create the target file under directory D1, then D1 is the target directory of F. When creating a hard link for the target file, it is specified to create a hard link from D2 / F to D1 / F, then D2 is the associated directory.

[0060] In this embodiment, the metadata related to the target file in the associated directory, that is, there is a directory entry, which represents the association relationship between the associated directory and the target file. For example, the Key value of the directory entry is: the inode_id of the associated directory and the directory entry name, and the Value value of the directory entry is: the type of the target file and the inode_id of the target file. The metadata related to the target file in the target directory is similar.

[0061] In this embodiment, the metadata of the target file and the target directory can be unbound, so that the updates of the two will be independent of each other and do not need to be restricted by the same transaction. The metadata of the target file, the metadata related to the target file in the associated directory, and the metadata related to the target file in the target directory can also be stored in the same node device to avoid modifying the metadata through cross-node transactions. Both methods can avoid modifying metadata through cross-node transactions, achieving the technical effect of supporting hard links and reducing the impact of cross-node transactions on the overall performance of the distributed file system.

[0062] For these two methods, there are at least two corresponding implementation methods for step S102. Each implementation method and at least one processing process for metadata information based on each implementation method will be introduced in turn below.

[0063] First, we introduce the implementation method of unbinding the metadata of the target file and the target directory (referred to as method 1 in this article). In this implementation method, the attributes of the directory are divided into two parts: one is the immutable part, that is, the part with low modification frequency, such as the access rights and owner of the directory, and the other is the mutable part, that is, the part with high modification frequency, such as timestamps and directory entries. Correspondingly, the metadata of the directory also includes two categories, one is the metadata of the directory related to the immutable part, and the other is the metadata of the directory related to the mutable part. In addition, there are metadata related to the attributes of the directory items and metadata related to the attributes of the files. The KV of these metadata is organized as follows: (1) The part of the directory attribute that is not mutable (the name is determined based on inode-Stable) Key: self_inode_id + stable tag Value: parent_inode_id, owner permissions and other related information Distribution basis: parent_inode_id The stable tag is used to identify the non-volatile part of the directory attribute. self_inode_id is the inode_id of the directory itself, parent_inode_id is the inode_id of the parent directory of the directory, and the distribution basis means the basis for determining the node device. For example, the inode_id can be used for hash calculation, and the node device can be determined by the result of the hash calculation. For the non-volatile part of the directory attribute, the metadata of the directory is stored in the node device determined by the parent_inode_id.

[0064] (2) Volatile part of directory attributes (name determined based on inode-volatile) Key: self_inode_id + volatile flag Value: timestamp and other information Distribution basis: self_inode_id The volatile tag is used to identify the volatile part of the directory attribute. self_inode_id is the inode_id of the directory itself, and the distribution basis means that the metadata of the directory is stored in the node device determined by self_inode_id. Each timestamp includes, but is not limited to, the creation time, the last modification time, etc.

[0065] (3) Directory entry (name determined based on DEntry) Key: self_inode_id + name (directory entry name) Value: type (child type) + child_inode_id (child inode_id) Distribution basis: self_inode_id Among them, self_inode_id is the inode_id of each directory entry itself, name is the name of the directory entry. If the directory entry is a file, self_inode_id is the inode_id of the file, name is the file name of the file, if the directory entry is a directory, self_inode_id is the inode_id of the directory, name is the directory name of the directory. type is the child item type, including two types: directory type and file type. child_inode_id is the inode_id of the child item of the directory item. For example, for a directory item of directory type, its child item is a subdirectory or file belonging to the directory. Distribution basis means that the metadata of the directory item is stored in the node device determined by self_inode_id.

[0066] (4) File attributes (determine the name based on inode) Key: self_inode_id (corresponding to child_inode_id in the directory item Value) Value: parent_inode_id, all other attributes of the file (such as permission configuration, time, length, etc.) Distribution basis: parent_inode_id Among them, self_inode_id is the inode_id of the file itself, which corresponds to the child_inode_id in the directory entry of the directory to which the file belongs, and parent_inode_id is the inode_id of the directory to which the file belongs. The distribution basis means that the metadata of the file is stored in the node device determined by the parent_inode_id.

[0067] Based on the above organization method, when a file is created, the file belongs to only one directory. The node device that stores the file's metadata is determined by the inode_id of the directory to which it belongs (that is, the parent directory). The mutable part of the parent directory attribute of the file is also determined according to the inode_id of the parent directory itself. The directory entry of the parent directory is also determined according to the inode_id of the parent directory itself. All three are determined according to the inode_id of the parent directory, that is, the metadata involved in the operation of creating a file are stored in the same node device, avoiding the creation of cross-node transactions.

[0068] When no hard links are created for a file, the file belongs to only one directory. By managing metadata in the above manner, cross-node transactions can be avoided as much as possible, reducing the impact on the overall performance of the distributed file system. When a hard link is created for a file, the file belongs to two directories at the same time. In order to continue to support the normal management of each metadata, this embodiment provides the following method for updating metadata: (1) querying metadata of a target file according to the file index node identifier, where the metadata of the target file includes the directory index node identifier of the target directory; (2) Update the directory index node identifier to an invalid value; (3) Migrate the metadata of the target file to a preset node device among the multiple node devices.

[0069] In this embodiment, before creating a hard link, the file index node identifier is the inode_id of the file, the directory index node identifier is the inode_id of the target directory, the Key value of the metadata of the target file is the inode_id of the target file, and the Value value is the inode_id of the target directory. Since creating a hard link will cause the target file to have two parent directories at the same time, at this time, in order to continue to support the normal management of metadata, the node device storing the metadata of the target file and the target directory are unbound. One implementation method is to update the directory index node identifier to an invalid value. The invalid value can be configured according to actual needs, for example, 0 is used to represent an invalid value.

[0070] In this embodiment, since the node device storing the metadata of the target file and the target directory are unbound, the node device storing the metadata of the target file can no longer be determined according to the inode_id of the target directory. In order to be able to manage the metadata of the target file normally, the metadata of the target file is migrated to a preset node device. The preset node device can be selected from multiple node devices in the distributed file system according to a preset principle. The preset principle can be a random selection principle, that is, random selection, or a minimum load pressure principle, that is, selecting a node device with the minimum load pressure.

[0071] In order to more intuitively demonstrate the metadata of the hard link created by using the first method, this embodiment combines Figure 3 The structure of each file and directory after creating a hard link in , showing the metadata of each file and directory, please refer to Figure 5 , Figure 5 The embodiment provided Figure 3 An example diagram of metadata after a hard link is created in . Figure 5 In, for Figure 3 The Key column is the Key value of each metadata, the Value column is the Value corresponding to the Key, and the Partition column is the distribution basis of the corresponding metadata. For example, for the D1 directory, Figure 5 Stable_D1_inode_id in the table represents the key value of the metadata of the non-volatile part of the directory D1 attribute, and its corresponding value includes the non-volatile part of the directory D1 attribute: owner, permission, and distribution is based on D1_inode_id. With the inode_id of directory D1, Volatile_D1_inode_id represents the key value of the metadata of the volatile part of the directory D1 attribute, and its corresponding value includes the volatile part of the directory D1 attribute: various timestamps, and distribution is based on D1_inode_id. For the directory entry of D1, Figure 5 DEntry_D1_inode_id+"D2" indicates the directory entry of the D2 subdirectory under the D1 directory. The type of D2 is directory, and the corresponding Value is D2_inode_id. The distribution basis is D1_inode_id. For the file F3 with a hard link, the Key value of its metadata is FILE_F3_inode_id, that is, the _inode_id of file F3. The corresponding Value is all the file attributes of F3. The distribution basis is fixed to database shard No. 0, that is, the database in node device No. 0. The rest of the metadata is similar and will not be repeated here.

[0072] It should be noted that Figure 5 This is just an example. In fact, those skilled in the art can expand and change the specific representation of Key value and Value value based on this according to actual needs to meet different requirements of actual scenarios. For example, Value can include more or more information.

[0073] In this embodiment, for the query scenario of querying the properties of the target file through the inode_id of the target file, since the target file is no longer bound to its parent directory, and it is not necessarily determined according to the inode_id of its parent directory that the node device storing its metadata is determined, one implementation method is: query the inode_id of the parent directory of the target file to all node devices. This method has poor query performance. In order to improve the query performance of this method, this embodiment provides an improved method, that is, in the search and traversal process from the root directory to the target file, the parent_inode_id of each inode_id is cached, for example, cached in the service process of the client of the distributed file system. When a request to modify the parent directory of the target file is received, such as unlink, rename, etc., the mapping relationship cache of inode_id -> parent_inode_id is synchronously modified. Thereby, the performance loss caused by querying the parent directory of the target file to all node devices is avoided. When the mapping relationship between the file index node identifier and the directory index node identifier is cached, the method of querying the target file is: First, based on the file query operation, the directory index node identifier is obtained according to the mapping relationship, and the file query operation is used to query the target file attribute information according to the file index node identifier; Secondly, according to the directory index node identifier, the target node is determined from multiple node devices; Third, use the file index node identifier as the Key value and search for the Value value corresponding to the Key value in the target node; Fourth, if the Value value is an invalid value, the target file attribute information is obtained from the preset node device, otherwise the target file attribute information is obtained from the target node according to the file index node identifier.

[0074] In this embodiment, when all hard links of the target file are deleted, when querying the attribute information of the target file in the above manner, the file index node identifier is still used as the Key value, and the Value value corresponding to the Key value is searched in the target node. If the Value value is found to be invalid, the target file attribute information is obtained from the preset node device. In order to avoid unnecessary query and judgment, in this embodiment, when there is no hard link to the target file, the node device of the target file is migrated from the preset node device to the node device determined by the directory index node identifier. The processing method is as follows: First, when there is no hard link to the target file, obtain the directory index node identifier; Secondly, the directory index node identifier is updated to the metadata of the target file; Finally, the metadata of the target file is migrated from the preset node device to the node device determined by the directory index node identifier.

[0075] After all hard links of the target file are deleted, the target file returns to the situation with only one parent directory. Therefore, its metadata is also returned to the situation with only one parent directory, thereby avoiding the performance loss caused by the need to judge invalid values ​​each time and then obtain the attribute information of the target file from the preset node device.

[0076] In this embodiment, in addition to the query operation for querying files, there are also operations for querying directories and directory items under the directory, such as ReadDir, ReadDirPlus, etc. For ReadDir, all directory items under the specified directory are obtained, and for ReadDirPlus, all directory items under the specified directory and the attribute information of each directory item are obtained. When a query operation is performed on all directory items in a specified directory, all directory items in the specified directory and the attribute information corresponding to each directory item are obtained. The information of all directory items is in the node device (referred to as node device 1) determined by the inode_id of the specified directory. If the directory item type is a file, it is also in node device 1; if the directory item type is a directory, the non-volatile part of its attributes is in node device 1, and the volatile part needs to calculate the hash distribution based on the inode_id of the directory item itself, which may be in other node devices. At this time, a cross-node query request will be involved.

[0077] For a file with a hard link created for a directory entry, the value will first be found to be an invalid value when querying node device 1, and then the attribute information of the file will be obtained from the preset node device. Due to common application scenarios, there are only a small number of directories and most files do not have hard links. Therefore, this query operation has good performance. This embodiment provides a specific implementation method for querying all directory entries in a specified directory: First, based on a directory query operation for querying all directory items of a specified directory, a node to be queried for each directory item is determined from a plurality of node devices; In this embodiment, the nodes to be queried are different in different situations. Whether the directory item is a file or a directory, the corresponding nodes to be queried are also different. For a file, there may be only one node to be queried, and for a directory, there may be two nodes to be queried. This embodiment provides an implementation method for determining the node to be queried: (1) obtaining an index node identifier of a specified directory, and determining a first node to be queried from a plurality of node devices according to the index node identifier of the specified directory; (2) querying the first node to be queried for the type and directory entry index node identifier of each directory entry in the specified directory; (3) For any directory entry, if the type of the directory entry is a directory type, a second node to be queried is determined from a plurality of node devices according to the directory entry index node identifier, and the first node to be queried and the second node to be queried are used as two nodes to be queried of the directory entry, wherein the first node to be queried stores first metadata of the directory entry, the second node to be queried stores second metadata of the directory entry, and the modification frequency of the first metadata is less than that of the second metadata; In this embodiment, the first metadata corresponds to the non-mutable part of the directory attribute, and the second metadata corresponds to the mutable part of the directory attribute. The metadata of the directory includes the first metadata and the second metadata.

[0078] (4) If the type of the directory entry is a file type and the index node identifier of the directory entry is not an invalid value, the first node to be queried is used as the node to be queried of the directory entry; (5) If the type of the directory entry is a file type and the index node identifier of the directory entry is an invalid value, the preset node device is used as the node to be queried for the directory entry.

[0079] Secondly, the metadata of the corresponding directory item is queried from the to-be-queried node of each directory item, and the directory query operation is responded to according to the metadata of all the queried directory items.

[0080] The above is to support hard link of target file by unbinding target file and target directory. This embodiment also provides a method to store metadata of target file, metadata related to target file in associated directory and metadata related to target file in target directory in the same node device of multiple node devices to support hard link (referred to as method 2 in this article). This method is still based on the method of dividing the attributes of directory into non-volatile part and volatile part in method 1, which will not be repeated here. In addition, metadata related to target file in associated directory can belong to the volatile part of associated directory, metadata related to target file in target directory can belong to the volatile part of target directory, this method still also includes metadata related to attributes of directory items and metadata related to attributes of files. Before introducing this method, the KV organization method of these metadata is still introduced first.

[0081] (1) The part of the directory attribute that is not mutable (the name is determined based on inode-Stable) Key: self_inode_id + stable tag Value: parent_inode_id, owner permissions and other related information Distribution basis: self_inode_id (2) Volatile part of directory attributes (name determined based on inode-volatile) Key: self_inode_id + volatile flag Value: Timestamp information Distribution basis: child_inode_id (3) Directory entry DEntry (determine the name based on DEntry) Key: self_inode_id + name (directory entry name) Value: type (child type) + child_inode_id (child inode_id) Distribution basis: child_inode_id (4) File attribute inode (name determined based on inode) Key: self_inode_id (corresponding to child_inode_id in the directory item Value) Value: parent_inode_id, all other attributes of the file (such as permission configuration, time, length, etc.) Distribution basis: self_inode_id The Key, Value and distribution basis in (1) to (4) above have been introduced in the above method 1 and will not be repeated here.

[0082] Since in the second method, the node device storing the metadata of the target file is determined by the inode_id of the file, before introducing the process of updating the metadata when creating a hard link for the target file, this embodiment first introduces the creation process of the target file: First, based on the creation operation of creating the target file in the target directory, the file index node identifier of the target file is obtained; In this embodiment, the file index node identifier is allocated by the distributed file system when the target file is created, and the file index node identifier may be the inode_id of the target file.

[0083] Secondly, determine the node to be updated from multiple node devices according to the file index node identifier; In this embodiment, a hash calculation is performed using the file index node identifier, and the node to be updated can be determined according to the hash calculation result. The node to be updated is a node that stores metadata of the target file.

[0084] Finally, a first transaction is created at the node to be updated, and metadata of the target file and metadata of the target directory are updated in the first transaction.

[0085] In this embodiment, since the node storing the metadata related to the target directory and the target file is also determined according to the file index node identifier, both are stored in the node to be updated.

[0086] Based on the above method of creating a target file, this embodiment provides a method of updating metadata when creating a hard link for a target file: A second transaction is created at the node to be updated. In the second transaction, the index node identifier of the associated directory is updated into the metadata of the target file and the metadata of the associated directory is updated.

[0087] In this embodiment, metadata related to the target file in the associated directory is also stored in the node to be updated. At this time, the target file has two parent directories: the target directory and the associated directory. Therefore, the associated directory needs to be updated to the metadata of the target file.

[0088] In order to more intuitively demonstrate the metadata of the hard link created by using the second method, this embodiment combines Figure 3 The structure of each file and directory after creating a hard link in , showing the metadata of each file and directory, please refer to Figure 6 , Figure 6 The embodiment provided Figure 3 An example of metadata after creating a hard link in . Figure 6 The meaning of Key, Value and Partition can be found in Figure 5 , I will not go into details here.

[0089] It should be noted that Figure 6 This is just an example. In fact, those skilled in the art can expand and change the specific representation of Key value and Value value based on this according to actual needs to meet different requirements of actual scenarios. For example, Value can include more or more information.

[0090] Based on the second method, this embodiment provides a method for implementing a query operation on the target file attribute information: First, based on the file query operation of querying the target file attribute information, the file index node identifier is obtained; Secondly, according to the file index node identifier, the target node is determined from multiple node devices; Finally, the target file attribute information is obtained from the target node.

[0091] In addition to querying file attribute information, similar to method one, method two also has operations for querying directories. If you only need to obtain all directory items of a specified directory, such as the ReadDir command, since the metadata of the volatile part of the directory and the metadata of the non-volatile part are in different node devices, the request needs to be sent to all node devices. If, in addition to obtaining all directory items of a specified directory, you also need to obtain the attribute information of each directory item, such as the ReadDirPlus command, since the metadata of the volatile part of the directory item and its attribute information are in the same node device, compared to the ReadDir command, no additional cross-node device requests will be added.

[0092] In addition, for commands that need to be sent to all node devices, such as the lookup command and the ReadDir command, in order to alleviate the additional performance overhead of sending operations to all node devices, the directory entry cache mechanism can be used to alleviate this. The directory entry cache mechanism stores the most recently accessed directory entries in memory to avoid hard disk access every time the directory is accessed. Through the directory entry cache, the system can locate the hard disk location of the required file more quickly, thereby speeding up file access.

[0093] In order to execute the corresponding steps in the above embodiment and each possible implementation method, a method for implementing the metadata updating device 100 is given below. Figure 7 , Figure 7 This is a block diagram of the metadata updating device provided in this embodiment. It should be noted that the basic principle and technical effects of the metadata updating device 100 provided by the present invention are the same as those of the corresponding above-mentioned embodiments, which are not mentioned in this embodiment for the sake of brief description.

[0094] The metadata updating apparatus 100 includes an acquiring module 110 and an updating module 120 .

[0095] An acquisition module 110 is used to acquire a file index node identifier of a target file for which a hard link needs to be created; An updating module 120, configured to update metadata of a target file according to a file index node identifier, and to update metadata of an associated directory related to a hard link of the target file; The metadata of the updated target file does not contain the index node identifiers of the associated directory and the target directory, or the metadata of the target file, the metadata related to the target file of the associated directory, and the metadata related to the target file of the target directory are stored in the same node device among the multiple node devices, and the target directory is the directory specified when the target file is created.

[0096] In an optional implementation manner, the updating module 120 is specifically configured to: The metadata of the target file is queried according to the file index node identifier, wherein the metadata of the target file includes the directory index node identifier of the target directory; the directory index node identifier is updated to an invalid value; and the metadata of the target file is migrated to a preset node device among the multiple node devices.

[0097] In an optional implementation, the metadata updating apparatus 100 further includes a query module 130, the node device caches a mapping relationship between the file index node identifier and the directory index node identifier, and the query module 130 is used to: Based on the file query operation, the directory index node identifier is obtained according to the mapping relationship. The file query operation is used to query the target file attribute information according to the file index node identifier; according to the directory index node identifier, the target node is determined from multiple node devices; the file index node identifier is used as the Key value, and the Value value corresponding to the Key value is searched in the target node; if the Value value is an invalid value, the target file attribute information is obtained from the preset node device, otherwise the target file attribute information is obtained from the target node according to the file index node identifier.

[0098] In an optional implementation, the query module 130 is further configured to: Based on a directory query operation for querying all directory items of a specified directory, a node to be queried for each directory item is determined from multiple node devices; metadata of the corresponding directory item is queried from the node to be queried for each directory item, and the directory query operation is responded to according to the metadata of all the queried directory items.

[0099] In an optional implementation manner, when the query module 130 is specifically used to determine the node to be queried for each directory entry from multiple node devices, it is also specifically used to: The index node identifier of the specified directory is obtained, and a first node to be queried is determined from multiple node devices according to the index node identifier of the specified directory; the type of each directory item in the specified directory and the index node identifier of the directory item are queried from the first node to be queried; for any directory item, if the type of the directory item is a directory type, a second node to be queried is determined from multiple node devices according to the index node identifier of the directory item, and the first node to be queried and the second node to be queried are used as two nodes to be queried of the directory item, wherein the first node to be queried stores first metadata of the directory item, the second node to be queried stores second metadata of the directory item, and the modification frequency of the first metadata is less than that of the second metadata; if the type of the directory item is a file type and the index node identifier of the directory item is not an invalid value, the first node to be queried is used as the node to be queried of the directory item; if the type of the directory item is a file type and the index node identifier of the directory item is an invalid value, a preset node device is used as the node to be queried of the directory item.

[0100] In an optional implementation, the updating module 120 is further configured to: When there is no hard link to the target file, obtain the directory index node identifier; update the directory index node identifier to the metadata of the target file; and migrate the metadata of the target file from the preset node device to the node device determined by the directory index node identifier.

[0101] In an optional implementation, the metadata updating apparatus 100 further includes a creation module 140, and the creation module 140 is used to: Based on a creation operation of creating a target file in a target directory, a file index node identifier of the target file is obtained; a node to be updated is determined from multiple node devices according to the file index node identifier; a first transaction is created at the node to be updated, and metadata of the target file and metadata of the target directory are updated in the first transaction.

[0102] In an optional implementation manner, the creation module 140 is specifically used to: A second transaction is created at the node to be updated. In the second transaction, the index node identifier of the associated directory is updated into the metadata of the target file and the metadata of the associated directory is updated.

[0103] In an optional implementation, the query module 130 is further configured to: Based on the file query operation of querying the target file attribute information, the file index node identifier is obtained; according to the file index node identifier, the target node is determined from multiple node devices; and the target file attribute information is obtained from the target node.

[0104] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the metadata updating method in the aforementioned implementation is implemented.

[0105] In summary, an embodiment of the present invention provides a metadata updating method, apparatus, node device and computer-readable storage medium, the method comprising: obtaining a file index node identifier of a target file for which a hard link needs to be created; updating the metadata of the target file according to the file index node identifier, and updating the metadata of an associated directory related to the hard link of the target file; the index node identifiers of the associated directory and the target directory do not exist in the metadata of the updated target file, or the metadata of the target file, the metadata related to the target file in the associated directory and the metadata related to the target file in the target directory are stored in the same node device among multiple node devices, and the target directory is the directory specified when the target file is created. Compared with the prior art, this embodiment has at least the following advantages: (1) by eliminating the index node identifiers of the associated directory and the target directory in the metadata of the updated target file, the metadata of the target file and the directory to which it belongs are unbound, so that the updates of the two are independent of each other and do not need to be restricted by the same transaction, thereby avoiding the modification of metadata through cross-node transactions; (2) by storing the metadata of the target file, the metadata related to the target file in the associated directory, and the metadata related to the target file in the target directory in the same node device, the updates of the three can be completed in the same node device, thereby avoiding the modification of metadata through cross-node transactions; (3) the mapping relationship between the file index node identifier and the directory index node identifier is pre-cached, thereby improving the performance of file query operations; (4) when there is no hard link to the file, the metadata of the target file is migrated from the preset node device to the node device determined by the directory index node identifier, thereby avoiding unnecessary judgment and search, and optimizing the search efficiency.

[0106] The above are only various embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A metadata updating method, characterized in that: Applied to each node device of a plurality of node devices in a distributed file system, the method comprises: Get the file index node identifier of the target file for which a hard link needs to be created; updating the metadata of the target file according to the file index node identifier, and updating the metadata of the associated directory related to the hard link of the target file; The index node identifiers of the associated directory and the target directory do not exist in the metadata of the updated target file, or the metadata of the target file, the metadata of the associated directory related to the target file, and the metadata of the target directory related to the target file are stored in the same node device among the multiple node devices, and the target directory is the directory specified when the target file is created.

2. The metadata updating method according to claim 1, characterized in that: The step of updating the metadata of the target file according to the file index node identifier comprises: Querying metadata of the target file according to the file index node identifier, where the metadata of the target file includes a directory index node identifier of the target directory; Updating the directory index node identifier to an invalid value; The metadata of the target file is migrated to a preset node device among the multiple node devices.

3. The metadata updating method according to claim 2, characterized in that: The node device cache has a mapping relationship between the file index node identifier and the directory index node identifier, and the method further includes: Based on a file query operation, obtaining the directory index node identifier according to the mapping relationship, the file query operation is used to query target file attribute information according to the file index node identifier; Determine a target node from the plurality of node devices according to the directory index node identifier; Using the file index node identifier as the Key value, searching the target node for a Value value corresponding to the Key value; If the Value value is an invalid value, the target file attribute information is obtained from the preset node device; otherwise, the target file attribute information is obtained from the target node according to the file index node identifier.

4. The metadata updating method according to claim 2, characterized in that: The method further comprises: Based on a directory query operation for querying all directory items of a specified directory, determining a node to be queried for each directory item from the plurality of node devices; The metadata of the corresponding directory item is queried from the to-be-queried node of each directory item, and the directory query operation is responded to according to the metadata of all the queried directory items.

5. The metadata updating method according to claim 4, characterized in that: The step of determining the node to be queried for each directory entry from the plurality of node devices comprises: Obtaining an index node identifier of the specified directory, and determining a first node to be queried from the multiple node devices according to the index node identifier of the specified directory; Query the type and directory entry index node identifier of each directory entry in the specified directory from the first node to be queried; For any directory entry, if the type of the directory entry is a directory type, a second node to be queried is determined from the multiple node devices according to the directory entry index node identifier, and the first node to be queried and the second node to be queried are used as two nodes to be queried for the directory entry, wherein the first node to be queried stores first metadata of the directory entry, the second node to be queried stores second metadata of the directory entry, and the modification frequency of the first metadata is less than that of the second metadata; If the type of the directory entry is a file type, and the directory entry index node identifier is not an invalid value, taking the first node to be queried as the node to be queried of the directory entry; If the type of the directory entry is a file type and the index node identifier of the directory entry is an invalid value, the preset node device is used as the node to be queried for the directory entry.

6. The metadata updating method according to claim 2, characterized in that: The method further comprises: When the target file does not have a hard link, obtaining the directory index node identifier; Update the directory index node identifier to the metadata of the target file; The metadata of the target file is migrated from the preset node device to the node device determined by the directory index node identifier.

7. The metadata updating method according to claim 1, characterized in that: The step of obtaining the file index node identifier of the target file for which a hard link needs to be created includes: Based on the creation operation of creating the target file in the target directory, obtaining a file index node identifier of the target file; Determine a node to be updated from the plurality of node devices according to the file index node identifier; A first transaction is created at the node to be updated, and metadata of the target file and metadata of the target directory are updated in the first transaction.

8. The metadata updating method according to claim 7, characterized in that: The step of updating the metadata of the target file according to the file index node identifier, and updating the metadata of the associated directory related to the hard link of the target file includes: A second transaction is created at the node to be updated, and in the second transaction, the index node identifier of the associated directory is updated into the metadata of the target file and the metadata of the associated directory is updated.

9. The metadata updating method according to claim 7, characterized in that: The method further comprises: Based on the file query operation of querying the attribute information of the target file, obtaining the file index node identifier; Determine a target node from the plurality of node devices according to the file index node identifier; The target file attribute information is obtained from the target node.

10. A metadata updating device, characterized in that: Applied to each node device of a plurality of node devices in a distributed file system, the device comprises: An acquisition module is used to obtain a file index node identifier of a target file for which a hard link needs to be created; An update module, used to update the metadata of the target file according to the file index node identifier, and update the metadata of the associated directory related to the hard link of the target file; The index node identifiers of the associated directory and the target directory do not exist in the metadata of the updated target file, or the metadata of the target file, the metadata of the associated directory related to the target file, and the metadata of the target directory related to the target file are stored in the same node device among the multiple node devices, and the target directory is the directory specified when the target file is created.

11. A node device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a program, and the processor is used to implement the metadata updating method according to any one of claims 1 to 9 when executing the program.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the metadata updating method according to any one of claims 1 to 9 is implemented.

Images