Write-in method and device of distributed file system, electronic equipment and medium

Abstract

The embodiment of the invention provides a writing method and device of a distributed file system, electronic equipment and a medium. According to the method and the device, through improvement on metadata caching and persistence, when distributed write-in operations such as hard linking and renaming are executed, a link between a cache directory entry CDentry and a cache index node CInode in the cache is disconnected, the directory entry Dentry and the index node CInode are persisted respectively, so that only one part of metadata exists in the cache, and therefore, the content of the metadata in the cache is reduced, and the content of the metadata in the cache is reduced. Compared with the prior art, the method and the device have the advantages that occupation of memory resources is reduced, complicated cache synchronization is avoided, Dentry and Inode persistence are not forcibly bound, metadata migration generated during renaming or hard link operation is also avoided, and therefore the purposes of reducing request amplification and metadata migration and ensuring the performance of a file system are achieved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, in particular to a method, device, electronic device and medium for writing to a distributed file system. Background Art

[0002] With the advent of the big data era, massive amounts of data are generated, and distributed file systems (such as Ceph) are increasingly widely used. In a distributed file system, metadata is managed by multiple metadata servers (Metadataserver MDS).

[0003] The existing persistence and caching mechanisms of metadata make distributed write operations (such as renaming or hard linking) across MDSs very complicated, resulting in metadata migration and request amplification. For example, when performing distributed write operations such as renaming or hard linking, there will be multiple copies of metadata in the cache, and in order to keep the copies synchronized, a complex distributed lock mechanism CAP must be introduced, which will cause metadata migration and request amplification, thereby affecting system performance. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a distributed file system writing method, device, electronic device and medium to achieve the purpose of reducing request amplification and metadata migration, thereby ensuring the performance of the file system.

[0005] An embodiment of the present application provides a method for writing a distributed file system, the method being applied to a network device, the method comprising:

[0006] For the metadata of the file to be cached, if the file is created for the first time in this file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; if the file is currently in the state of being executed with a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected;

[0007] For the metadata of the file to be persisted, if CDentry and CInode in the metadata in the cache are linked to each other, the key key of the directory item Dentry is constructed based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name, and the value value of Dentry is constructed using the preset first-class value construction method Dn(I) to persist Dentry and Inode; wherein Dn(I) refers to the method of constructing value based on all the information of Inode; the bucketid of the metadata of any file is determined based on the principle of metadata scattering;

[0008] If the link between CDentry in the metadata in the cache and the index node CInode is disconnected, the key of Dentry is constructed based on the bucketid corresponding to the metadata and the index node number of the parent index node of Inode, the value of Dentry is constructed using the preset second-class value construction method Dn(L), the key of Inode is constructed based on the bucketid corresponding to the metadata and the index node number of Inode, and the value of Inode is constructed based on the Dn(I) method to persist the Dentry and Inode respectively; Dn(L) refers to the method of constructing the value of the Dentry using the key when the Inode corresponding to the Dentry is persisted.

[0009] The embodiment of the present application also provides a distributed file system writing device, which is applied to a network device, and includes:

[0010] The cache module is used for the metadata of the file to be cached. If the file is created for the first time in the file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; if the file is currently in the state of being executed with a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected;

[0011] The first persistence module is used for constructing the key key of the directory item Dentry based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name for the metadata of the file to be persisted, and constructing the value value of Dentry in a preset first-class value construction method Dn(I) to persist Dentry and Inode; wherein Dn(I) refers to a method of constructing value based on all information of Inode; the bucketid of the metadata of any file is determined according to the principle of metadata scattering;

[0012] The second persistence module is used for, if the link between CDentry in the metadata and the index node CInode in the cache is disconnected, constructing the key of Dentry based on the bucketid corresponding to the metadata and the index node number of the parent index node of Inode, constructing the value of Dentry value by the preset second type of value construction method Dn(L), constructing the key of Inode based on the bucketid corresponding to the metadata and the index node number of Inode, and constructing the value of Inode based on the Dn(I) method, so as to store Dentry and Inode persistently respectively; Dn(L) refers to the method of constructing the value of Dentry by the key when the Inode corresponding to Dentry is persisted;

[0013] Among them, when caching metadata, for files created for the first time in this file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; for files that have been subjected to specified write operations, the link between CDentry and CInode of the metadata of the file is disconnected.

[0014] An embodiment of the present application also provides an electronic device, comprising: a processor and a computer-readable storage medium for storing computer program instructions, wherein the computer program instructions, when executed by the computer-readable storage medium, enable the processor to execute the steps of the above method.

[0015] An embodiment of the present application also provides a machine-readable storage medium, which stores computer program instructions. When the computer program instructions are executed, the steps of the above method can be implemented.

[0016] It can be seen from the above technical solution that in this embodiment, when caching metadata, for a file created for the first time in this file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; for a file that has been subjected to a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected. This makes the metadata only have one copy in the cache, reduces memory resource usage, and avoids complex cache synchronization, thereby reducing request amplification and ensuring system performance.

[0017] Furthermore, when persisting metadata, if CDentry and CInode in the cache are linked to each other, the key key of the directory item Dentry is constructed based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name, and the value value of Dentry is constructed using the preset first-class value construction method Dn(I) to persist Dentry and Inode. If the link between CDentry and index node CInode in the metadata in the cache is disconnected, De is constructed based on the bucketid corresponding to the metadata and the index node number of the parent index node of Inode. The key of ntry is constructed by using the preset second-class value construction method Dn(L) to construct the value of Dentry. The key of Inode is constructed based on the bucketid corresponding to the metadata and the index node number of Inode. The value of Inode is constructed based on the Dn(I) method to store Dentry and Inode persistently respectively. This persistence method can ensure that the required metadata information can be obtained through simple queries. On the other hand, Dentry and Inode are persisted separately without forcibly binding Dentry and Inode. This further guarantees system performance when renaming or hard link operations across multiple MDSs occur. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a process for a method provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the process of hard link persistence provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the renaming persistence process provided in an embodiment of the present application;

[0021] Figure 4 A flow chart for determining whether a directory is empty provided in an embodiment of the present application;

[0022] Figure 5 A flow chart of distributed transaction management provided in an embodiment of the present application;

[0023] Figure 6 A renaming process interaction diagram provided for an embodiment of the present application;

[0024] Figure 7 A schematic diagram of the structure of the device provided in the embodiment of the present application;

[0025] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0027] To facilitate understanding of this solution, the technical issues are described in detail before describing this solution:

[0028] A distributed file system is a large, scalable cluster system built by multiple independent devices over a network. The directory file of the entire file system is a tree structure. To avoid system resource contention and eliminate system bottlenecks, the directory tree structure needs to be cut so that each MDS manages resources separately. That is, in a distributed file system, metadata is managed jointly by multiple MDSs.

[0029] In a distributed file system, common distributed write operations involving multiple MDSs include hard links (link / unlink) and renames. A hard link is a special file link that allows a file to have multiple file names pointing to it. Because of the existence of hard links, multiple dentry points to the same inode. How to build metadata in the cache and how to persist it to reduce space usage are key considerations for distributed file systems. The rename operation involves changes to the metadata information of two directories, two dentry, and two inodes. How to ensure metadata cache consistency is also a very complicated issue.

[0030] The inventors have analyzed and found that although the persistence mechanism and cache mechanism of metadata can achieve the above functions, there are also the following problems: in addition to the metadata cache generated during creation, link or rename may generate copies of the same inode on different MDSs in the cache, and the generation of new copies has the concepts of authority (auth) and replica (replica). In addition, modification requests must be processed by the auth node, which means that the request must be forwarded by the replica to the auth node for processing, which will inevitably lead to request amplification. In order to maintain the consistency of these replicas, a complex distributed lock mechanism CAP must be introduced, which further leads to request methods and causes metadata migration, which seriously affects system performance.

[0031] Based on this, in order to solve the above problems, the embodiments of the present application provide a distributed file system writing method, device, electronic device and medium to achieve the purpose of reducing request amplification and metadata migration, thereby ensuring the performance of the file system.

[0032] The method provided in the embodiment of the present application is described below:

[0033] See also Figure 1 , Figure 1 A schematic diagram of a method flow chart provided in an embodiment of the present application. In this embodiment, the method is executed by a network device.

[0034] like Figure 1 As shown, the process may include the following steps:

[0035] S101, for the metadata of the file to be cached, if the file is created for the first time in the file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; if the file is currently in the state of being executed with a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected. The specific caching method of metadata will be described in the following text in the form of a specific embodiment, which will not be repeated here.

[0036] In this embodiment, the designated write operation includes a hard link or a duplicate name designated write operation.

[0037] It should be noted that although the link between CDentry and CInode is disconnected, the CDentry and CInode of the same file can be determined based on the file name of CDentry and the information in CInode.

[0038] S102, for the metadata of the file to be persisted, if the CDentry and CInode in the metadata in the cache are linked to each other, then the key key of the directory item Dentry is constructed based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name, and the value value of Dentry is constructed using the preset first-class value construction method Dn(I) to persist Dentry and Inode; wherein Dn(I) refers to a method of constructing value based on all the information of Inode; the bucketid of the metadata of any file is determined based on the principle of metadata scattering.

[0039] In this embodiment, the construction of a complete directory tree on each MDS is abandoned. According to the metadata scattering rule, each MDS only manages its own metadata, and the metadata is only cached on one MDS. The specific implementation method of the metadata scattering rule will also be explained in the form of a specific embodiment later, and will not be repeated here.

[0040] S103, if the link between CDentry in the metadata in the cache and the index node CInode is disconnected, then the key of Dentry is constructed based on the bucketid corresponding to the metadata, the index node number of the parent index node of Inode, and the file name name, and the value of Dentry is constructed using the preset second-class value construction method Dn(L). The key of Inode is constructed based on the bucketid corresponding to the metadata and the index node number of Inode, and the value of Inode is constructed based on the Dn(I) method to store Dentry and Inode persistently respectively; Dn(L) refers to the method of constructing the value of Dentry using the key when the Inode corresponding to Dentry is persisted.

[0041] In this embodiment, the specific implementation of the above S102 will also be described in detail later in the form of a specific embodiment, which will not be repeated here.

[0042] So far, completed Figure 1 The process shown.

[0043] pass Figure 1 As can be seen from the flow, in this embodiment, when caching metadata, for a file created for the first time in this file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; for a file that has been subjected to a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected. This makes only one copy of the metadata in the cache, reduces memory resource usage, and avoids complex cache synchronization, thereby reducing request amplification and ensuring system performance.

[0044] Furthermore, when persisting metadata, if CDentry and CInode in the cache are linked to each other, the key key of the directory item Dentry is constructed based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name, and the value value of Dentry is constructed using the preset first-class value construction method Dn(I) to persist Dentry and Inode. If the link between CDentry and index node CInode in the metadata in the cache is disconnected, De is constructed based on the bucketid corresponding to the metadata, the index node number of the parent index node of Inode, and the file name name. The key of ntry is constructed by using the preset second-class value construction method Dn(L) to construct the value of Dentry. The key of Inode is constructed based on the bucketid corresponding to the metadata and the index node number of Inode. The value of Inode is constructed based on the Dn(I) method to store Dentry and Inode persistently respectively. This persistence method can ensure that the required metadata information can be obtained through simple queries. On the other hand, Dentry and Inode are persisted separately without forcibly binding Dentry and Inode. When renaming or hard linking operations across multiple MDSs occur, metadata migration is avoided, which further ensures system performance.

[0045] In order to explain the method provided by the present application in more detail, the scheme provided by the present application is described in more detail below in the form of specific embodiments.

[0046] The first aspect: In order to better explain this solution, the above metadata is first broken down and explained:

[0047] As an embodiment, the directory fragmentation is divided into 8 pieces by default, which can be configured and adjusted. The metadata is scattered to a certain directory fragment according to the file name (denoted as, Dentry->name), and the directory fragmentation is scattered to a fixed bucket (8192) according to the inode number of the directory and the ID of the directory fragment (denoted as, dir->inode+dir->frag). The bucket is then addressed to a certain MDS, which is the home node. The metadata operation (such as cache and hard link) is processed by the home node. At this time, the MDS cluster is no longer in the master-slave mode, but in the full master mode. Each MDS only processes its own metadata.

[0048] In the above method, instead of building a complete directory tree on each MDS, each MDS only manages its own metadata based on the metadata sharding rules and combined with the metadata cache method. The metadata has no copies and is cached on only one MDS. In addition, the directory shards are hashed to a certain bucket. When the number of MDSs remains unchanged, the mapping relationship between buckets and MDSs remains unchanged, the name remains unchanged, the bucket remains unchanged, and the MDS remains unchanged. The metadata will not change the MDS to which the inode belongs due to link / rename operations, thus not causing metadata migration.

[0049] The above is an explanation of the above metadata breakdown.

[0050] The second aspect: The following is a detailed explanation of metadata persistence and caching:

[0051] As an embodiment, the metadata cache stage specifically includes the following steps:

[0052] 1. For newly created common files, CInode and CDentry are linked to each other, CInode's parent points to the corresponding CDentry, and CDentry's Inode points to the corresponding CInode.

[0053] 2. When linking or renaming a file, disconnect the link between CInode and CDentry, link CInode to the index node list (recorded as mdcache->nlinode_lru) for cache elimination; assign link_info of CDentry to save bucketid, inode number, and file type of CInode; assign bucketid of CInode.

[0054] In this way, even when link and rename operations are performed, there is only one copy of the Inode in the cache, and the link between the Dentry and the Inode no longer requires the MDS to send requests to carry metadata information, thus avoiding metadata migration.

[0055] 3. Each CDentry is linked to the directory necklace table mdcache->dentry_lru, and each CInode that is not linked to CDentry is linked to the index node list mdcache->nlinode_lru, so that cache elimination can be performed when the set cache elimination conditions are met. CInodes in the cache are linked to the index node mapping map (denoted as mdcache->inode_map), and CDirs in the cache are linked to the index node mapping map (denoted as mdcache->dir_map). At the same time, CDentry in the cache is linked to its own parent directory fragment dir->dentry_items for cache search.

[0056] As an embodiment, the metadata that needs to be persisted include: directory fragment Dir, Dentry, Inode, Backtrace. The persistent storage stage specifically includes the following steps:

[0057] 1. Persistent storage of dir

[0058] dir is persistent. The key contains bucketid, key_type, ino (inode number of the directory itself), and frag (fragment number). The key_type is TYPE_DIR, and the value contains the statistical information of the subfiles and subdirectories under the directory.

[0059] For example, the dir persistent key-value is shown in the following table:

[0060]

[0061] 2. Persistent storage of Dentry

[0062] Dentry persistence, the key includes bucketid, key_type, ino (parent directory inode number), frag (fragment number), and file name.

[0063] For example, the Dentry persistence key is shown in the following table:

[0064] Bucketid key_type Ino Frag name u16 u16 u64 u32 Str

[0065] Dentry persistence, there are two types of value, Cdentry and Cinode are linked to each other, the preset first type of value construction method Dn(I) constructs the value of Dentry. At this time, the type of the value persisted by Dentry is "I", which records all the information of the metadata inode, where ino is the inode number of the file itself. The value after Dn(I) is subsequently referred to as Dn(I).

[0066] For example, the value after Dn(I) is shown in the following table:

[0067]

[0068] When the link between Cdentry and Cinode is disconnected, the persistence will also be separated. The format of Dentry's key is the same as the first case, and the key still contains bucketid, key_type, ino (parent directory inode number), frag (fragment number), and file name name. The value of Dentry is constructed using the preset second-class value construction method Dn(L). The type of value is "L" and link_info is recorded. Bucketid is the bucketid at the time of initial creation, not the bucketid of the current dentry (the value may be the same), remote_ino is the inode number of the file / directory itself, and the value after Dn(L) is abbreviated as dn(L).

[0069] For example, the value after Dn(L) is shown in the following table:

[0070]

[0071] 3. Persistent storage of Inode

[0072] When the link between Cdentry and Cinode is disconnected, dn(L) information only records link_info, and Inode information needs to be persisted separately. The key includes bucketid, key_type, and ino. Among them, key_type is TYPE_INODE, bucketid is the bucketid when it is initially created, and ino number is the inode number of the file / directory itself. Its value is the same as bucketid and remote_ino in link_info in dn(L).

[0073] For example, the values ​​of a single persistent Inode are shown in the following table:

[0074] Bucketid key_type ino u16 u16 u64

[0075] The value of a single persistent Inode is the same as the value of dn(I), as shown in the following table.

[0076]

[0077] It can be seen that the value of dn(L) actually records the key information of the key that persists its inode information.

[0078] 4. Persistent storage of backtrace

[0079] Backtrace is generally used in scenarios where only the file inode number is known and the cache of the complete path is constructed upwards, such as supporting filehandle mapping in nfs scenarios and supporting the provision of the inode_to_fullpath interface. As can be seen from the above description, only the file inode number cannot construct the key for metadata persistence, so only the inode cannot directly read metadata from the Database (database, db). Therefore, for operations such as getattr, setattr, etc. on the inode number, if the corresponding metadata is not found in the cache, backtrace needs to be used to obtain enough information to construct the db key.

[0080] The backtrace key includes ino, where ino is the inode number of the file itself.

[0081] For example, the backtrace key is shown in the following table:

[0082] Prefix Ino "b" u64

[0083] There are also two types of backtrace values, corresponding to the value types of Dentry. CDentry and CInode are linked to each other. In the first case, the backtrace value is constructed using the preset third-class value construction method bt(I). The backtrace value contains bucketid, key_type, ino (parent directory inode number), frag (fragment number), and file name, denoted as bt(I).

[0084] For example, the value of bt(I) is as follows:

[0085] Prefix dentry_db_key "I" bucketid key_type ino Frag name

[0086] Corresponding to the value type of Dentry, the link between CDentry and CInode is disconnected. In the second case, the value of backtrace is constructed using the default fourth type value construction method bt(L). The value of backtrace contains filepaths, which is a linked list. Each linked list node stores the relevant information of a dentry, referred to as fp, and recorded as bt(L).

[0087] For example, the value of bt(L) is as follows:

[0088]

[0089] In this way, when the client has a cache and can obtain the bucketid, the request is addressed to the MDS according to the bucketid; if the client only has the inode number, it can first take the middle 10 bits of the inode number to obtain the MDS where the backtrace is located, and then query the backtrace to obtain the required information.

[0090] By caching and persisting in the above way, the following operations are performed when reading:

[0091] For the cache search of Dentry, first find the directory fragment dir through the directory ino and frag, and then search for Dentry from dir->dentry_items; for the cache elimination of Dentry, traverse mdcache->dentry_lru to eliminate the Dentry cache.

[0092] For cache lookup of Inode linked to Dentry, path lookup can be performed based on Dentry to obtain Inode; or it can be searched from mdcache->Inode_map based on Inode number; for cache elimination of Inode linked to Dentry, Inode elimination is performed simultaneously with Dentry elimination.

[0093] For cache lookups of Inodes that are not linked to Dentry, you can only look them up from mdcache->inode_map based on the Inode number; for cache elimination of Inodes that are not linked to Dentry, traverse mdcache->nlinode_lru to eliminate the Cinode cache.

[0094] For path lookup, when the authoritative MDS of the Inode of the Dentry is not the same MDS, the request will be sent to the MDS of the Dentry, and the link_info will be returned to the client; the client will then calculate the MDS of the Inode by the bucketid in the link_info, and send a getattr request to the MDS of the Inode to obtain the Inode information. Although distributed read is still required, it does not actually increase network requests and reduces access to data objects.

[0095] The above describes metadata persistence and caching in detail.

[0096] The third aspect: The following describes the creation of persistent storage for hard links:

[0097] See also Figure 2 , Figure 2A schematic diagram of the process of hard link persistence provided by an embodiment of the present application. The hard link operation involves two files, the target newly created file is called dest, and the source file is called src.

[0098] like Figure 2 As shown, the persistent storage of hard links specifically includes the following steps:

[0099] S201, when establishing a hard link, if it is determined that it is the first time to establish a hard link for the source index node src inode, the source cache directory item src CDentry and the source cache index node src Cinode corresponding to the source file in the cache are disconnected, and the new target directory item dest dn (L) is persisted in the form of Dn (L), and the new source directory item src dn (L) created in the form of Dn (L) overwrites the source directory item src dn (I) constructed in the form of Dn (I), and the number of links in the src inode is updated.

[0100] S202: If it is determined that this is not the first time to establish a hard link for the src inode, the newly created target directory entry dest dn (L) is persisted in the form of Dn (L), and the number of links in the src inode is updated.

[0101] Specifically, when a hard link is created for the first time, a new dest dentry is created, the src inode in the cache is disconnected from the srcdentry, a new dest dn(L) is created persistently, the new src dn(L) overwrites the original src dn(I), the src inode is updated, the dest dir count is updated, and the backtrace is updated at the same time, bt(I) is updated to bt(L), and filepaths records the information of src and destdentry.

[0102] It should be noted that if no new inode is added, no new backtrace will be created.

[0103] Create a hard link again, create a new dest dentry, increase the src inode nlink by 1, create a new destdn(L) persistently, update the src inode, update the count of the dest dir, and update bt(L) at the same time. Filepaths adds the information of the dest dentry.

[0104] For example, the specific process of link operation persistence is shown in the following figure:

[0105]

[0106] When it is determined to delete any established hard link, the dest dn (L) corresponding to the hard link is deleted, and the number of links in the srcinode is updated, or the src inode is deleted when the deletion condition is met.

[0107] Specifically, when deleting a hard link, delete the dest dentry, update the inode's nlink minus 1, delete the dest dn(L) persistently, update or delete the inode (when nlink-1 is 0), update the dir count, and delete the dest dentry from the filepaths of bt(L) or delete the entire bt(L) (when the last dentry is deleted from filepaths).

[0108] For example, the specific process of persistence of deleting a link operation is shown in the following figure:

[0109]

[0110] The above describes in detail the following steps for creating persistent storage for hard links.

[0111] The fourth aspect: The following is a detailed description of the renamed persistent storage:

[0112] See also Figure 3 , Figure 3 The following is a schematic diagram of the renaming persistence process provided by the embodiment of the present application. Figure 3 As shown, the persistent storage of hard links specifically includes the following steps:

[0113] S301, when determining to rename the source file or source directory to the target file or target directory, if the target file or target directory does not exist in the system, when there is a link between the source cache directory entry srcCDentry and the source cache index node src Cinode corresponding to the source file or source directory, disconnect the srcCDentry and src Cinode in the cache, delete the source directory entry src dn(I) constructed in the Dn(I) mode, persist the target directory entry dest dn(L) of the newly created target file in the Dn(L) mode, and persist the src inode.

[0114] S302, when the link between src CDentry and src Cinode corresponding to the source file or source directory is disconnected, the source directory entry src dn (L) constructed in the Dn (L) mode is deleted, and the target directory entry dest dn (L) of the newly created target file is persisted in the Dn (L) mode.

[0115] Specifically, during renaming, if the target file / target directory does not exist and the file is being renamed, if the source file is an unsplit dentry, it needs to be split, delete the src dn(I), create a new src inode and dest dn(L), update bt(I) to bt(L), and filepaths records the dest dentry information.

[0116] If the source file has been split, you only need to delete one src dn(L), create a new dest dn(L), update the filepaths in bt(L), delete the src dentry information, and record the dest dentry information.

[0117] If the target is renamed, the Inode number will not be changed during the renaming, and the bucketid calculated based on the Inode number and frag will not change. Therefore, the directory fragment will not be migrated due to the change of the directory name, which is the same as the file renaming operation.

[0118] For example, the specific process of persistence of the rename operation when the target file / target directory does not exist is shown in the following figure:

[0119]

[0120] S303, use the existing target file as the historical target file, or use the existing target directory as the historical target directory; when there is a link between the historical target file cache directory item oldCDentry corresponding to the historical target file and the historical target file cache index node old Cinode, disconnect the old CDentry and old Cinode in the cache, and persist the target directory item dest dn(L) of the newly created target file in the form of Dn(L) to overwrite the historical target file directory item old Dn(I) constructed in the form of Dn(I).

[0121] S304, when the link between the historical target file cache directory entry old CDentry corresponding to the historical target file and the historical target file cache index node old Cinode is disconnected, the target directory entry dest dn(L) of the newly created target file is persisted in the Dn(L) manner to overwrite the historical target file directory entry old Dn(L) constructed in the Dn(L) manner.

[0122] Specifically, if dest is an existing file or an existing target directory. If the target file dest exists, it is called old for easy distinction. There is no difference in the operation of src metadata. Only the operation of old related metadata kv needs to be added. For old dn, if it is an unsplit dentry, a new dest dn (L) is created to overwrite old dn (I), and old bt (I) is deleted. If it is a target directory and the directory is determined to be empty and unsplit, in addition to the above steps, all directory fragments dir need to be deleted.

[0123] If the original file has been split, the new dest dn(L) will overwrite the old dn(L), the old inode will be updated or deleted (when nlink-1 is 0), and the destdentry will be deleted from the filepaths of the old bt(L) or the entire old bt(L) will be deleted (when the last dentry is deleted from filepaths).

[0124] For example, the specific process of persistence of the rename operation when the target file / target directory exists is shown in the following figure:

[0125]

[0126] It should be noted that the above hard link persistence operations involve at most two MDS nodes in a non-expanded cluster: the MDS to which dest dn belongs and the MDS to which src in belongs. In an expanded cluster, at most three MDS nodes are involved: the MDS to which dest dn belongs, the MDS to which src in belongs, and the MDS where backtrace is located.

[0127] The request is sent to the MDS to which dest dn belongs, which is called Master MDS. The requirements for atomicity and cache consistency for the MDS to which src in belongs and the MDS where backtrace is located are not exactly the same. If the backtrace update is delayed, we access the backtrace to obtain metadata ownership-related information and related information for constructing db keys. We also need to obtain the final metadata from the MDS to which dn or in belongs. As long as dentry and inode are atomically modified to the client, the delay in backtrace updates only affects efficiency. Therefore, the MDS to which src in belongs is called slave MDS, and the request between Master MDS and slave MDS is called slave req; the MDS where backtrace is located is called witnessMDS, and the request between Master MDS and slave MDS is internal req.

[0128] Similarly, the above rename persistence operations involve at most three MDS nodes in a non-expanded cluster: the MDS to which dest dn belongs, the MDS to which src dn belongs, and the MDS to which old in belongs; in an expanded cluster, at most five MDS nodes are involved: the MDS to which dest dn belongs, the MDS to which src dn belongs, the MDS to which old in belongs, and the MDS to which src and oldbacktrace are located. Renaming a directory involves more MDSs, namely the MDS to which the directory shards belong.

[0129] The request is sent to the MDS of dest dn, i.e., Master MDS, the MDS of src dn and old in, i.e., Slave MDS, and the MDS where backtrace is located, i.e., Witness MDS. The above is the same as hard link. Because the operation of directory fragments must first access inode, if old in has been deleted, the directory fragments cannot be accessed after deletion, so the MDS of directory fragments is also called Witness MDS, and the request between it and Master MDS is internal req.

[0130] Based on the above description, whether the target directory is empty is determined in the following way: the master metadata server Master MDS sends an instruction to the associated MDS associated with the target directory in the system to check whether the directory slice of the local target directory is empty; the Master MDS is the MDS in the system used to receive external input write requests; the associated MDS refers to the MDS where the target slice of the target directory is located. When it is determined that the directory slice on any associated MDS is not empty, it is determined that the target directory is not empty; when it is determined that the directory slices on all associated MDSs are empty, it is determined that the target directory is empty.

[0131] For example, if Figure 4 As shown, judging whether the target directory is empty includes the following steps:

[0132] 1. For Master MDS (that is, inode belongs to MDS)

[0133] After the pre-check is completed, special marks are added to CDentry and CInode, and revoke cap messages are sent to all clients holding the dir cap where the dentry is located and all clients holding the inode cap;

[0134] When the client receives the revoke cap message, it invalidates the corresponding metadata cache mark. If there is an operation in the directory later, it needs to resend the getattr request to the MDS. The request is sent to the MDS where the inode is located. If it is checked for a special mark, it needs to wait.

[0135] The MDS continues to send EMPTY_PRECHECK requests to the MDSs of all shards to check whether the directory is empty. After receiving a reply that a directory shard is not empty, the MDS cancels the special marks of CDentry and CInode and replies ENOEMPTY to the client.

[0136] If the directory is empty, you need to receive a reply that all directory fragments are empty before continuing the request (rename / rmdir). After the request is processed, the directory fragment has been deleted, the special marks of CDentry and CInode are canceled, and the client is replied.

[0137] After the mark is cancelled, the previously waited op is triggered. If the directory is not empty, the client re-acquires the metadata information; if the directory is empty and the deletion is successful, an error is returned to the client.

[0138] 2. The slave MDS is not the inode-owned MDS

[0139] The MDS to which the inode belongs is the slave MDS. After the Master MDS completes the pre-check of destdn, it adds a special mark to CDentry and sends a revoke cap message to all clients holding the dir cap where the dentry is located. Then it sends slavereq to the slave MDS, which processes the pre-check of the inode and the judgment of whether the directory is empty.

[0140] After receiving the slave req, the slave MDS completes the inode pre-check, adds a special mark to the CInode, sends a revoke cap message to all clients holding the inode cap, and sends an EMPTY_PRECHECK request to the MDSs of all shards to check whether the directory is empty.

[0141] The above describes the renamed persistent storage in detail:

[0142] The fifth aspect: The following is a detailed explanation of distributed transactions:

[0143] In this embodiment, the distributed write operation involves multiple MDSs, which are further divided into MasterMDS, slaveMDS, and witness MDS. The request between the Master MDS and the slave MDS is called slave req, and the request between the Master MDS and the witness MDS is called internal req.

[0144] It should be noted that the witness MDS here includes Figure 4 The associated MDS is shown.

[0145] When the master metadata server Master MDS receives a write request, after completing the local pre-check, after determining to execute the write request, it adds a specified tag to the metadata in the local cache to prevent other write requests from occupying the metadata, and sends an instruction to each Slave MDS to instruct each slave metadata server slave MDS to perform a local pre-check to add a specified tag to the metadata in the local cache to prevent other write requests from occupying the metadata, and after executing the first write operation corresponding to the write request locally, it sends a reply message to the Master MDS; wherein the Master MDS is the node to which the metadata corresponding to the write operation is determined based on the metadata scattering principle; the node where the inode of the metadata corresponding to the Slave MDS write operation is stored;

[0146] After the Master MDS receives the reply information from all the Slave MDSs, it executes the persistence operation corresponding to the write request;

[0147] When the write request of either the Master MDS or the slave MDS fails, the Master MDS deletes the specified tag from the metadata in the local cache and sends an error code to each slave MDS to trigger the rollback operation of each slave MDS.

[0148] When the Master MDS executes a write request successfully, it updates the original data in the local cache, deletes the specified tag, and sends an instruction to each Slave MDS to instruct each Slave MDS to perform the second write operation corresponding to the write request locally. When each Slave MDS returns the write request success information, it outputs a write completion indication.

[0149] In this embodiment, the first write operation is an operation related to persistent SET, and the second write operation is an operation related to delete.

[0150] Distributed transactions are mainly transactions between the master and the slave, so the stages of distributed transactions are divided into the PREP stage and the FINISH stage according to the sending of two slave reqs; both the PREP stage and the FINISH stage may interact with the witness and send internal requests, but the difference is that the internal req in the PREP stage does not involve database write operations. The following example illustrates distributed transactions from the PREP stage and the FINISH stage.

[0151] For example, if Figure 5 As shown, distributed transaction processing specifically includes the following steps:

[0152] For the PREP stage, the following steps are included:

[0153] 1. The Master MDS receives the client request and creates a coroutine to process the request, recorded as m-ult-1. After the Master MDS completes the pre-check, it adds a special mark to the metadata in the local cache, locks the resources, sends an OP_PREP request to all slave MDSs, waits for a condition variable and switches out the coroutine.

[0154] 2. The slave MDS receives the slave req and creates a coroutine for processing, recorded as s-ult-1. The slave MDS completes the pre-check. If it can be executed, it adds a special mark to the metadata in the cache, locks the resources, records event_rollback, executes the operation, submits db SET, does not submit db DEL, and does not modify the cache metadata.

[0155] It should be noted that event_rollback requires the submission of db SET, so the metadata is submitted at the same time; if db DEL is not submitted, the information that event_rollback needs to save can be reduced, and the data can be truly deleted and cannot be rolled back.

[0156] 3. The slave MDS completes the above operations, replies to the Master MDS, waits for a condition variable and cuts out the coroutine.

[0157] 4. Master MDS receives the reply from slave MDS, creates a coroutine for processing, recorded as m-ult-2, records the result and checks whether m-ult-1 can be triggered to continue execution, and m-ult-2 ends; after all slave MDSs reply, Master MDS summarizes the results and triggers m-ult-1 to continue execution.

[0158] 5. The Master MDS receives a reply from the slave MDS. If one slave MDS fails, it enters the FINISH phase.

[0159] 6. If the slave MDS executes successfully, the Master MDS records event_uncommitted, executes the operation, and submits db SET and db DEL. After db is submitted successfully, it enters the FINISH stage.

[0160] For the FINISH stage, the following steps are included:

[0161] 7. After both the Master MDS and the Slave MDS have completed the PREP phase, the Master MDS processes the Finish phase first. If the Slave MDS fails or the Master MDS fails to submit the db, the Master MDS cancels the special mark, unlocks the resources, and sends an OP_FINISH to all Slave MDSs with the error code FAILED, notifying the Slave MDSs to perform a rollback.

[0162] 8. If the Master MDS successfully submits the db and there is no witness MDS (if there is a witness MDS, add a handle_uncommitted process, which belongs to the FINISH stage), update the cache, cancel the special mark, unlock the resource, and reply to the client; and send OP_FINISH to all slave MDSs with the error code SUCCESS, wait for a condition variable and cut out the coroutine;

[0163] 9. The slave MDS receives OP_FINISH and creates a coroutine for processing, recorded as s-ult-2. After recording the result, it triggers s-ult-1 to continue execution, and s-ult-2 ends. s-ult-1 determines whether to execute slaverollback or slavefinish based on the error code.

[0164] 10. If a slave rollback is executed, the rollback operation is performed according to the event_rollback record, and event_rollback DEL, db SET, and db DEL are submitted;

[0165] If slave finish is executed, first update the cache, cancel the mark, unlock the resources, and then submit db DEL and event_rollback DEL according to the event_rollback record; after db is submitted successfully, reply to the Master MDS and end the coroutine task s-ult-1.

[0166] 11. Master MDS also needs to summarize the replies from all slave MDSs and trigger m-ult-1 to continue execution; m-ult-1 submits event_uncommitted DEL, and after db submits successfully, the coroutine task m-ult-1 ends.

[0167] The request between the Master MDS and the Witness MDS is called an internal req, and only one request is required. Only internal reqs for cache reading or db reading, such as directory non-empty judgment, are executed in the PREP phase of the distributed transaction; internal reqs that may involve db writing, such as updating backtrace, deleting directory fragments, etc., are executed in the FINISH phase of the distributed transaction; all are initiated by the Master MDS. After the Master MDS sends INTERNAL_OP_XX, it waits for a condition variable and exits the coroutine;

[0168] 12. The witness MDS receives the request, executes the operation, submits the request to the database, and returns the request to the Master MDS.

[0169] 13. Master MDS receives the reply and creates a coroutine, recorded as m-ult-i. It records the result and checks whether it can trigger m-ult-1 to continue execution. Then m-ult-i ends. After all witness MDSs reply, Master MDS summarizes the results and triggers m-ult-1 to continue execution.

[0170] In order to more specifically reflect the above five aspects, the solution provided in this application is described in more detail below by renaming the specific embodiments.

[0171] See also Figure 6 , Figure 6 The interactive flow chart of renaming provided in the embodiment of the present application. Figure 6 As shown, the process specifically includes the following steps:

[0172] 1. The client receives the rename instruction, parses the path, and sends a request to the mds to which the target file destdn belongs, that is, the Master MDS.

[0173] 2. Master MDS performs pre-check:

[0174] According to the cost, whether the directory is empty needs to be checked last, so the metadata acquisition and checking order is: destdn-->srcdn-->oldin, Master MDS (except the directory is not empty) and resource lock, find destdn, if destdn does not exist, create a new Dentry.

[0175] destdn exists, oldin is local and linked to destdn, check the status of destdn and oldin, and if there is a special status mark, request a retry.

[0176] Mark the state of destdn, oldin (linked to destdn) as RENAMING.

[0177] Send a cap REVOKE request to the client holding oldin, destdir (current shard) caps. If srcdn belongs to the local mds, pre-check and lock the resources of srcdn.

[0178] If oldin belongs to the local mds and is not linked to destdn, send SLAVE_PREP for oldin pre-check, resource lock and directory non-empty check.

[0179] If the MDS to which srcdn and oldin belong is not the local MDS, send slave req to the MDS to which srcdn and oldin belong.

[0180] 3. Slave MDS pre-check, resource lock, revoke cap, if it is oldin and it is a directory, check if the directory is not empty.

[0181] After the check is completed, submit event_rollback SET and metadata db SET. For the specific persistence behavior, please refer to the renaming persistence step in the fourth aspect above, which will not be repeated here.

[0182] After the persistence is successful, reply to the Master MDS; srcdn needs to reply link_info, and if oldin is a directory, it needs to reply the number of shards; the key information is only 15 bytes.

[0183] 4. Master MDS prep

[0184] After receiving all slave req replies, master mds commits event_uncommitted SET and metadata dbSET and db DEL.

[0185] 5. Master MDS finish

[0186] Cache update, handle_uncommitted, send internal req to the corresponding mds to update backtrace or delete directory fragments.

[0187] Unmark, unlock the resource, and reply to the client.

[0188] If the mds to which srcdn or oldin belongs is not the local mds, send SLAVE_OP_FINISH.

[0189] 6. Slave MDS finish

[0190] The Slave MDS updates the cache, cancels the mark, unlocks the resource, and then submits dbDEL and event_rollback DEL according to the event_rollback record; after the persistence is successful, it replies to the Slave MDS

[0191] 7. Slave MDS commit

[0192] Submit event_uncommitted DEL. After persistence succeeds, the rename process ends.

[0193] The method provided in the embodiment of the present application is described above. The device provided in the embodiment of the present application is described below:

[0194] See also Figure 7 , Figure 7 This is a diagram of the structure of the device provided in the embodiment of the present application. The device is applied to network equipment, such as Figure 7 As shown, the device may include: a cache module 701, a first persistence module 702, and a second persistence module 703.

[0195] The cache module 701 is used for the metadata of the file to be cached. If the file is created for the first time in the file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; if the file is currently in the state of executing a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected;

[0196] The first persistence module 702 is used for constructing the key key of the directory item Dentry based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name for the metadata of the file to be persisted, and constructing the value value of Dentry in a preset first-class value construction method Dn(I) to persist Dentry and Inode; wherein Dn(I) refers to a method of constructing value based on all information of Inode; the bucketid of the metadata of any file is determined according to the principle of metadata scattering;

[0197] The second persistence module 703 is used for, if the link between CDentry in the metadata and the index node CInode in the cache is disconnected, constructing the key of Dentry based on the bucketid corresponding to the metadata, the index node number of the parent index node of Inode, and the file name name, constructing the value value of Dentry in the preset second type of value construction method Dn(L), constructing the key of Inode based on the bucketid corresponding to the metadata and the index node number of Inode, and constructing the value of Inode based on the Dn(I) method, so as to store Dentry and Inode persistently respectively; Dn(L) refers to the method of constructing the value of Dentry with the key when the Inode corresponding to Dentry is persisted;

[0198] Among them, when caching metadata, for files created for the first time in this file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; for files that have been subjected to specified write operations, the link between CDentry and CInode of the metadata of the file is disconnected.

[0199] As an embodiment, each CDentry is linked to the directory necklace table mdcache->dentry_lru, and each CInode not linked to CDentry is linked to the index node linked list mdcache->nlinode_lru, so as to perform cache elimination when the set cache elimination condition is met.

[0200] As an embodiment, the device further includes:

[0201] A hard link module is used for, when establishing a hard link, if it is determined that it is the first time to establish a hard link for the source index node src inode, disconnecting the source cache directory entry src CDentry and the source cache index node srcCinode corresponding to the source file in the cache, persisting the newly created target directory entry dest dn (L) in the form of Dn (L), and overwriting the source directory entry src dn (I) constructed in the form of Dn (I) with the newly created source directory entry src dn (L) in the form of Dn (L), and updating the number of links in the src inode;

[0202] If it is determined that this is not the first time a hard link is established for the src inode, the new target directory entry dest dn(L) is persisted in the form of Dn(L), and the number of links in the src inode is updated.

[0203] As an embodiment, the hard link module is further used to:

[0204] When it is determined to delete any established hard link, the dest dn (L) corresponding to the hard link is deleted, and the number of links in the srcinode is updated, or the src inode is deleted when the deletion condition is met.

[0205] As an embodiment, the device further includes:

[0206] The rename module is used to determine whether to rename a source file or source directory to a target file or target directory. If the target file or target directory does not exist in the system, then:

[0207] When there is a link between the source cache directory entry src CDentry and the source cache index node srcCinode corresponding to the source file or source directory, disconnect the link between the src CDentry and src Cinode in the cache, delete the source directory entry src dn(I) constructed in the Dn(I) mode, persist the target directory entry dest dn(L) of the newly created target file in the Dn(L) mode, and persist the src inode;

[0208] When the link between src CDentry and src Cinode corresponding to the source file or source directory is disconnected, the source directory entry src dn(L) constructed in the Dn(L) mode is deleted, and the target directory entry dest dn(L) of the newly created target file is persisted in the Dn(L) mode.

[0209] As an embodiment, the renaming module is further used to:

[0210] If the target file or target directory exists in the system and is determined to be empty, then:

[0211] If the target file or target directory exists in the system and is determined to be empty, then:

[0212] Use an existing target file as a historical target file, or use an existing target directory as a historical target directory;

[0213] When there is a link between the historical target file cache directory entry old CDentry corresponding to the historical target file and the historical target file cache index node old Cinode, disconnect the link between old CDentry and old Cinode in the cache, and persist the target directory entry dest dn(L) of the newly created target file in the form of Dn(L) to overwrite the historical target file directory entry old Dn(I) constructed in the form of Dn(I);

[0214] When the link between the historical target file cache directory entry old CDentry corresponding to the historical target file and the historical target file cache index node old Cinode is disconnected, the target directory entry dest dn(L) of the newly created target file is persisted in the Dn(L) manner to overwrite the historical target file directory entry old Dn(L) constructed in the Dn(L) manner.

[0215] As an embodiment, determining whether the target directory is empty includes:

[0216] The master metadata server Master MDS sends an instruction to the associated MDS associated with the target directory in the system to check whether the directory slice of the local target directory is empty; the Master MDS is the MDS in the system for receiving external input write requests; the associated MDS is the MDS where the target slice of the target directory is located;

[0217] When it is determined that the directory fragment on any associated MDS is not empty, it is determined that the target directory is not empty;

[0218] When it is determined that the directory slices on all the associated MDSs are empty, it is determined that the target directory is empty.

[0219] As an embodiment, the device further includes:

[0220] Distributed processing modules,

[0221] A distributed processing module is used for, when the master metadata server Master MDS receives a write request, after completing the local pre-check and determining to execute the write request, adding a specified tag to the metadata in the local cache to prevent other write requests from occupying the metadata, and sending an instruction to each Slave MDS to instruct each slave metadata server slave MDS to perform a local pre-check to add a specified tag to the metadata in the local cache to prevent other write requests from occupying the metadata, and after executing the first write operation corresponding to the write request locally, sending a reply message to the Master MDS; wherein the Master MDS is the node to which the metadata corresponding to the write operation is determined based on the metadata scattering principle; the node where the inode of the metadata corresponding to the Slave MDS write operation is stored;

[0222] After the Master MDS receives the reply information from all the Slave MDSs, it executes the persistence operation corresponding to the write request.

[0223] As an embodiment, the distributed processing module is further used to:

[0224] When the write request of either the Master MDS or the slave MDS fails, the Master MDS deletes the specified tag from the metadata in the local cache and sends an error code to each slave MDS to trigger the rollback operation of each slave MDS.

[0225] When the Master MDS executes a write request successfully, it updates the original data in the local cache, deletes the specified tag, and sends an instruction to each Slave MDS to instruct each Slave MDS to perform the second write operation corresponding to the write request locally. When each Slave MDS returns the write request success information, it outputs a write completion indication.

[0226] So far, completed Figure 7 Structural description of the device shown.

[0227] See also Figure 8 , Figure 8 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0228] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0229] Exemplarily, the above-mentioned machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0230] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for writing to a distributed file system, characterized in that: The method is applied to a network device, and the method comprises: For the metadata of the file to be cached, if the file is created for the first time in this file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; if the file is currently in the state of being executed with a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected; For the metadata of the file to be persisted, if CDentry and CInode in the metadata in the cache are linked to each other, the key key of the directory item Dentry is constructed based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name, and the value value of the Dentry is constructed in the preset first-class value construction method Dn(I) to persist the Dentry and the Inode; wherein, the Dn(I) refers to the method of constructing value based on all the information of the Inode; the bucketid of the metadata of any file is determined according to the principle of metadata scattering; If the link between CDentry and CInode in the metadata in the cache is disconnected, the key of the Dentry is constructed based on the bucketid corresponding to the metadata, the index node number of the parent index node of the Inode, and the file name name, and the value of the Dentry is constructed using the preset second-class value construction method Dn(L). The key of the Inode is constructed based on the bucketid corresponding to the metadata and the index node number of the Inode, and the value of the Inode is constructed based on the Dn(I) method, so that the Dentry and the Inode are respectively persistently stored; the Dn(L) refers to a method of constructing the value of the Dentry using the key when the Inode corresponding to the Dentry is persisted.

2. The method according to claim 1, characterized in that The method further comprises: Each CDentry is linked to the directory necklace table mdcache->dentry_lru, and each CInode that is not linked to CDentry is linked to the index node list mdcache->nlinode_lru to perform cache elimination when the set cache elimination conditions are met.

3. The method according to claim 1, characterized in that The method further comprises: When establishing a hard link, if it is determined that it is the first time to establish a hard link for the source index node src inode, the source cache directory entry src CDentry and the source cache index node src Cinode corresponding to the source file in the cache are disconnected, the new target directory entry dest dn (L) is persisted in the manner of Dn (L), and the new source directory entry src dn (L) created in the manner of Dn (L) overwrites the source directory entry src dn (I) constructed in the manner of Dn (I), and the number of links in src inode is updated; If it is determined that this is not the first time that a hard link is established for the src inode, the newly created target directory entry dest dn (L) is persisted in the manner of Dn (L), and the number of links in the src inode is updated.

4. The method according to claim 3, characterized in that The method further comprises: When it is determined to delete any established hard link, the dest dn (L) corresponding to the hard link is deleted, and the number of links in the srcinode is updated, or the src inode is deleted when the deletion condition is met.

5. The method according to claim 1, characterized in that The method further comprises: When determining to rename a source file or source directory to a target file or target directory, if the target file or target directory does not exist in the system, then: When there is a link between the source cache directory entry src CDentry and the source cache index node srcCinode corresponding to the source file or source directory, disconnect the link between the src CDentry and the src Cinode in the cache, delete the source directory entry src dn(I) constructed in the Dn(I) manner, persist the target directory entry dest dn(L) of the newly created target file in the Dn(L) manner, and persist the src inode; When the link between src CDentry and src Cinode corresponding to the source file or source directory is disconnected, the source directory entry src dn (L) constructed in the Dn (L) manner is deleted, and the target directory entry dest dn (L) of the newly created target file is persisted in the Dn (L) manner.

6. The method according to claim 5, characterized in that The method further comprises: If the target file or the target directory exists in the system, and it is determined that the target directory is empty, then: Use an existing target file as a historical target file, or use an existing target directory as a historical target directory; When there is a link between the historical target file cache directory entry old CDentry corresponding to the historical target file and the historical target file cache index node old Cinode, disconnect the old CDentry and old Cinode in the cache, and persist the target directory entry dest dn(L) of the newly created target file in the manner of Dn(L) to overwrite the historical target file directory entry old Dn(I) constructed in the manner of Dn(I); When the link between the historical target file cache directory entry old CDentry corresponding to the historical target file and the historical target file cache index node old Cinode is disconnected, the Dn(L) method persists the target directory entry dest dn(L) of the newly created target file to overwrite the historical target file directory entry old Dn(L) constructed in the Dn(L) method.

7. The method according to claim 6, characterized in that Determining whether the target directory is empty includes: The master metadata server Master MDS sends an instruction to the associated MDS associated with the target directory in the system to check whether the directory slice of the local target directory is empty; the Master MDS is the MDS in the system for receiving external input write requests; the associated MDS is the MDS where the target slice of the target directory is located; When it is determined that the directory slice on any associated MDS is not empty, determining that the target directory is not empty; When it is determined that the directory slices on all the associated MDSs are empty, it is determined that the target directory is empty.

8. The method according to claim 1, characterized in that The method further comprises: When the master metadata server Master MDS receives a write request, after completing the local pre-check, after determining to execute the write request, a specified tag is added to the metadata in the local cache to prevent other write requests from occupying the metadata, and an instruction is sent to each Slave MDS to instruct each slave metadata server slave MDS to perform a local pre-check to add a specified tag to the metadata in the local cache to prevent other write requests from occupying the metadata, and after executing the first write operation corresponding to the write request locally, a reply message is sent to the Master MDS; wherein the Master MDS is the home node determined by the metadata corresponding to the write operation according to the metadata scattering principle; the Slave MDS is the node where the inode of the metadata corresponding to the write operation is stored; After the Master MDS receives the reply information from all the Slave MDSs, it executes the persistence operation corresponding to the write request.

9. The method according to claim 8, characterized in that The method further comprises: When any one of the master MDS and each slave MDS fails to execute the write request, the master MDS deletes the specified tag from the metadata in the local cache and sends an error code to each slave MDS to trigger each slave MDS to perform a rollback operation; When the Master MDS successfully executes the write request, it updates the original data in the local cache, deletes the specified tag, and sends an instruction to each Slave MDS to instruct each Slave MDS to perform the second write operation corresponding to the write request locally, and when each Slave MDS returns the write request success information, it outputs a write completion indication.

10. A writing device for a distributed file system, characterized in that: The device is applied to a network device, and the device comprises: The cache module is used for the metadata of the file to be cached. If the file is created for the first time in the file system, the cache directory entry CDentry of the metadata of the file and the cache index node CInode are linked to each other; if the file is currently in the state of executing a specified write operation, the link between the CDentry of the metadata of the file and CInode is disconnected; The first persistence module is used for constructing the key key of the directory item Dentry based on the bucket identifier bucketid corresponding to the metadata, the index node number of the parent index node of the index node Inode, and the file name name, and constructing the value value of the Dentry in a preset first-class value construction method Dn(I) for the metadata of the file to be persisted, if the CDentry and CInode in the metadata in the cache are linked to each other, so as to persist the Dentry and the Inode; wherein the Dn(I) refers to the method of constructing value based on all the information of the Inode; the bucketid of the metadata of any file is determined according to the principle of metadata scattering; The second persistence module is used to construct the key of the Dentry based on the bucketid corresponding to the metadata, the index node number of the parent index node of the Inode, and the file name name if the link between the CDentry in the metadata in the cache is disconnected, construct the value value of the Dentry based on the preset second-class value construction method Dn(L), construct the key of the Inode based on the bucketid corresponding to the metadata and the index node number of the Inode, and construct the value of the Inode based on the Dn(I) method, so as to persist the Dentry and the Inode respectively; the Dn(L) refers to the method of constructing the value of the Dentry with the key when the Inode corresponding to the Dentry is persisted.

11. The device according to claim 10, characterized in that Each CDentry is linked to the directory necklace table mdcache->dentry_lru, and each CInode that is not linked to CDentry is linked to the index node list mdcache->nlinode_lru to perform cache elimination when the set cache elimination conditions are met; And / or, the device further comprises: A hard link module is used for, when establishing a hard link, if it is determined that it is the first time to establish a hard link for a source index node src inode, disconnecting the source cache directory entry src CDentry and the source cache index node src Cinode corresponding to the source file in the cache, persisting the newly created target directory entry dest dn (L) in the manner of Dn (L), and overwriting the source directory entry src dn (I) constructed in the manner of Dn (I) with the newly created source directory entry src dn (L) in the manner of Dn (L), and updating the number of links in srcinode; If it is determined that this is not the first time to establish a hard link for the src inode, the newly created target directory entry dest dn (L) is persisted in the manner of Dn (L), and the number of links in the src inode is updated; and / or, The hard link module is further used for: When it is determined to delete any established hard link, the dest dn (L) corresponding to the hard link is deleted, and the number of links in srcinode is updated, or the src inode is deleted when the deletion condition is met; And / or, the device further comprises: The renaming module is used for, when determining to rename a source file or source directory to a target file or target directory, if the target file or target directory does not exist in the system, then: When there is a link between the source cache directory entry src CDentry and the source cache index node srcCinode corresponding to the source file or source directory, disconnect the link between the src CDentry and the src Cinode in the cache, delete the source directory entry src dn(I) constructed in the Dn(I) manner, persist the target directory entry dest dn(L) of the newly created target file in the Dn(L) manner, and persist the src inode; When the link between src CDentry and src Cinode corresponding to the source file or source directory is disconnected, the source directory entry src dn(L) constructed in the Dn(L) manner is deleted, and the target directory entry dest dn(L) of the newly created target file is persisted in the Dn(L) manner; And / or, the renaming module is further used for: If the target file or the target directory exists in the system, and it is determined that the target directory is empty, then: If the target file or the target directory exists in the system, and it is determined that the target directory is empty, then: Use an existing target file as a historical target file, or use an existing target directory as a historical target directory; When there is a link between the historical target file cache directory entry old CDentry corresponding to the historical target file and the historical target file cache index node old Cinode, disconnect the old CDentry and old Cinode in the cache, and persist the target directory entry dest dn(L) of the newly created target file in the manner of Dn(L) to overwrite the historical target file directory entry old Dn(I) constructed in the manner of Dn(I); When the link between the historical target file cache directory entry old CDentry corresponding to the historical target file and the historical target file cache index node old Cinode is disconnected, the target directory entry dest dn(L) of the newly created target file is persisted in the Dn(L) manner to overwrite the historical target file directory entry old Dn(L) constructed in the Dn(L) manner; and / or, Determining whether the target directory is empty includes: The master metadata server Master MDS sends an instruction to the associated MDS associated with the target directory in the system to check whether the directory slice of the local target directory is empty; the Master MDS is the MDS in the system for receiving external input write requests; the associated MDS is the MDS where the target slice of the target directory is located; When it is determined that the directory slice on any associated MDS is not empty, determining that the target directory is not empty; When it is determined that the directory slices on all the associated MDSs are empty, determining that the target directory is empty; The device also includes: A distributed processing module is used for, when a master metadata server Master MDS receives a write request, after completing a local pre-check, after determining to execute the write request, adding a specified tag to the metadata in the local cache to prevent other write requests from occupying the metadata, and sending an instruction to each Slave MDS to instruct each slave metadata server slaveMDS to perform a local pre-check to add a specified tag to the metadata in the local cache to prevent other write requests from occupying the metadata, and after executing the first write operation corresponding to the write request locally, sending a reply message to the Master MDS; wherein the Master MDS is the home node determined by the metadata corresponding to the write operation according to the metadata scattering principle; the Slave MDS is the node where the inode of the metadata corresponding to the write operation is stored; After the Master MDS receives the reply information from all the Slave MDSs, it executes the persistence operation corresponding to the write request; And / or, the distributed processing module is further used for: When any one of the master MDS and each slave MDS fails to execute the write request, the master MDS deletes the specified tag from the metadata in the local cache and sends an error code to each slave MDS to trigger each slave MDS to perform a rollback operation; When the Master MDS successfully executes the write request, it updates the original data in the local cache, deletes the specified tag, and sends an instruction to each Slave MDS to instruct each Slave MDS to perform the second write operation corresponding to the write request locally, and when each Slave MDS returns the write request success information, it outputs a write completion indication.

12. An electronic device, characterized in that: The electronic device includes: Processor; and A computer-readable storage medium, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, enable the processor to perform the steps of any one of the methods of claims 1 to 9.

Images