File system supporting rich metadata management and method for implementing rich metadata service thereof

Abstract

The application discloses a file system supporting rich metadata management and relates to the field of electronic digital data processing.The file system comprises a registration module, a namespace query module, a first data management module and a second data management module, and can be used as a file daemon of a FUSE framework to provide traditional file access and newly-added rich metadata management services for users.The rich metadata management service can provide metadata insertion, reading, deletion and display operations in addition to POSIX semantics, and can pack files with the same metadata attribute values or similar metadata attribute values into wrappers through automatic wrapper packing and mounting operations, and mount the files under the root directory of the file system in a virtual directory mode, so that users can quickly filter interested data from mass files by accessing the virtual directory.

Description

Technical Field

[0001] This invention belongs to the field of electronic digital data processing, and more specifically, relates to a file system for rich metadata management and a method for implementing rich metadata services. Background Technology

[0002] The rise of new computing paradigms such as Artificial Intelligence (AI) computing, graph computing, and high-performance analytics has led to a continuous increase in the scale of metadata assisting in data retrieval. Compared to traditional applications, these new applications not only need to read and write relational data but also frequently read and write non-relational data (such as documents, images, audio / video, and vectors). To address this, new applications define more diverse and higher-dimensional metadata to enrich data descriptions (referred to as rich metadata). Based on this rich metadata, they support data retrieval based on complex relationships such as ancestor, neighbor, and approximation, rather than simply using basic data metadata (filename, file size, file extension, etc.) for simple file retrieval. Existing storage systems, such as file systems, only support the management of basic data metadata and cannot meet the ever-increasing demands for metadata storage and management. Therefore, a file system that supports rich metadata management services needs to be designed, or rich metadata management services need to be implemented within the file system.

[0003] Several existing methods for implementing rich metadata management include:

[0004] (1) Modify the metadata management structure (inode or dentry)

[0005] Mature file systems such as ext4, brtfs, and xfs all provide extended attribute mechanisms (xattrs) to assist users in managing custom extended metadata. This mechanism uses key-value pairs to represent extended attributes, associating user-defined metadata with specific files or directories in the format <attribute name, attribute value>. This extended attribute mechanism requires the support of the underlying file system, and the attribute name and value have certain length limitations, resulting in insufficient versatility.

[0006] Taking the ext4 file system as an example, ext4 provides two storage methods for extended attributes: 1) directly stored in the inode structure; 2) stored in a new data block. Regardless of the storage method, extended attributes consist of an attribute header (xattr_head) and several extended attribute entries (xattr_entry), the only difference being the attribute header field. Each attribute entry contains the name of the extended attribute and an offset address pointing to the storage area of ​​the extended attribute value. To facilitate the management of extended attributes, the extended attribute value is stored at the end of the extended attribute area, and is written sequentially from the end to the beginning of the extended attribute area, following the order in which extended attributes are written. When an extended attribute value overflows, the removed portion needs to be stored in an additional data block.

[0007] (2) Design additional metadata storage structures

[0008] To address the performance limitations of existing solutions that use relational databases to manage application metadata and provide metadata search functionality, Sasha et al. designed a file system (Queriability Metadata services, or QMDS) that uses a graph model with attribute nodes and relational edges to manage metadata, and provides a query interface for file location and extended attribute retrieval. Figure 1 A diagram illustrating QMDS's use of a graph model to manage metadata is provided. Figure 1 As can be seen, to achieve graph-based metadata management, QMDS needs to use data structures such as file tables, link tables, file attribute tables, link attribute tables, attribute name tables, file attribute index red-black trees, and link attribute index red-black trees to manage metadata relationships. In short, these data structures are persisted as memory-mapped files, with each type of data structure stored in a separate memory-mapped file.

[0009] (3) Use a mature database system internally to manage metadata.

[0010] To provide richer data search services, Hyogi Sim et al. proposed a file system called TagIt with integrated indexing and search services. This scheme incorporates a relational database, SQLite, within the file system for metadata management. Figure 2 A schematic diagram of TagIt is provided. From Figure 2 As can be seen from this, in order to manage more extended metadata, TagIt uses a relational database to manage metadata, instantiating file tables, extended attribute name tables, extended attribute value tables, and building inverted indexes for them.

[0011] However, all of the above-mentioned metadata management methods have some significant drawbacks.

[0012] First, the approach of directly modifying the metadata management structure (inode or dentry) usually faces the problem of insufficient universality and does not yet support the function of retrieval based on extended metadata;

[0013] Secondly, designing additional metadata storage structures, such as graph-based metadata management solutions, can functionally support rich metadata retrieval. However, their management structure is complex and uses memory-mapped files for metadata persistence. Each time, the entire metadata structure needs to be read from disk into memory, resulting in two problems: firstly, excessive metadata read / write overhead and severe read / write amplification; secondly, extremely high memory usage, with memory overhead continuously increasing as the metadata size grows.

[0014] Third, using a mature internal database system to manage metadata offers the most comprehensive functionality, but also incurs the highest overhead. This is because database systems are typically built on top of file systems, leading to significant read / write amplification. Furthermore, database systems are more complex than file systems, have higher runtime requirements, and can suffer from severe performance degradation due to insufficient functional integration. Essentially, while placing the database system within the file system reduces the path from filename to metadata inode retrieval, it lengthens the path for metadata persistence and retrieval. Additionally, using an external database presents metadata inconsistency issues. Summary of the Invention

[0015] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a file system for rich metadata management and a method for implementing rich metadata services. Its purpose is to solve the technical problems of existing methods that directly modify the metadata management structure, which typically lack versatility and do not support retrieval based on extended metadata; the technical problems of excessive metadata read / write overhead and severe read / write amplification in existing designs of additional metadata storage structures; the technical problems of extremely high memory usage, with memory overhead continuously increasing as the metadata scale continues to grow; and the technical problems of using mature database systems internally to manage metadata, which prolongs the path of metadata persistence and querying, and metadata inconsistency.

[0016] To achieve the above objectives, according to one aspect of the present invention, a file system supporting metadata management is provided, including a registration module, a namespace query module, a first data management module, and a second data management module. The registration module is used to receive input / output (IO) requests sent by users, obtain the access path corresponding to the IO request, query the corresponding wrapper tag in a pre-established registry according to the IO request, and send the access path and wrapper tag to the namespace query module.

[0017] The namespace query module is used to parse the access path and generate a wrapper key using the parsed current directory and wrapper tag. It determines whether the wrapper key can be used to query the first, second, third, and fourth mapping tables in memory to obtain the wrapper and rich index node data corresponding to the current directory. If so, the node ID of the rich index node is obtained in memory and sent to the first data management module. Otherwise, the wrapper tag and the access path are sent to the second data management module, which loads the corresponding data from the disk.

[0018] The first data management module is used to read and write corresponding file data in the local file system based on the node ID of the rich index node.

[0019] The second data management module is used to retrieve the corresponding wrapper data and rich index node data from the disk based on the keys of the four mapping tables provided by the namespace query module, and to construct the wrapper and rich index node in memory based on the retrieved wrapper data and rich index node data to populate the corresponding mapping table.

[0020] The namespace query module is also used to send the updated wrapper data and the rich index node data to the second data management module.

[0021] The second data management module is also used to write the updated wrapper data and rich index node data back to disk in key-value pairs using SPDK technology.

[0022] According to another aspect of the present invention, a method for implementing rich metadata services using the above-described file system that supports rich metadata management is provided, comprising the following steps:

[0023] (1) Obtain the IO request from the user and determine whether the IO request is an insert operation request, a read operation request, a delete operation request, or a display operation request. If it is an insert operation request, proceed to step (2); if it is a read operation request, proceed to step (3); if it is a delete operation request, proceed to step (4); if it is a display operation request, proceed to step (5).

[0024] (2) Extract the access path, extended metadata name and extended metadata value from the IO request, determine whether the extended metadata name and extended metadata value can be inserted into the rich index node of the file where the access path is located, if successful, return a notification that the metadata insertion was successful and then the process ends; otherwise, return a notification that the specified access path does not exist and then the process ends.

[0025] (3) Extract the access path and extended metadata name from the IO request, determine whether the corresponding metadata value can be read from the rich index node of the file where the access path is located according to the extended metadata name, if the reading is successful, return the metadata value and then the process ends; otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends.

[0026] (4) Extract the access path and extended metadata name from the IO request, determine whether the corresponding metadata value can be deleted from the rich index node of the file where the access path is located according to the extended metadata name, if the reading is successful, return the metadata value and then the process ends; otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends.

[0027] (5) Extract the access path from the IO request, determine whether all extended metadata names and extended metadata values ​​can be read from the rich index node of the file where the access path is located. If the reading is successful, return the corresponding extended metadata names and extended metadata values ​​as a list and then the process ends. Otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends.

[0028] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0029] (1) Due to the design of rich index nodes and wrapper structure, this invention can expand the management of more types and sizes of metadata compared with the traditional metadata management structure (inode, dentry). Furthermore, based on the metadata that the application needs to retrieve, data with the same retrieval behavior can be packaged to accelerate the filtering of data of interest from massive amounts of data. Therefore, it can solve the technical problem that existing schemes that directly modify the metadata management structure usually face the problem of insufficient universality and do not yet support the function of retrieval based on extended metadata.

[0030] (2) This invention uses a namespace query module to replace the traditional hierarchical namespace. On the one hand, it uses different encoding rules to encode the metadata involved in different retrieval applications, reducing the namespace search space that needs to be retrieved. On the other hand, it uses key-value pairs to load the required metadata from the disk instead of loading the entire metadata file, which solves the problems of metadata read / write amplification and memory occupation. Therefore, it can solve the technical problems of excessive read / write overhead, severe read / write amplification, and extremely high memory occupation of the scheme that designs additional metadata storage structures for metadata. As the size of metadata continues to increase, the memory overhead also continues to increase.

[0031] (3) This invention adopts the design of a second data management module, which directly drives SPDK technology to write metadata IO from user space to the local metadata SSD disk, thus reducing the metadata IO path; on the other hand, it implements metadata partition layout on SSD, further optimizes the single-point query and range query performance of metadata by using a hybrid index mechanism, and ensures the consistency of metadata by using a log mechanism. Therefore, it can solve the technical problems of using a mature database system to manage metadata internally, which prolongs the path of metadata persistence and query, as well as the inconsistency of metadata. Attached Figure Description

[0032] Figure 1 This is a diagram illustrating how QMDS uses a graph model to manage metadata;

[0033] Figure 2 This is a diagram illustrating the metadata management of the TagIt file system;

[0034] Figure 3 This is a general framework diagram of the file system that supports rich metadata management according to the present invention;

[0035] Figure 4 This is a schematic diagram of the rich metadata index node provided by the present invention;

[0036] Figure 5 This is a schematic diagram of the working mode of the rich element index node provided by the present invention;

[0037] Figure 6 This is a schematic diagram of the packaging structure provided by the present invention;

[0038] Figure 7 This is a schematic diagram of the second data management module provided by the present invention;

[0039] Figure 8 This is a flowchart of a method for implementing rich metadata services using a file system that supports rich metadata management, as per the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0041] The technical terms used in this invention will be explained and described below:

[0042] Filesystem in Userspace (FUSE): FUSE is a foundational framework for developing file systems in user space. It consists of three parts: the FUSE file-system daemon, the FUSE library, and the FUSE driver. Users develop their own file systems based on the FUSE file-system daemon, implementing data and metadata management functions. They then use the FUSE library to mount the file system to the ` / dev / fuse` directory, allowing users to access the file system through this mount point. File access requests to the FUSE file system first go through the file system kernel interface exposed by the Virtual File System (VFS). The VFS then determines whether the request is a FUSE file request based on the mount type. If it is, the request is forwarded to the FUSE driver, which then forwards the request to the FUSE library and the corresponding FUSE file-system daemon. The FUSE daemon receives the request, processes it, and returns a response, which is then sent back to the user.

[0043] Access path: When a user accesses the file system, they provide the file system with the path to the file or directory to be accessed. For example, "D: / Invention / Invention Specification.word".

[0044] FUSE Operation Types: The FUSE framework automatically calls the corresponding response module in the FUSE file system daemon to handle different operation requests from users based on the FUSE operation type. Common FUSE operation types include FUSE_OPEN, FUSE_READ, FUSE_WRITE, FUSE_OPENDIR, and FUSE_READDIR. Based on their function, FUSE operations can be further divided into operations involving file data, such as FUSE_READ and FUSE_WRITE, which read and write file data respectively; and operations involving file or directory metadata, such as FUSE_OPEN and FUSE_OPENDIR.

[0045] Solid State Drive (SSD): A type of hard drive that uses flash memory as its storage medium. NVMe SSDs specifically refer to SSDs that allow data to be read using the NVMe interface protocol.

[0046] Non-volatile memory host controller interface protocol (NVM Express, or NVMe for short): a hard disk access protocol.

[0047] SPDK (Storage Performance Development Kit): In response to the emergence of NVM-SSDs, Intel developed a development kit based on NVM-SSDs (Storage Performance Development Kit, SPDK). The goal of SPDK is to maximize the performance of solid-state storage media by utilizing Intel's networking, processing, and storage technologies. Compared to traditional I / O methods, SPDK employs two key technologies: UIO (User Space I / O) and pooling.

[0048] Rich Index Node (rnode): Traditional file systems use index nodes (inodes) to store metadata information for files and directories, such as file size, owner, creation time, disk location, and file operation methods. This invention extends the concept of inodes by introducing rich index nodes, which are used to store more diverse types of metadata.

[0049] Wrapper: Traditional file systems use directory entries (dentries) to store a mapping table from directory IDs to file IDs, maintaining the directory relationships of files. The file system designed in this invention needs to maintain not only the directory relationships of files, but also, based on user-provided custom metadata (such as hash values, feature vectors), maintain hash approximation or vector approximation relationships for files. Therefore, a wrapper structure is designed to replace the dentry structure, storing more types of relationship mapping tables.

[0050] Wrapper Tag: Each type of wrapper is bound to a unique wrapper tag that indicates the metadata used by the wrapper query and the access path through which the query takes effect.

[0051] SSD partitioning storage technology refers to dividing an entire SSD into several regions and storing data with similar characteristics (hot / cold, function, etc.) in specific regions. Typical partitioning technologies include hot / cold storage separation and data metadata separation. This invention designs its own partitioning storage technology within the rich metadata management module.

[0052] Hybrid Key-Value Indexes: Common key-value indexes include hash indexes, B+ tree indexes, and LSM-Tree indexes. Hybrid key-value indexes refer to key-value index structures that use two or more key-value indexes simultaneously. This invention designs its own hybrid indexing technology in the rich metadata management module. The purpose of this invention is to provide a file system that supports rich metadata management, which can act as a file daemon within the FUSE framework, providing users with traditional file access and new rich metadata management services through the FUSE framework. The rich metadata management service provides insertion, deletion, update, and query operations for metadata other than POSIX semantics; furthermore, through automatic wrapper packaging and mounting operations, it packages files with the same or similar metadata attribute values ​​into wrappers and mounts them as virtual directories in the file system root directory. Subsequently, users can quickly filter data of interest from a massive number of files by accessing the virtual directory.

[0053] The basic idea of ​​this invention is as follows: First, the access path is obtained from the IO request and submitted to the registration module to query the registry, thereby obtaining the wrapper tag used for retrieval. Then, the access path and wrapper tag are submitted to the namespace query module to parse the access path, query the wrapper and rich index node, and obtain the node ID of the rich index node to be queried. During the parsing of the access path, if the wrapper or rich index node to be queried is not yet in memory, the second data management module is called to use SPDK technology to read the data of the wrapper and rich index node from the disk in key-value pairs, and build the rich index node and wrapper in memory. Then, the namespace query module will further select, according to the FUSE operation type, to submit the node ID of the rich index node to the first data management module and delegate the file data read and write operations to the local file system, or select to directly read and write the data of the wrapper or rich index node in memory. Finally, the namespace query module periodically submits the updated wrapper and rich index node data to the second data management module in key-value pairs, and the second data management module uses SPDK technology to write it back to the disk. In particular, the second data management module uses partitioned storage technology to manage data on the metadata SSD and provides hybrid key-value indexes to accelerate the reading and writing of wrapper and rich index node data.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] like Figure 3 As shown, this invention provides a file system that supports rich metadata management. It resides in user space, needs to be stacked on top of existing mature file systems, and requires additional metadata storage devices to store the metadata. The system includes a registration module, a namespace query module, a first data management module, and a second data management module.

[0056] The registration module is used to receive input / output (IO) requests sent by users, obtain the access path corresponding to the IO request, query the corresponding wrapper tag in the pre-established registry (as shown in Table 1 below) according to the IO request, and send the access path and wrapper tag to the namespace query module.

[0057]

[0058] Table 1

[0059] Specifically, entries in the registry exist as key-value pairs, where the key is the access path and the value is the wrapper tag, which is used to establish associations with different wrappers.

[0060] like Figure 4 As shown, the rich index node retains the basic metadata of the regular file, but additionally includes a data extension area to store extended attributes and file data indexes. The schema field in the header of the data extension area specifies the data content and storage method, while the length field specifies the actual valid data length in the extended data area.

[0061] Specifically, the data expansion area provides the following storage modes, such as... Figure 5 As shown.

[0062] 1) Key-value pair-based extended attribute storage (0001): In this mode, users add extended metadata to rich index nodes based on key-value pairs, i.e., <attribute name, attribute value>. To accelerate extended attribute retrieval, the storage order of attribute names is conventionally from the head to the tail of the extended data, and the storage order of attribute values ​​is conventionally from the tail to the head. The effective length of the extended data area is the length of the set of effective attribute names.

[0063] 2) Key-value separation storage of extended attributes (0010): In this mode, users also add extended metadata to rich index nodes based on key-value pairs. However, considering the length of attribute values, the attribute values ​​are not directly stored in the extended attribute area. Instead, an additional data block is used to store the original values ​​of the extended attributes, and the address information of the extended attribute values ​​is recorded. Since the attribute values ​​are indexed using pointers, the extended attribute name and the extended attribute pointer should be stored contiguously in the extended data area, while the extended attribute values ​​can be stored arbitrarily in the additional data block.

[0064] 3) Inline storage of file content (0100): In this mode, the user stores the file content directly inline within the rich inode. However, considering the limited size of the extended data area, this mode is only recommended when the file size is less than 4KB. In this case, the effective length of the extended data area is the same as the effective length of the file content.

[0065] 4) Indexed File Content Storage (1000): In this mode, a file index area is reserved in the extended data area to store the actual offset address of the file data within the data block. The effective length of the extended data area is the effective length of the file content. This mode is suitable for storing large files that do not require storage in blocks according to attributes.

[0066] 5) Inline Hybrid Storage (0110): In this mode, the first part of the extended data area stores extended metadata, while the second part stores the actual file content. The effective length of the extended data area is the length of the extended attribute list. This mode is suitable for storing small files with limited annotation information.

[0067] 6) Indexed Hybrid Storage (1010): In this mode, the first part of the extended data area stores extended metadata, and the second part stores the file's data content address. The effective length of the extended data area is the length of the extended attribute list. This mode is suitable for storing large files with limited annotation information.

[0068] 7) Key-value splitting block storage (1110): In this mode, file content is no longer stored as a continuous block, but is divided into blocks according to different extended attributes. Different file blocks are labeled and indexed using different extended attributes. Therefore, after storing the extended attributes, the block data is stored immediately afterwards.

[0069] 8) Packed Block Storage (1111): This mode works by conveniently storing data in blocks according to extended attributes. However, unlike key-value block storage, this mode enables an anonymous wrapping mechanism, using a metadata wrapper to package extended attributes and data blocks together and store them using an additional data structure, storing the corresponding wrapper ID only in the extended data area.

[0070] As can be seen from the design of the rich index node above, compared with the traditional inode structure, the rich index node can manage more metadata and can use different storage modes to optimize the storage of rich metadata.

[0071] like Figure 6As shown, a wrapper consists of three parts: wrapper metadata, metadata, attribute positioning metadata, and relation index metadata. Wrapper metadata records the common basic metadata of files belonging to that wrapper, such as wrapper ID, group access permissions, access time, and group size. It also specifically records wrapper tags to identify which attributes the metadata wrapper uses as positioning attributes. Attribute positioning metadata records which rich index node IDs are included in the wrapper. Relationship index metadata records the relationships between wrappers.

[0072] As can be seen from the design of the wrapper above, compared with the traditional dentry structure, the wrapper can not only save the file directory relationship, but also package data with the same search behavior according to the metadata that the application needs to retrieve, and provide other relationship queries besides directory relationship, thereby accelerating the filtering of data of interest from massive data.

[0073] When the wrapper is loaded from disk into memory, three different mapping tables are generated in memory (as shown in Table 1). The entries in the first mapping table exist as key-value pairs. The key is a string concatenated from the wrapper tag and the wrapper's node ID, referred to as the wrapper key. The value is the wrapper metadata, including the wrapper tag, main attribute name, main attribute value, group access permissions, and metrics. Specifically, metrics refer to query conditions, and group access permissions include read, write, and execute permissions.

[0074] The entries in the second mapping table exist in the form of key-value pairs, where the key is a string composed of the wrapper label, the wrapper node ID, and the metric, referred to as the relation index key, and the value is the node ID of the wrapper associated with it.

[0075] The entries in the third mapping table exist in the form of key-value pairs, where the key is a string composed of the wrapper tag, the wrapper node ID, and the next-level path name of the current directory, which is simply called the attribute locator key, and the value is the node ID of the rich index node.

[0076] When a rich inode is loaded from disk into memory, a mapping table, referred to as the fourth mapping table, is generated in memory (as shown in Table 1). The entries in the fourth mapping table exist in key-value pairs, where the key is the node ID of the rich metadata inode, and the value is file metadata, including Portable Operating System Interface (POSIX) metadata, Extendable Attribute (EA) metadata, and small file data (where data smaller than 4MB is called a small file).

[0077] The namespace query module is used to parse the access path and generate a wrapper key using the parsed current directory and wrapper tag. It determines whether the wrapper key can be used to query the first, second, third, and fourth mapping tables in memory to obtain the wrapper and rich index node data corresponding to the current directory. If so, the node ID of the rich index node is obtained in memory and sent to the first data management module. Otherwise, the wrapper tag and the access path are sent to the second data management module, which loads the corresponding data from the disk.

[0078] More specifically, the namespace query module parses the access path and generates a wrapper key using the parsed current directory and wrapper tag. It then determines whether the wrapper key can be used to sequentially query the first, second, third, and fourth mapping tables in memory to obtain the wrapper and rich index node data corresponding to the current directory. If so, it further obtains the node ID of the rich index node in memory and sends it to the first data management module. Otherwise, it sends the wrapper tag and the access path to the second data management module, which then loads the corresponding data from the disk. This process includes the following sub-steps:

[0079] (1-1) Parse the access path to extract the root directory " / " as the current directory, and obtain the node ID of the wrapper corresponding to the current directory. The node ID is initialized to 0 by default.

[0080] (1-2) Concatenate the wrapper tag corresponding to the access path and the node ID of the wrapper obtained in step (1-1) to generate the wrapper key ID, and determine whether the corresponding wrapper metadata can be obtained in the pre-established first mapping table based on the wrapper key ID. If it can, proceed to step (1-3); otherwise, proceed to step (1-4).

[0081] (1-3) Evaluate whether the user has permission to access the wrapper corresponding to the wrapper key ID based on the obtained metadata. If yes, proceed to step (1-5); otherwise, return a notification that the user does not have sufficient permissions and the process ends.

[0082] (1-4) Submit the wrapper key ID to the second data management module and determine whether the second data management module can load the metadata corresponding to the wrapper key ID from the disk into memory. If it can, put the wrapper key ID as the key and the metadata as the value into the first mapping table, and then return to step (1-2). Otherwise, it means that the access path does not exist, return the access path does not exist notification, and the process ends.

[0083] (1-5) Further parse the access path to extract the next level path of the current directory as the current directory, and determine whether the current directory is the last level path. If it is, proceed to step (1-8); otherwise, proceed to step (1-6).

[0084] (1-6) Take the next level path of the current directory as the metric value, concatenate the wrapper tag, the node ID of the wrapper corresponding to the current directory, and the metric value string to generate a relation index key, and determine whether the node ID of the wrapper corresponding to the next level path can be read in the pre-established second mapping table according to the relation index key. If yes, return to step (1-3); otherwise, proceed to step (1-7).

[0085] (1-7) Submit the relation index key to the second data management module and determine whether the second data management module can load the metadata corresponding to the relation index key from the disk into memory. If it can, put the relation index key as the key and the metadata as the value into the second mapping table, and then return to step (1-6). Otherwise, it means that the access path does not exist, and return an access path non-existent notification, and the process ends.

[0086] (1-8) Encode the wrapper tag and the node ID of the wrapper corresponding to the current directory and the next level path of the current directory into strings to generate an attribute locator key, and determine whether the node ID of the rich index node corresponding to the attribute locator key can be obtained in the pre-established third mapping table. If it can be obtained, proceed to step (1-9); otherwise, proceed to step (1-11).

[0087] (1-9) Determine whether the corresponding file metadata can be obtained from the pre-established fourth mapping table using the node ID of the rich index node as the key. If yes, the namespace query process is completed and the process ends; otherwise, proceed to step (1-10).

[0088] (1-10) Send the node ID of the rich index node to the second data management module and determine whether the corresponding file metadata can be obtained from the disk. If so, put the node ID of the rich index node as the key and the file metadata as the value into the fourth mapping table and return to step (1-9). Otherwise, the user queries a directory instead of a file and proceed to step (1-11).

[0089] (1-11) Take the next level path of the current directory as the metric value, concatenate the wrapper tag, the node ID of the wrapper corresponding to the current directory, and the metric value string to generate a relation index key, and determine whether the node ID of the wrapper corresponding to the next level path can be read in the pre-established second mapping table according to the relation index key. If yes, proceed to step (1-12); otherwise, proceed to step (1-14).

[0090] (1-12) Concatenate the wrapper tag corresponding to the access path and the node ID of the wrapper corresponding to the next level path obtained in step (1-11) to generate a wrapper key, and determine whether the corresponding wrapper metadata can be obtained in the pre-established first mapping table based on the wrapper key. If it can, the namespace query process is completed and the process ends; otherwise, proceed to step (1-13).

[0091] (1-13) Submit the wrapper key obtained in step (1-12) to the second data management module, and determine whether the second data management module can load the metadata corresponding to the wrapper key from the disk into memory. If it can, put the wrapper key as the key and the metadata as the value into the first mapping table, and then return to step (1-12). Otherwise, it means that the access path does not exist, and return the access path does not exist notification, and the process ends.

[0092] (1-14) Submit the relationship index key obtained in step (1-11) to the second data management module, and determine whether the second data management module can load the metadata corresponding to the relationship index key from the disk into memory. If it can, put the relationship index key as the key and the metadata as the value into the second mapping table, and then return to step (1-11). Otherwise, it means that the access path does not exist, and return an access path non-existent notification, and the process ends.

[0093] As can be seen from the process of the namespace query module, this invention pre-establishes four mapping tables in memory to store various metadata required for the query process, and reads and modifies the mapping tables in a key-value pair manner. Only when the query fails does it need to load the searched key-value pairs from disk, rather than reloading the entire mapping table from disk. This table partitioning design reduces the size of the namespaces that need to be searched, while also avoiding the problem of increased namespace query overhead caused by the increase in data and metadata scale.

[0094] The first data management module is used to read and write corresponding file data in the local file system based on the node ID of the rich index node.

[0095] The second data management module is used to retrieve the corresponding wrapper data and rich index node data from the disk (e.g., SSD) based on the keys of the four mapping tables provided by the namespace query module, and to construct the wrapper and rich index node in memory based on the retrieved wrapper data and rich index node data to populate the corresponding mapping table.

[0096] like Figure 7As shown, the disk is configured with a first storage area, a second storage area, a third storage area, and a fourth storage area. The first storage area stores rich index node metadata and wrapper metadata of wrappers. The second storage area stores extended attributes and small files. The third storage area stores wrapper attribute positioning metadata and relation index metadata. The fourth storage area stores logs.

[0097] Specifically, the second data management module retrieves the corresponding wrapper data and rich index node data from the disk based on the keys of the four mapping tables provided by the namespace query module, and constructs the wrapper and rich index node in memory based on the retrieved wrapper data and rich index node data to populate the corresponding mapping table. This process includes the following sub-steps:

[0098] (2-1) Determine the type of the obtained key. If the key is the attribute location key of the wrapper and the relationship index key of the wrapper, proceed to step (2-2) and use the B+ tree index to read data from the third storage area of ​​the disk. If the key is the wrapper key and the rich index node key, proceed to step (2-3) and use the hash index to read data from the first or second storage area of ​​the disk.

[0099] (2-2) Take the key as input, query the B+ tree index, and determine whether the data can be read. If yes, return the read data; otherwise, return failure.

[0100] This step specifically includes the following sub-steps:

[0101] (2-2-1) Use the root node ID of the B+ tree index as the page table ID, and use SPDK technology to read the corresponding data page table from the metadata SSD;

[0102] (2-2-2) Convert the data page table obtained in step (2-2-1) into B+ tree index nodes;

[0103] (2-2-3) Determine whether the B+ tree index node in step (2-2-2) is an intermediate node of the B+ tree index. If so, proceed to step (2-2-4); otherwise, proceed to step (2-2-8).

[0104] (2-2-4) Convert the B+ tree index node in step (2-2-2) into a B+ tree index intermediate node;

[0105] (2-2-5) Determine whether the input key in (2-2) is within the key value range of the middle node in step (2-2-4). If so, jump to step (2-2-8); otherwise, proceed to step (2-2-6).

[0106] (2-2-6) Determine whether the input key in (2-2) is less than the key of the middle section of the B+ tree index obtained in step (2-2-4). If it is less, obtain the node ID of the left node of the middle node and then proceed to step (2-2-7). Otherwise, obtain the node ID of the right node of the middle node and then proceed to step (2-2-7).

[0107] (2-2-7) Use the node ID in step (2-2-6) as the page table ID, use SPDK technology to read the corresponding data page table from the metadata SSD, and return to step (2-2-3);

[0108] (2-2-8) Convert the B+ tree index node into a B+ tree index leaf node;

[0109] (2-2-9) Determine whether there is a key-value pair in the data page table of the leaf node of the B+ tree index obtained in step (2-2-8) that matches the key obtained in (2-2). If it exists, return the data corresponding to the key and the process ends; otherwise, return that the search failed and the process ends.

[0110] (2-3) Take the key as input, query the hash index, and determine whether the data can be read. If it can, return a read success notification; otherwise, return a failure notification.

[0111] This step specifically includes the following sub-steps:

[0112] (2-3-1) Use the hash algorithm murmur3 to obtain the hash value of the key input in step (2-1);

[0113] (2-3-2) Convert the hash value obtained in step (2-3-1) into a global index value;

[0114] (2-3-3) Calculate and generate a local index value using the key from step (2-3-1) and the global index value from step (2-3-2);

[0115] (2-3-4) Read the hash bucket based on the local index value obtained in step (2-3-3), and read the page table ID from the hash bucket;

[0116] (2-3-5) Based on the page table ID read in (2-3-4), the corresponding data page table is read from the metadata SSD using SPDK technology;

[0117] (2-3-6) Use the key entered in step (2-3) to find the corresponding key-value pair in the data page table obtained in step (2-3-5), and determine whether the data corresponding to the key can be read. If it can, it means that the key-value pair query is successful, and the data corresponding to the key is returned and the process ends. Otherwise, it means that the search failed, and a read failure notification is returned and the process ends.

[0118] The advantages of steps (2-1) to (2-3) above are that the second data management module optimizes disk read / write using partitioning technology. By dividing the disk into a first storage area, a second storage area, a third storage area, and a fourth storage area to store metadata with different functions, this approach reduces the search space for metadata on the disk. Simultaneously, by designing a hybrid persistent index mechanism, the performance of single-point read / write of metadata in the first and second storage areas is improved by using a hash index, which has better single-point query performance, and the performance of range read / write of metadata in the third storage area is improved by using a B+ tree index, which has better range query performance. Furthermore, the second data management module, developed using the latest SPDK technology, has a much smaller IO software stack depth than the traditional file system IO software stack, enabling data to be transferred to the disk faster and reducing the overhead of data transfer in the IO stack.

[0119] The namespace query module is also used to send the updated wrapper data and the rich index node data to the second data management module.

[0120] The second data management module is also used to write updated wrapper data and rich index node data back to disk in key-value pairs using SPDK technology. The wrapper data is written back in three key-value pairs: <wrapper key, wrapper metadata>, <relationship index key, wrapper node ID>, and <attribute locator key, rich index node ID>. The rich index node data is written back in key-value pairs: <rich index node ID, file metadata>.

[0121] Specifically, the second data management module uses SPDK technology to write the updated wrapper data and rich index node data back to disk. This process includes the following sub-steps:

[0122] (3-1) Obtain the key and data that need to be updated, determine the type of the obtained key, if the key is the attribute positioning key of the wrapper and the relationship index key of the wrapper, then proceed to step (3-2), and use the B+ tree index to write the corresponding data to the third storage area; if the key is the wrapper key and the rich index node key, then proceed to step (3-3), and use the hash index to write the corresponding data to the first or second storage area of ​​the disk.

[0123] (3-2) Using the key and data input in step (3-1) as input, query the B+ tree index, insert the key and the data corresponding to the updated table entry into the B+ tree index, and use SPDK technology to write the data page table of the updated B+ tree index to disk;

[0124] This step specifically includes the following sub-steps:

[0125] (3-2-1) Determine if the root node of the B+ tree index exists. If it exists, proceed to step (3-2-3); otherwise, proceed to step (3-2-2).

[0126] (3-2-2) Request a new page table ID, initialize the B+ tree index root node using the new page table ID, and return to step (3-2-1);

[0127] (3-2-3) Read the root node of the B+ tree index, obtain its page table ID from the root node, and load the corresponding data page from the disk based on the page table ID and using SPDK technology;

[0128] (3-2-4) Convert the data page table obtained in step (3-2-3) into B+ tree index nodes;

[0129] (3-2-5) Determine whether the B+ tree index node in step (3-2-4) is an intermediate node. If it is, proceed to (3-2-6); otherwise, proceed to (3-2-10).

[0130] (3-2-6) Convert the B+ tree index node in step (3-2-5) into an intermediate node;

[0131] (3-2-7) Determine whether the key input in (3-1) is within the key value range of the middle node in step (3-2-6). If it is, jump to step (3-2-10); otherwise, proceed to step (2-2-8).

[0132] (3-2-8) Determine whether the input key in (3-1) is less than the key of the middle node in step (3-2-6). If it is less, obtain the node ID of the left node of the middle node; otherwise, obtain the node ID of the right node of the middle node.

[0133] (3-2-9) Use the node ID in step (3-2-8) as the page table ID, use SPDK technology to read the corresponding data page table from the metadata SSD, and return to step (3-2-3);

[0134] (3-2-10) Convert the B+ tree index node in step (3-2-5) or step (3-2-7) into a B+ tree index leaf node;

[0135] (3-2-11) Compare the key in step (3-1) with the data page table in the leaf node of the B+ tree index in step (3-2-10) to determine whether there is a key-value pair that matches the key in the data page table. If there is, use the data input in step (3-1) to overwrite the data in the key-value pair that matches the key, return a notification that the data insertion was successful, and the process ends. Otherwise, proceed to step (3-2-12).

[0136] (3-2-12) Determine if there is any remaining space in the leaf table. If so, insert the key and value entered in step (3-1) into the leaf table and return that the data insertion was successful. Otherwise, proceed to step (3-2-13).

[0137] (3-2-13) Perform paging operation on the B+ index node where the leaf table in step (3-2-12) is located, create left and right nodes respectively, apply for left and right data page tables, and then write the page table data to the left and right data page tables respectively according to the key value range of the left and right nodes, while releasing the page table space of the B+ index node.

[0138] (3-2-14) Determine whether the key in step (3-1) is within the key value range of the left data page table in step (3-2-13). If so, write the key and value in step (3-1) to the left subpage and update the left node index information in step (3-2-13). Otherwise, write the key and value in step (3-1) to the right data page table and update the right node index information in step (3-2-13).

[0139] (3-2-15) Write the data of the left page table back to the disk according to the ID of the left page table and using SPDK technology. At the same time, write the data of the right page table back to the disk according to the ID of the right page table and using SPDK technology. Return that the data insertion was successful.

[0140] (3-3) Using the key and data input in step (3-1) as input, query the hash index, insert the key and data into the hash index, and use SPDK technology to write the updated hash bucket to disk;

[0141] This step specifically includes the following sub-steps:

[0142] (3-3-1) Use the hash algorithm murmur3 to obtain the hash value of the key input in step (3-1);

[0143] (3-3-2) Convert the hash value obtained in step (3-3-1) into a global index value;

[0144] (3-3-3) Calculate the local index value by combining the input key and the global index value extracted in step (3-3-2);

[0145] (3-3-4) Use the local hash value from step (3-3-3) to read the hash bucket and read the leaf table ID from the hash bucket;

[0146] (3-3-5) Based on the leaf table ID extracted in step (3-3-4), the corresponding data page is read from the disk using SPDK technology;

[0147] (3-3-6) Convert the data pages extracted in step (3-3-5) into hash buckets;

[0148] (3-3-7) Determine if there is any remaining space in the hash bucket. If there is, write the input key and value into the hash bucket and proceed to step (3-3-9); otherwise, proceed to step (3-3-8).

[0149] (3-3-8) Request a new page table, obtain the new page table ID, and return to step (3-3-5);

[0150] (3-3-9) Update the local index value extracted in step (3-3-3) and the global index value extracted in step (3-3-2);

[0151] (3-3-10) Use SPDK technology to convert the updated hash bucket into a page table and write it back to disk, and return a notification that the data insertion was successful;

[0152] like Figure 8 As shown, the present invention also provides a method for implementing rich metadata services using the file system that supports rich metadata management as described above, comprising the following steps:

[0153] (1) Obtain the IO request from the user and determine whether the IO request is an insert operation request, a read operation request, a delete operation request, or a display operation request. If it is an insert operation request, proceed to step (2); if it is a read operation request, proceed to step (3); if it is a delete operation request, proceed to step (4); if it is a display operation request, proceed to step (5).

[0154] (2) Extract the access path, extended metadata name and extended metadata value from the IO request, determine whether the extended metadata name and extended metadata value can be inserted into the rich index node of the file where the access path is located, if successful, return a notification that the metadata insertion was successful and then the process ends; otherwise, return a notification that the specified access path does not exist and then the process ends.

[0155] This step specifically includes the following sub-steps:

[0156] (2-1) Extract the access path, extended metadata name, and extended metadata value from the IO request;

[0157] (2-2) Submit the access path obtained in step (2-1) to the registration module to obtain the corresponding wrapper tag;

[0158] (2-3) Submit the wrapper tag obtained in step (2-2) and the access path extracted in step (2-1) to the namespace query module. The namespace query module recursively parses the access path to obtain the node ID of the rich index node;

[0159] (2-4) Submit the node ID of the rich index node extracted in step (2-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node.

[0160] (2-5) Based on the selected storage mode, write the metadata name and extended metadata value extracted in step (2-1) to the data extension area of ​​the rich index node.

[0161] The advantage of this step (2-5) is that it designs a dedicated rich index node, which can use different storage modes to store user-provided metadata, enabling the file system to store more types of metadata, such as content hash codes, feature vectors, etc.

[0162] (2-6) The second data management module writes the updated data of the rich index nodes back to the disk in real time;

[0163] (2-7) Determine whether step (2-6) was executed successfully. If it was executed successfully, return that the rich metadata insertion was successful and proceed to step (2-8) asynchronously. Otherwise, return that the rich metadata lookup failed.

[0164] (2-8) Perform the packing operation using the packer;

[0165] This step specifically includes the following sub-steps:

[0166] (2-8-1) Submit the extended metadata name and extended metadata value extracted in step (2-1) to the namespace query module. The namespace query module will trigger the extended attribute query operation. If the query is successful, obtain the node ID of the wrapper whose main attribute is the extended attribute value and proceed to step (2-8-2); otherwise, proceed to step (2-8-3).

[0167] (2-8-2) Generate an attribute positioning key by taking the wrapper ID extracted in step (2-8-1), the extended attribute value extracted in step (2-1), and the wrapper tag encoding extracted in step (2-2). Query the third mapping table to determine whether the wrapper positioning attribute can be obtained. If it can be obtained, proceed to step (2-8-3); otherwise, proceed to step (2-8-4).

[0168] (2-8-3) Insert the node ID of the rich index node obtained in step (2-3) and the extended attribute name obtained in step (2-1) into the wrapper positioning attribute obtained in step (2-8-2), and proceed to step (2-8-6).

[0169] (2-8-4) The naming query module creates a new wrapper, assigns a node ID to the new wrapper, generates group metadata, relation index metadata and attribute location metadata for the wrapper, and inserts the generated wrapper data into the first mapping table, the second mapping table and the third mapping table in key-value pairs in sequence.

[0170] (2-8-5) Insert the node ID of the rich index node obtained in step (2-3) and the extended attribute name obtained in step (2-1) into the newly created wrapper positioning attribute in step (2-8-2), and proceed to step (2-8-6).

[0171] (2-8-6) The namespace query module periodically triggers the wrapper update operation, which hands over the key-value pairs in the first, second, and third mapping tables that are in the update state to the second metadata management module, which then writes the key-value pairs that need to be updated back to the disk.

[0172] The core innovation of this invention is the wrapper packaging operation in step (2-8). The idea is to use a wrapper to encapsulate the node IDs of rich index nodes with the same extended attribute name and similar extended attribute values ​​into the attribute definition metadata of the wrapper. Its advantage is that the wrapper collects data that users are interested in.

[0173] (3) Extract the access path and extended metadata name from the IO request, determine whether the corresponding metadata value can be read from the rich index node of the file where the access path is located according to the extended metadata name, if the reading is successful, return the metadata value and then the process ends; otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends.

[0174] This step includes the following sub-steps:

[0175] (3-1) Extract the access path and extend the metadata name from the IO request;

[0176] (3-2) Submit the access path obtained in step (3-1) to the registration module to obtain the corresponding wrapper tag;

[0177] (3-3) Submit the wrapper tag obtained in step (3-2) and the access path extracted in step (3-1) to the namespace query module. The namespace query module recursively parses the access path to obtain the node ID of the rich index node;

[0178] (3-4) Submit the node ID of the rich index node extracted in step (3-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node.

[0179] (3-5) From the data extension area of ​​the rich index node extracted in step (3-3), determine whether the extension metadata value can be read according to the extension metadata name extracted in step (3-1). If the reading is successful, return the read extension metadata value and proceed to step (3-6) asynchronously; otherwise, return reading failure.

[0180] (3-6) Perform the packaging device mounting operation;

[0181] This step specifically includes the following sub-steps:

[0182] (3-6-1) Using the wrapper tag extracted in step (3-2) as the directory name, determine whether the directory exists in the root directory of the file system. If it exists, obtain the node ID of its wrapper and proceed to step (3-6-3); otherwise, proceed to step (3-6-2).

[0183] (3-6-2) Create the directory in the root directory of the file system with the wrapper label as the directory name, record the directory path, and insert the wrapper label and the newly created directory path into the registry. Return to step (3-6-1).

[0184] (3-6-3) Using the extended attribute name extracted in step (3-1) and the extended attribute value read in step (3-5) as query conditions, query the namespace query module to obtain the node ID list of the corresponding wrapper.

[0185] (3-6-4) Take the node ID of the wrapper in step (3-6-1) and the wrapper label in step (3-2), calculate the metric value, generate the relation index key, and take the ID from the list of node IDs of the wrapper in step (3-6-3) as the value and insert it into the second mapping table as an entry.

[0186] (3-6-5) The namespace query module periodically triggers the wrapper update operation, which hands over the key-value pairs in the update state in the second mapping table to the second metadata management module, and the second data management module writes the key-value pairs that need to be updated back to the disk;

[0187] Steps (3-6), specifically the wrapper mounting, are another core innovation of this invention. The idea is to dynamically mount wrappers of interest to the corresponding wrapper directory in the root directory using directory mounting. The advantage is that users can directly access files with the same extended attribute values ​​in the root directory without manually traversing the entire directory structure to find the files of interest, thus accelerating the process of filtering out relevant data from massive amounts of data.

[0188] (4) Extract the access path and extended metadata name from the IO request, determine whether the corresponding metadata value can be deleted from the rich index node of the file where the access path is located according to the extended metadata name, if the reading is successful, return the metadata value and then the process ends; otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends.

[0189] This step includes the following sub-steps:

[0190] (4-1) Extract the access path and extend the metadata name from the IO request;

[0191] (4-2) Submit the access path obtained in step (4-1) to the registration module to obtain the corresponding wrapper tag;

[0192] (4-3) Submit the wrapper tag obtained in step (4-2) and the access path extracted in step (4-1) to the namespace query module. The namespace query module recursively parses the access path to obtain the node ID of the rich index node;

[0193] (4-4) Submit the node ID of the rich index node extracted in step (4-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node.

[0194] (4-5) From the data extension area of ​​the rich index node extracted in step (4-3), determine whether the extension metadata value can be deleted based on the extension metadata name extracted in step (4-1). If it can be deleted, remove the corresponding extension metadata name and extension attribute value, and proceed to step (4-6) asynchronously; otherwise, return deletion failure.

[0195] (4-6) Using the extended metadata name obtained in step (4-1) and the extended metadata value that needs to be deleted in step (4-5) as query conditions, query the namespace query module to obtain the node ID of the wrapper that packages the extended attribute.

[0196] (4-7) Encode the wrapper tag extracted in step (4-2) and the node ID of the wrapper extracted in step (4-6) into an attribute locating key, and read the locating attribute metadata from the third mapping table;

[0197] (4-8) Remove the extended metadata name extracted in step (4-1) and the node ID of the rich index node extracted in step (4-3) from the location attribute metadata of the sheet in step (4-7);

[0198] (4-9) The namespace query module periodically triggers the wrapper update operation, which hands over the key-value pairs in the updated state in the third mapping table to the second metadata management module, which then writes the key-value pairs that need to be updated back to the disk.

[0199] (5) Extract the access path from the IO request, determine whether all extended metadata names and extended metadata values ​​can be read from the rich index node of the file where the access path is located. If the reading is successful, return the corresponding extended metadata names and extended metadata values ​​as a list and then the process ends. Otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends.

[0200] This step includes the following sub-steps:

[0201] (5-1) Extract the access path from the IO request;

[0202] (5-2) Submit the access path obtained in step (5-1) to the registration module to obtain the corresponding wrapper tag;

[0203] (5-3) Submit the wrapper tag obtained in step (5-2) and the access path extracted in step (5-1) to the namespace query module. The namespace query module recursively parses the access path to obtain the node ID of the rich index node;

[0204] (5-4) Submit the node ID of the rich index node extracted in step (5-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node.

[0205] (5-5) From the data extension area of ​​the rich index node extracted in step (5-3), determine whether the extension metadata name can be read. If it can be read, read all the extension attribute names and insert the read extension attribute names into the extension attribute name list, and proceed to step (5-6). Otherwise, return and display a failure notification.

[0206] (5-6) Take out the extended attribute names from the extended attribute name list in turn, and together with the input access path, use them as parameters to construct an extended metadata read IO request to trigger the extended metadata read request.

[0207] (5-7) The file system responds to the extended attribute metadata read requests in sequence and completes the response;

[0208] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A file system supporting metadata management, comprising a registration module, a namespace query module, a first data management module, and a second data management module, characterized in that, The registration module is used to receive input / output (IO) requests sent by users, obtain the access path corresponding to the IO request, query the corresponding wrapper tag in the pre-established registry according to the IO request, and send the access path and wrapper tag to the namespace query module. The namespace query module is used to parse the access path and generate a wrapper key using the parsed current directory and wrapper tag. It determines whether the wrapper key can be used to query the first, second, third, and fourth mapping tables in memory to obtain the wrapper and rich index node data corresponding to the current directory. If so, the node ID of the rich index node is obtained in memory and sent to the first data management module. Otherwise, the wrapper tag and the access path are sent to the second data management module, which loads the corresponding data from the disk. When the wrapper is loaded from disk into memory, three different mapping tables are generated in memory; The entries in the first mapping table exist in the form of key-value pairs, where the key is a string composed of the wrapper tag and the wrapper node ID, i.e., the wrapper key, and the value is the wrapper metadata, including the wrapper tag, main attribute name, main attribute value, group access permissions, and metrics. The entries in the second mapping table exist in the form of key-value pairs, where the key is a string composed of the wrapper label, the wrapper node ID, and the metric, i.e., the relation index key, and the value is the node ID of the wrapper associated with it; The entries in the third mapping table exist in the form of key-value pairs, where the key is a string concatenated from the wrapper tag, the wrapper node ID, and the next-level path name of the current directory, i.e., the attribute locator key, and the value is the node ID of the rich index node. When a rich index node is loaded from disk into memory, a fourth mapping table is generated in memory. The entries in the fourth mapping table exist in the form of key-value pairs, where the key is the node ID of the rich metadata index node and the value is the file metadata, including POSIX metadata of the portable operating system interface, extended metadata EA and small file data. The first data management module is used to read and write corresponding file data in the local file system based on the node ID of the rich index node; The second data management module is used to retrieve the corresponding wrapper data and rich index node data from the disk based on the keys of the four mapping tables provided by the namespace query module, and to build the wrapper and rich index node in memory based on the retrieved wrapper data and rich index node data to populate the corresponding mapping table. The namespace query module is also used to send the data of the updated wrapper and the data of the rich index node to the second data management module; The second data management module is also used to write the updated wrapper data and rich index node data back to disk in key-value pairs using SPDK technology.

2. The file system supporting metadata management according to claim 1, characterized in that, Entries in the registry exist as key-value pairs, where the key is the access path and the value is the wrapper tag, which is used to establish associations with different wrappers; The wrapper structure includes wrapper metadata, attribute location metadata, and relation index metadata; The wrapper metadata record contains the basic metadata common to the files belonging to the wrapper, and records the wrapper tag, which is used to mark which attribute the metadata wrapper uses as the positioning attribute; The attribute location metadata record contains the rich index node IDs contained in this wrapper; The relation index metadata records the relationships between wrappers.

3. The file system supporting metadata management according to claim 2, characterized in that, The namespace query module parses the access path and generates a wrapper key using the parsed current directory and wrapper tag. It then determines whether the wrapper key can be used to sequentially query the first, second, third, and fourth mapping tables in memory to obtain the wrapper and rich index node data corresponding to the current directory. If so, it further retrieves the node ID of the rich index node in memory and sends it to the first data management module. Otherwise, it sends the wrapper tag and access path to the second data management module, which then loads the corresponding data from the disk. This process includes the following sub-steps: (1-1) Parse the access path, extract the root directory " / " as the current directory, and obtain the node ID of the wrapper corresponding to the current directory. The node ID is initialized to 0 by default. (1-2) Concatenate the wrapper tag corresponding to the access path and the node ID of the wrapper obtained in step (1-1) to generate a wrapper key, and determine whether the corresponding wrapper metadata can be obtained in the pre-established first mapping table based on the wrapper key. If it can, proceed to step (1-3); otherwise, proceed to step (1-4). (1-3) Evaluate whether the user has permission to access the wrapper corresponding to the wrapper key based on the obtained metadata. If yes, proceed to step (1-5); otherwise, return a notification that the user does not have sufficient permissions and the process ends. (1-4) Submit the wrapper key to the second data management module and determine whether the second data management module can load the metadata corresponding to the wrapper key from the disk into memory. If it can, put the wrapper key as the key and the metadata as the value into the first mapping table, and then return to step (1-2). Otherwise, it means that the access path does not exist, return the access path does not exist notification, and the process ends. (1-5) Further parse the access path to extract the next level path of the current directory as the current directory, and determine whether the current directory is the last level path. If it is, proceed to step (1-8); otherwise, proceed to step (1-6). (1-6) Take the next level path of the current directory as the metric value, concatenate the wrapper tag, the node ID of the wrapper corresponding to the current directory, and the metric value string to generate a relation index key, and determine whether the node ID of the wrapper corresponding to the next level path can be read in the pre-established second mapping table according to the relation index key. If yes, return to step (1-3); otherwise, proceed to step (1-7). (1-7) Submit the relation index key to the second data management module and determine whether the second data management module can load the metadata corresponding to the relation index key from the disk into memory. If it can, put the relation index key as the key and the metadata as the value into the second mapping table, and then return to step (1-6). Otherwise, it means that the access path does not exist, return the access path does not exist notification, and the process ends. (1-8) Encode the wrapper tag and the node ID of the wrapper corresponding to the current directory and the next level path of the current directory into strings to generate an attribute locator key, and determine whether the node ID of the rich index node corresponding to the attribute locator key can be obtained in the pre-established third mapping table. If it can be obtained, proceed to step (1-9); otherwise, proceed to step (1-11). (1-9) Determine whether the corresponding file metadata can be obtained from the pre-established fourth mapping table using the node ID of the rich index node as the key. If yes, the namespace query process is completed and the process ends; otherwise, proceed to step (1-10). (1-10) Send the node ID of the rich index node to the second data management module and determine whether the corresponding file metadata can be obtained from the disk. If so, put the node ID of the rich index node as the key and the file metadata as the value into the fourth mapping table and return to step (1-9). Otherwise, the user queries a directory instead of a file and proceed to step (1-11). (1-11) Take the next level path of the current directory as the metric value, concatenate the wrapper tag, the node ID of the wrapper corresponding to the current directory, and the metric value string to generate a relation index key, and determine whether the node ID of the wrapper corresponding to the next level path can be read in the pre-established second mapping table according to the relation index key. If yes, proceed to step (1-12); otherwise, proceed to step (1-14). (1-12) Concatenate the wrapper tag corresponding to the access path and the node ID of the wrapper corresponding to the next level path obtained in step (1-11) to generate a wrapper key, and determine whether the corresponding wrapper metadata can be obtained in the pre-established first mapping table based on the wrapper key. If it can, the namespace query process is completed and the process ends; otherwise, proceed to step (1-13). (1-13) Submit the wrapper key obtained in step (1-12) to the second data management module and determine whether the second data management module can load the metadata corresponding to the wrapper key from the disk into memory. If it can, put the wrapper key as the key and the metadata as the value into the first mapping table, and then return to step (1-12). Otherwise, it means that the access path does not exist, and return the access path does not exist notification, and the process ends.

4. The file system supporting metadata management according to claim 3, characterized in that, The disk is configured with a first sorting area, a second sorting area, a third sorting area, and a fourth sorting area. The first sorting area is used to store file metadata of rich index nodes and grouping metadata of wrappers. The second sorting area is used to store extended attributes and small files. The third sorting area is used to store attribute positioning metadata and relation index metadata of wrappers. The fourth sorting area is used to store logs. The second data management module retrieves the corresponding wrapper data and rich index node data from the disk based on the keys of the four mapping tables provided by the namespace query module. It then constructs the wrapper and rich index node in memory based on the retrieved wrapper and rich index node data to populate the corresponding mapping table. This process includes the following sub-steps: (2-1) Determine the type of the obtained key. If the key is the attribute location key of the wrapper and the relationship index key of the wrapper, proceed to step (2-2) and read the data from the third storage area of ​​the disk using the B+ tree index. If the key is the wrapper key and the rich index node key, proceed to step (2-3) and read the data from the first or second storage area of ​​the disk using the hash index. (2-2) Using the key as input, query the B+ tree index to determine if data can be read. If so, return the read data; otherwise, return failure. This step specifically includes the following sub-steps: (2-2-1) Use the root node ID of the B+ tree index as the page table ID, and use SPDK technology to read the corresponding data page table from the metadata SSD; (2-2-2) Convert the data page table obtained in step (2-2-1) into B+ tree index nodes; (2-2-3) Determine whether the B+ tree index node in step (2-2-2) is a B+ tree index intermediate node. If so, proceed to step (2-2-4); otherwise, proceed to step (2-2-8). (2-2-4) Convert the B+ tree index nodes in step (2-2-2) into B+ tree index intermediate nodes; (2-2-5) Determine whether the input key in (2-2) is within the key value range of the middle node in step (2-2-4). If it is, jump to step (2-2-8); otherwise, proceed to step (2-2-6). (2-2-6) Determine whether the input key in (2-2) is less than the key of the middle section of the B+ tree index obtained in step (2-2-4). If it is less, obtain the node ID of the left node of the middle node and then proceed to step (2-2-7). Otherwise, obtain the node ID of the right node of the middle node and then proceed to step (2-2-7). (2-2-7) Use the node ID in step (2-2-6) as the page table ID, use SPDK technology to read the corresponding data page table from the metadata SSD, and return to step (2-2-3). (2-2-8) Convert the B+ tree index node into a B+ tree index leaf node; (2-2-9) Determine whether there is a key-value pair in the data page table of the leaf node of the B+ tree index obtained in step (2-2-8) that matches the key obtained in (2-2). If it exists, return the data corresponding to the key and the process ends; otherwise, return that the search failed and the process ends. (2-3) Take the key as input, query the hash index, and determine if the data can be read. If it can, return a success notification; otherwise, return a failure notification. This step specifically includes the following sub-steps: (2-3-1) Use a hash algorithm to obtain the hash value of the key input in step (2-1); (2-3-2) Convert the hash value obtained in step (2-3-1) into a global index value; (2-3-3) Calculate and generate a local index value using the key from step (2-3-1) and the global index value from step (2-3-2); (2-3-4) Read the hash bucket based on the local index value obtained in step (2-3-3), and read the page table ID from the hash bucket; (2-3-5) Based on the page table ID read in (2-3-4), the corresponding data page table is read from the metadata SSD using SPDK technology; (2-3-6) Use the key entered in step (2-3) to find the corresponding key-value pair in the data page table obtained in step (2-3-5), and determine whether the data corresponding to the key can be read. If it can, it means that the key-value pair query is successful, and the data corresponding to the key is returned and the process ends. Otherwise, it means that the search failed, and a read failure notification is returned and the process ends.

5. The file system supporting metadata management according to claim 4, characterized in that, The second data management module uses SPDK technology to write the updated wrapper data and rich index node data back to disk. This process includes the following sub-steps: (3-1) Obtain the key and data that need to be updated, determine the type of the obtained key, if the key is the attribute positioning key of the wrapper and the relationship index key of the wrapper, then proceed to step (3-2), and use the B+ tree index to write the corresponding data to the third storage area; if the key is the wrapper key and the rich index node key, then proceed to step (3-3), and use the hash index to write the corresponding data to the first or second storage area of ​​the disk. (3-2) Using the key and data input in step (3-1) as input, query the B+ tree index, insert the key and the data corresponding to the updated table entry into the B+ tree index, and use SPDK technology to write the data page table of the updated B+ tree index to disk; this step specifically includes the following sub-steps: (3-2-1) Determine if the root node of the B+ tree index exists. If it exists, proceed to step (3-2-3); otherwise, proceed to step (3-2-2). (3-2-2) Request a new page table ID, initialize the B+ tree index root node using the new page table ID, and return to step (3-2-1). (3-2-3) Read the root node of the B+ tree index, obtain its page table ID from the root node, and load the corresponding data page from the disk based on the page table ID and using SPDK technology; (3-2-4) Convert the data page table obtained in step (3-2-3) into B+ tree index nodes; (3-2-5) Determine whether the B+ tree index node in step (3-2-4) is an intermediate node. If it is, proceed to (3-2-6); otherwise, proceed to (3-2-10). (3-2-6) Convert the B+ tree index node in step (3-2-5) into an intermediate node; (3-2-7) Determine whether the key input in (3-1) is within the key value range of the middle node in step (3-2-6). If it is, jump to step (3-2-10); otherwise, proceed to step (2-2-8). (3-2-8) Determine whether the input key in (3-1) is less than the key of the middle node in step (3-2-6). If it is less, obtain the node ID of the left node of the middle node; otherwise, obtain the node ID of the right node of the middle node. (3-2-9) Use the node ID in step (3-2-8) as the page table ID, use SPDK technology to read the corresponding data page table from the metadata SSD, and return to step (3-2-3). (3-2-10) Convert the B+ tree index node in step (3-2-5) or step (3-2-7) into a B+ tree index leaf node; (3-2-11) Compare the key in step (3-1) with the data page table in the leaf node of the B+ tree index in step (3-2-10) to determine whether there is a key-value pair that matches the key in the data page table. If there is, use the data input in step (3-1) to overwrite the data in the key-value pair that matches the key, return a notification that the data insertion was successful, and the process ends. Otherwise, proceed to step (3-2-12). (3-2-12) Determine whether there is remaining space in the data page table of the leaf node of the B+ tree index. If so, insert the key and value entered in step (3-1) into the data page table of the leaf node of the B+ tree index and return that the data insertion was successful. Otherwise, proceed to step (3-2-13). (3-2-13) Perform paging operation on the B+ index node where the data page table in the leaf node of the B+ tree index in step (3-2-12), create left and right nodes respectively, allocate left and right data page tables, and then write the page table data to the left and right data page tables respectively according to the key value range of the left and right nodes, while releasing the page table space of the B+ index node; (3-2-14) Determine whether the key in step (3-1) is within the key value range of the left data page table in step (3-2-13). If so, write the key and value in step (3-1) to the left subpage and update the left node index information in step (3-2-13). Otherwise, write the key and value in step (3-1) to the right data page table and update the right node index information in step (3-2-13). (3-2-15) Write the data of the left page table back to the disk according to the ID of the left page table and using SPDK technology. At the same time, write the data of the right page table back to the disk according to the ID of the right page table and using SPDK technology. Return that the data insertion was successful. (3-3) Using the key and data input in step (3-1) as input, query the hash index, insert the key and data into the hash index, and use SPDK technology to write the updated hash bucket to disk; this step specifically includes the following sub-steps: (3-3-1) Use the hash algorithm murmur3 to obtain the hash value of the key input in step (3-1); (3-3-2) Convert the hash value obtained in step (3-3-1) into a global index value; (3-3-3) Calculate and generate a local index value using the input key and the global index value extracted in step (3-3-2); (3-3-4) Use the local hash value from step (3-3-3) to read the hash bucket and read the page table ID from the hash bucket; (3-3-5) Based on the page table ID extracted in step (3-3-4), the corresponding data page is read from the disk using SPDK technology; (3-3-6) Convert the data pages extracted in step (3-3-5) into hash buckets; (3-3-7) Determine if there is remaining space in the hash bucket. If there is, write the input key and value into the hash bucket and proceed to step (3-3-9); otherwise, proceed to step (3-3-8). (3-3-8) Request a new page table, obtain the new page table ID, and return to step (3-3-5). (3-3-9) Update the local index value extracted in step (3-3-3) and the global index value extracted in step (3-3-2); (3-3-10) Use SPDK technology to convert the updated hash bucket into a page table and write it back to disk, and return a notification that the data insertion was successful.

6. A method for implementing rich metadata services using a file system supporting metadata management as described in any one of claims 1 to 5, characterized in that, Includes the following steps: (1) Obtain the IO request from the user and determine whether the IO request is an insert operation request, a read operation request, a delete operation request, or a display operation request. If it is an insert operation request, proceed to step (2); if it is a read operation request, proceed to step (3); if it is a delete operation request, proceed to step (4); if it is a display operation request, proceed to step (5). (2) Extract the access path, extended metadata name and extended metadata value from the IO request, determine whether the extended metadata name and extended metadata value can be inserted into the rich index node of the file where the access path is located, if successful, return a notification that the metadata insertion was successful and then the process ends; otherwise, return a notification that the specified access path does not exist and then the process ends. (3) Extract the access path and extended metadata name from the IO request, determine whether the corresponding metadata value can be read from the rich index node of the file where the access path is located according to the extended metadata name, if the reading is successful, return the metadata value and then the process ends; otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends. (4) Extract the access path and extended metadata name from the IO request, determine whether the corresponding metadata value can be deleted from the rich index node of the file where the access path is located according to the extended metadata name, if the reading is successful, return the metadata value and then the process ends; otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends. (5) Extract the access path from the IO request, determine whether all extended metadata names and extended metadata values ​​can be read from the rich index node of the file where the access path is located. If the reading is successful, return the corresponding extended metadata names and extended metadata values ​​as a list and then the process ends. Otherwise, return that there is no metadata corresponding to the specified metadata name and then the process ends.

7. The method for implementing rich metadata services using a file system supporting metadata management according to claim 6, characterized in that, Step (2) specifically includes the following sub-steps: (2-1) Extract the access path, extended metadata name, and extended metadata value from the IO request; (2-2) Submit the access path obtained in step (2-1) to the registration module to obtain the corresponding wrapper tag; (2-3) Submit the wrapper tag obtained in step (2-2) and the access path extracted in step (2-1) to the namespace query module; the namespace query module recursively parses the access path and obtains the node ID of the rich index node; (2-4) Submit the node ID of the rich index node extracted in step (2-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node; (2-5) Based on the selected storage mode, write the metadata name and extended metadata value extracted in step (2-1) to the data extension area of ​​the rich index node; (2-6) The second data management module writes the data of the updated rich index nodes back to the disk in real time; (2-7) Determine whether step (2-6) was executed successfully. If it was executed successfully, return that the rich metadata insertion was successful and proceed to step (2-8) in an asynchronous manner. Otherwise, return that the rich metadata lookup failed. (2-8) Perform the packing operation using the packer; This step specifically includes the following sub-steps: (2-8-1) Submit the extended metadata name and extended metadata value extracted in step (2-1) to the namespace query module. The namespace query module will trigger the extended attribute query operation. If the query is successful, obtain the node ID of the wrapper whose main attribute is the extended attribute value and proceed to step (2-8-2). Otherwise, proceed to step (2-8-3). (2-8-2) Generate an attribute positioning key by taking the wrapper ID extracted in step (2-8-1), the extended attribute value extracted in step (2-1), and the wrapper tag code extracted in step (2-2). Query the third mapping table to determine whether the wrapper positioning attribute can be obtained. If it can be obtained, proceed to step (2-8-3); otherwise, proceed to step (2-8-4). (2-8-3) Insert the node ID of the rich index node obtained in step (2-3) and the extended attribute name obtained in step (2-1) into the wrapper positioning attribute obtained in step (2-8-2), and proceed to step (2-8-6). (2-8-4) The naming query module creates a new wrapper, assigns a node ID to the new wrapper, generates group metadata, relation index metadata and attribute location metadata for the wrapper, and inserts the generated wrapper data into the first mapping table, the second mapping table and the third mapping table in key-value pairs in sequence; (2-8-5) Insert the node ID of the rich index node obtained in step (2-3) and the extended attribute name obtained in step (2-1) into the newly created wrapper positioning attribute in step (2-8-2), and proceed to step (2-8-6). (2-8-6) The namespace query module periodically triggers the wrapper update operation, which hands over the key-value pairs in the first, second, and third mapping tables that are in the update state to the second metadata management module, which then writes the key-value pairs that need to be updated back to the disk.

8. The method for implementing rich metadata services using a file system supporting metadata management according to claim 7, characterized in that, Step (3) includes the following sub-steps: (3-1) Extract the access path and extend the metadata name from the IO request; (3-2) Submit the access path obtained in step (3-1) to the registration module to obtain the corresponding wrapper tag; (3-3) Submit the wrapper tag obtained in step (3-2) and the access path extracted in step (3-1) to the namespace query module; the namespace query module recursively parses the access path and obtains the node ID of the rich index node; (3-4) Submit the node ID of the rich index node extracted in step (3-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node; (3-5) From the data extension area of ​​the rich index node extracted in step (3-3), determine whether the extension metadata value can be read according to the extension metadata name extracted in step (3-1). If the reading is successful, return the read extension metadata value and proceed to step (3-6) in an asynchronous manner; otherwise, return reading failure. (3-6) Perform the packaging device mounting operation; this step specifically includes the following sub-steps: (3-6-1) Using the wrapper tag extracted in step (3-2) as the directory name, determine whether the directory exists in the root directory of the file system. If it exists, obtain the node ID of its wrapper and proceed to step (3-6-3); otherwise, proceed to step (3-6-2). (3-6-2) Create the directory in the root directory of the file system with the wrapper label as the directory name, record the directory path, and insert the wrapper label and the newly created directory path into the registry, then return to step (3-6-1). (3-6-3) Using the extended attribute name extracted in step (3-1) and the extended attribute value read in step (3-5) as query conditions, query the namespace query module to obtain the node ID list of the corresponding wrapper. (3-6-4) Take the node ID of the wrapper in step (3-6-1) and the wrapper label in step (3-2), calculate the metric value, generate the relation index key, and take the ID from the list of node IDs of the wrapper in step (3-6-3) as the value and insert it into the second mapping table as an entry. (3-6-5) The namespace query module periodically triggers the wrapper update operation, which hands over the key-value pairs in the second mapping table that are in the update state to the second metadata management module, and the second data management module writes the key-value pairs that need to be updated back to the disk.

9. The method for implementing rich metadata services using a file system supporting metadata management according to claim 8, characterized in that, Step (4) includes the following sub-steps: (4-1) Extract the access path and extend the metadata name from the IO request; (4-2) Submit the access path obtained in step (4-1) to the registration module to obtain the corresponding wrapper tag; (4-3) Submit the wrapper tag obtained in step (4-2) and the access path extracted in step (4-1) to the namespace query module; the namespace query module recursively parses the access path to obtain the node ID of the rich index node; (4-4) Submit the node ID of the rich index node extracted in step (4-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node; (4-5) The data extension area of ​​the rich index node extracted in step (4-3) is determined according to the extension metadata name extracted in step (4-1) to see if the extension metadata value can be deleted. If it can be deleted, the corresponding extension metadata name and extension attribute value are removed and the process proceeds to step (4-6) asynchronously. Otherwise, the deletion will fail. (4-6) Using the extended metadata name obtained in step (4-1) and the extended metadata value to be deleted in step (4-5) as query conditions, query the namespace query module to obtain the node ID of the wrapper that packages the extended attribute; (4-7) Encode the wrapper tag extracted in step (4-2) and the node ID of the wrapper extracted in step (4-6) into an attribute locating key, and read the locating attribute metadata from the third mapping table; (4-8) Remove the extended metadata name extracted in step (4-1) and the node ID of the rich index node extracted in step (4-3) from the location attribute metadata of the sheet in step (4-7); (4-9) The namespace query module periodically triggers the wrapper update operation, which hands over the key-value pairs in the updated state in the third mapping table to the second metadata management module, which then writes the key-value pairs that need to be updated back to the disk. Step (5) includes the following sub-steps: (5-1) Extract the access path from the IO request; (5-2) Submit the access path obtained in step (5-1) to the registration module to obtain the corresponding wrapper tag; (5-3) Submit the wrapper tag obtained in step (5-2) and the access path extracted in step (5-1) to the namespace query module; the namespace query module recursively parses the access path and obtains the node ID of the rich index node; (5-4) Submit the node ID of the rich index node extracted in step (5-3) to the second data management module, which will load the data of the rich index node into memory and construct the rich index node. (5-5) From the data extension area of ​​the rich index node extracted in step (5-3), determine whether the extension metadata name can be read. If it can be read, read all the extension attribute names and insert the read extension attribute names into the extension attribute name list, and proceed to step (5-6). Otherwise, return a notification of failure. (5-6) Take out the extended attribute names from the extended attribute name list in turn, and together with the input access path, use them as parameters to construct an extended metadata read IO request to trigger the extended metadata read request; (5-7) The file system responds to the extended attribute metadata read requests in sequence and completes the response.

Images