Metadata structure and operation, positioning and segmentation methods based on metadata structure

Abstract

The invention discloses a metadata structure and operation, positioning and segmentation methods based on the metadata structure. The structure comprises a directory metadata structure and a file metadata structure; the directory metadata structure comprises a directory index and multiple directory metadata, wherein data fields of all the directory metadata are fixed-length fields; the file metadata structure comprises a file index and multiple file metadata, wherein data fields of all the file metadata are fixed-length fields. The metadata structure has the following advantages that unified expression of the metadata in internal storage and external storage is achieved, and serialization and deserialization operations do not need to be conducted on the metadata fields.

Description

technical field [0001] The invention relates to the field of computers, in particular to a metadata structure and an operation method, a positioning method, and a segmentation method based on the metadata structure. Background technique [0002] A distributed file system is a commonly used storage system. It has many advantages, such as strong scalability, simple maintenance, and low cost. Therefore, it is very suitable for storing large amounts of data that grows rapidly. However, distributed file systems also have some shortcomings, such as high latency and poor scalability for metadata access. [0003] The most important thing in a distributed file system is the management of metadata. A widely used method is to store metadata in a key-value store (Key-Value Store) system. This key-value system can efficiently complete metadata queries, and can guarantee data consistency (Consistency) and durability (Durability). Since the key-value system is currently widely used, it ...

AI Technical Summary

Problems solved by technology

Although this method can provide fast data processing when the cache hits, there are problems that the memory space cannot be fully utilized and the serialization and deserialization overhead is large

Among them, the memory cannot be fully utilized, which is mainly reflected in the structured cache and the unstructured cache in the key-value database. The content of the data is consistent, but two copies of the content are stored in different forms.

The overhead of serialization and deserialization is reflected in the fact that in the case of a cache miss, every time a part of a file's metadata is modified or read, the metadata of the entire file needs to be serialized and deserialized , this method leads to relatively low data processing

Images