Distributed storage method supporting mounting of multi-file system

Abstract

The invention relates to a distributed storage system supporting multi-file system mounting, which comprises: a file space management module for dividing a plurality of file subspaces of different usescenes in a root file space of a file arrangement layer, and recording path mapping of a file directory level; the metadata management module which is used for maintaining files mounted on the file arrangement layer; the label management module which is used for recording and maintaining business meaning labels related to files stored in the distributed file storage system; the data life cycle management module which is used for processing storage strategies of different data life stages of cold and hot temperature data of the files in the file storage system, packaging and compressing the cold data and putting the cold data into the object storage system, putting the temperature data into an original underlying file system where the files are located, and loading the hot data into a memory for preparation at any time, the object storage system is used as a cold data storage system, a unique identifier is allocated to each object of the object storage system, and a server or an end user is allowed to retrieve the objects.

Description

technical field [0001] The invention relates to operating system technology, in particular to a distributed storage method supporting multi-file system mounting. Background technique [0002] Since the birth of modern operating systems, operating systems such as linux and windows support the mounting and reading and writing of various file systems (xfs, ntfs, ext4, etc.). These file systems are designed and implemented for stand-alone operating systems. Provide a unified access interface for operations such as reading, writing, modifying, and deleting files on a stand-alone system. Today, however, limited by the stagnant growth rate of Moore's Law, the expected growth in the processing power and vertical expansion capabilities of a single machine is not as good as before, and the mainstream large-scale data processing and heavy computing and analysis tasks are gradually transferred to distributed systems. Running, task scheduling, process management, file management and oth...

AI Technical Summary

Problems solved by technology

The data warehouse system built in this way often needs to rely on its own data extraction, data transmission, and data conversion tools. Each piece of data that finally reaches business users will generate multiple redundant data in the data processing process, which is important for storage. Waste of resources; and when there are multiple distributed block file system instances on the same platform, the file spaces between multiple distributed block file systems are not interoperable, and when files need to be read or written across distributed block file systems , there will be a lot of inconvenience

At the same time, in the distributed block file system, the minimum unit of the underlying storage is a file block of 128MB. When the data has a large number of small files (such as pictures, short audio and video), the data scene is not suitable for the distributed block file system as a storage component. Because a large number of small files will put pressure on the metadata lookup on the central metadata server

In terms of deployment, traditional big data platforms based on distributed block file systems need to deploy the execution nodes of data engines such as Hive, HBase, and Spark to the same batch of servers as the data nodes of the distributed block file system. Coupled solutions limit deployment flexibility

Images