Intelligent layering method and system based on distributed file storage

Abstract

The invention discloses an intelligent layering method and system based on distributed file storage, the system comprises a hot data layer and a cold data layer, the hot data layer comprises a metadata module, a data storage module, a coordinator and an agent module; the metadata module is used for describing system data of features of a file and comprises metadata information of the file; the data storage module comprises file data actually stored in the file system; the coordinator manages a timed scanning task and a cold and hot data definition strategy; the agent module stores a record of the to-be-migrated file conforming to the strategy; the cold data layer comprises an object barrel module, and an object barrel is used for storing cold data; a copying tool module is arranged between the hot data layer and the cold data layer, and the copying tool module executes copying between the cold data and the hot data. Data scanning is carried out according to time and a cold and hot data strategy, and cold data is migrated to an object which is relatively low in price and low in performance to be stored. Cold data is efficiently saved in low-cost object storage.

Description

Technical field [0001] The present invention relates to the field of data information processing, and more particularly to a smart hierarchical method and system based on distributed file storage. Background technique [0002] Currently, more and more users face the rapid growth of non-structural data (files, pictures, videos, etc.), reports according to Gartner and IDC, the annual composite growth rate of non-structured data exceeds 40%. In most applications where users, the accessibility of unstructured data has a clear difference in heat and heat data. The access performance of hot data is higher. After a certain period of time (in different applications, the life cycle of data is different) The thermal data gradually becomes cold, and the frequency of applying access to these cold data will become very low. How to solve massive non-structured data storage and access, and the layering of hot and cold data, can help solve the lower cost of enterprise users, that is, to meet the...

AI Technical Summary

Problems solved by technology

If the upper-layer application needs to access cold data, the application must first know that the data is in the cold data layer, and then modify the access method, which cannot be transparent and unaware of the application

[0005] In the solution of using different storage devices, since the access space and access methods of the two devices are different, when the data is archived to the cold data space such as the tape library, if the upper-layer application needs to access the cold data, the application must first know that the data is in the cold storage space. In the data layer, and then modify the access method, which means that the application cannot access all data transparently and imperceptibly

Images