Distributed storage method and system for massive small files

Abstract

The invention provides a distributed storage method and a distributed storage system for a mass of small files, the method comprising the steps of forming a plurality of virtual disks from physical disks through logical partitions; classifying the files according to the naming rules of a large number of small files, and creating directory index tree; according to the file naming rules, obtaining the storage location in the directory index tree, storing the massive small files, recording the corresponding storage directory as the original directory, and redundantly storing the massive small files, recording the corresponding storage directory as the redundant directory; triggering a data synchronization and recovery operation when the original directory and the file data in the redundant directory are inconsistent. The invention aims at the distributed storage of the massive small files, realizes the rapid positioning information in the massive data by establishing the index and the file name positioning when storing a large number of small files, and ensures the reliability by using the data redundancy, the mechanism is simple, the fault tolerance is strong, and the storage space of the metadata is reduced while the correctness is ensured.

Description

technical field [0001] The present invention relates to the field of distributed storage, in particular to a distributed storage method and system for a large number of small files, especially to a distributed data organization method applied in a typhoon analysis system. Background technique [0002] With the advancement of science and technology, human society is entering an era of digital information explosion. The popularity of the Internet has brought new growth points to the development of traditional industries. All walks of life have begun to transform to informationization, and the total amount of data is growing exponentially. Facing the explosion of data, the first problem that needs to be solved is how to store it effectively. Obviously, the traditional stand-alone storage mode is far from meeting the actual requirements. Distributed storage has become an inevitable requirement of modern information storage. [0003] The meteorological field is also in the pro...

AI Technical Summary

Problems solved by technology

This leads to serious waste of resources when HDFS stores a large number of small files

It can be seen that the common distributed file system cannot solve the problem of data storage and data positioning in the typhoon analysis system very well

Images