Method for storing mass of small files on basis of master-slave distributed file system

Abstract

The invention discloses a method for storing a mass of small files on the basis of a master-slave distributed file system, and aims to solve the problem about storing the mass small files in the master-slave distributed file system. The technical scheme includes that the method includes that a mass small file storage system is deployed and initialized, and then small files are created and read by a SmallFileAPI (small file-application program interface) of a client side according to a command received from a keyboard. When the small files are created by a system, the SmallFileAPI creates data files of the small files according to small file paths obtained from the client side, writes in small file data and simultaneously creates small file indexes at data nodes; when the small files are read by the system, data node information corresponding to a parent directory is acquired according to the small file paths, an index request is transmitted to any of the data nodes, and finally the small file data are read from the data files according to index information. By the aid of the method, the problem about mass of storage metadata of the mass of small files is solved, writing efficiency of the mass small file storage system is improved, and reliability of the system is guaranteed.

Description

technical field [0001] The invention relates to a method for storing a large number of small files on a master-slave distributed file system oriented to storage of a large number of large files. Background technique [0002] With the development of new computing technologies, both business and personal data began to grow rapidly. The growth of massive data brings not only the problem of storage capacity, but also brings challenges to data management and storage performance, which has become the core problem that needs to be solved in the era of cloud computing. In order to ensure high availability, high reliability and economy of data, cloud computing uses distributed storage to store data, and redundant storage to ensure data reliability. In order to meet the needs of a large number of users, the storage technology of cloud computing must have high throughput and high transfer rate. For the data storage problem of cloud computing, industry and academia have proposed a var...

AI Technical Summary

Problems solved by technology

[0011] The technical problem to be solved by the present invention is: the general master-slave distributed file system can store files with a large data scale, and has high fault tolerance and high scalability, but it will cause some problems when used to store a large number of small files: (1) Centralized metadata serv

Images