Method for storing mass of small files on basis of master-slave distributed file system

Abstract

The invention discloses a method for storing a mass of small files on the basis of a master-slave distributed file system, and aims to solve the problem about storing the mass small files in the master-slave distributed file system. The technical scheme includes that the method includes that a mass small file storage system is deployed and initialized, and then small files are created and read by a SmallFileAPI (small file-application program interface) of a client side according to a command received from a keyboard. When the small files are created by a system, the SmallFileAPI creates data files of the small files according to small file paths obtained from the client side, writes in small file data and simultaneously creates small file indexes at data nodes; when the small files are read by the system, data node information corresponding to a parent directory is acquired according to the small file paths, an index request is transmitted to any of the data nodes, and finally the small file data are read from the data files according to index information. By the aid of the method, the problem about mass of storage metadata of the mass of small files is solved, writing efficiency of the mass small file storage system is improved, and reliability of the system is guaranteed.

Description

technical field [0001] The invention relates to a method for storing a large number of small files on a master-slave distributed file system oriented to storage of a large number of large files. Background technique [0002] With the development of new computing technologies, both business and personal data began to grow rapidly. The growth of massive data brings not only the problem of storage capacity, but also brings challenges to data management and storage performance, which has become the core problem that needs to be solved in the era of cloud computing. In order to ensure high availability, high reliability and economy of data, cloud computing uses distributed storage to store data, and redundant storage to ensure data reliability. In order to meet the needs of a large number of users, the storage technology of cloud computing must have high throughput and high transfer rate. For the data storage problem of cloud computing, industry and academia have proposed a var...

AI Technical Summary

Problems solved by technology

[0011] The technical problem to be solved by the present invention is: the general master-slave distributed file system can store files with a large data scale, and has high fault tolerance and high scalability, but it will cause some problems when used to store a large number of small files: (1) Centralized metadata service The master-slave distributed file system has only a single metadata node, and the number of files determines the size of the metadata. A large number of small files will consume the memory of the metadata node, and its metadata will exhaust the memory of the metadata node. memory beyond the limits of computer hardware

Images