Small file storing and positioning method and system

Abstract

The invention discloses a small file storing and positioning method and system. The method comprises the steps that the name of a small file is subjected to Hash calculation, the number of a stored large file corresponding to the small file is positioned; according to the number of the stored large file, magnetic disks of all copies of the small file in the stored large file are positioned by using a consistent Hash method; according to index files on the magnetic disks, the offset of the small file in the stored large file is determined, and accordingly the storing position of the small file is determined. Central element data serving nodes are removed in a small file storing system, positioning of the multiple copies of the small file is achieved in a decentralized framework, the problems about single-point faults and expandability are solved, and meanwhile, extra position query requests are removed.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a method and system for storing and locating small files. Background technique [0002] The rapid development of the Internet has produced a large number of pictures, documents and other files. These files are characterized by small size (generally below 100KB) and huge number (hundreds of millions). The traditional POSIX interface file system has been difficult to meet the demand. This is the well-known problem of massive small files in the industry. [0003] For the problem of massive small files, the current common practice is to store small files in a traditional POSIX file. Since small files are stored on the disk in the form of combined storage, to locate a small file, in addition to the path of the physical large file , but also includes the offset of the small file in the large file, so simply using the traditional common hash method will not be able to locate the p...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to propose a small file storage positioning method and system to solve the need for the central metadata server to provide positioning services in the small file storage system in the prior art, and in order to ensure the availability and fault tolerance of the system, it is necessary to Files are saved in multiple copies, causing the client to interact with the server at least twice every time it reads and writes. The first time is for large file positioning, and the second time is for actual data reading and writing; and the cluster may There is a single point of failure and poor scalability

Images