Method for utilizing network storage data

Abstract

The invention provides a method for utilizing network storage data. The method includes the following steps: hashing is performed on IDs of data servers in a cloud platform storage system according to a predefined hash algorithm, and each data server is distributed on a ring of a whole Hash value range; each data server periodically sends a heartbeat message to a host node, and the host node manages topological information of a cloud platform; and a client accesses the host node to obtain the topological information of the cloud platform and cache locally, performs hashing calculation on file names, determines the data server where a corresponding small file is and performs reading or writing operation. The invention proposes a novel small-file-oriented cloud platform storage system and storage method, reduces the number of times of network connection for continuously reading and writing small files, storage is relatively lightweight, delay is relatively short, the host node load is reduced, service restoration is relatively fast after a failure, and availability of the whole storage system is improved.

Description

technical field [0001] The invention relates to data storage, in particular to a remote storage method. Background technique [0002] In recent years, network services need to store a large number of small files, such as pictures, emails, e-books, music files, Weibo text content, etc. At present, the storage and processing systems for massive large files mainly store metadata nodes and file data nodes. When applied to massive small files, they encounter many problems: first, massive small files bring a lot of metadata. Because the metadata information of each directory and file is stored in the memory of the name node, if there are a large number of small files in the system, it will undoubtedly reduce the storage efficiency and storage capacity of the entire cloud platform storage system. Second, accessing a large number of small files is much slower than accessing a few large files of the same capacity. Because if you access a large number of small files, you need to con...

AI Technical Summary

Problems solved by technology

At present, the storage and processing system for massive large files mainly stores metadata nodes and file data nodes, when applied to massive small files, there are many problems: First, massive small files bring a lot of metadata

However, it needs to use multiple service cloud platform units to complete the storage file function, so the reading and writing process is cumbersome, and multiple network connections need to be established for one reading and writing

At the same time, a high-load name server is also used, which makes the availability of the existing small file system still low

[0003] Therefore, for the above-mentioned problems existing in related technologies, effective solutions have not yet been proposed

Images