A distributed data access method, device and system

Abstract

The invention discloses a method, a device and a system for distributed data access. The method for distributed data access comprises the following steps: a small file processing module receives a file uploaded by a client; if determining that the data volume of the file is smaller than a preset data volume threshold, the file is saved in a local system; when the sum of the data volumes of files saved in the local system reaches a preset mergence threshold, the files are merged, and the large merged file is transmitted to a Hadoop distributed file system (HDFS). The method for distributed data access is capable of saving the occupied internal memory of a main node when lots of small files are stored, and improving the data access efficiency of the HDFS.

Description

technical field [0001] The present invention relates to data access technology, in particular to a distributed data access method, device and system. Background technique [0002] In the current distributed storage system, Hadoop is generally used as a storage technology, and Hadoop is an open source distributed system infrastructure. Each file stored in Hadoop Distributed File System (HDFS) needs to correspond to a block (Block), and the master node (NameNode) in HDFS needs to establish a cache mapping for each file and its corresponding block. The more files stored in HDFS, the more cache maps need to be established, and the more memory of the master node is occupied. [0003] Therefore, HDFS is only suitable for storing large files (such as files with a data volume greater than 64M), when HDFS stores a large number of small files (such as files with a data volume of less than 64M), such as pictures, documents, etc. When storing files, a large number of cache maps need t...

AI Technical Summary

Problems solved by technology

[0003] Therefore, HDFS is only suitable for storing large files (such as files with a data volume greater than 64M), when HDFS stores a large number of small files (such as files with a data volume of less than 64M), such as pictures, documents, etc. When storing files, a large number of cache maps need to be established, which will greatly occupy the memory of the master node, resulting in a greatly reduced efficiency of HDFS access data

Images