A distributed data access method, device and system

Abstract

The invention discloses a method, a device and a system for distributed data access. The method for distributed data access comprises the following steps: a small file processing module receives a file uploaded by a client; if determining that the data volume of the file is smaller than a preset data volume threshold, the file is saved in a local system; when the sum of the data volumes of files saved in the local system reaches a preset mergence threshold, the files are merged, and the large merged file is transmitted to a Hadoop distributed file system (HDFS). The method for distributed data access is capable of saving the occupied internal memory of a main node when lots of small files are stored, and improving the data access efficiency of the HDFS.

Description

technical field

[0001] The present invention relates to data access technology, in particular to a distributed data access method, device and system. Background technique

[0002] In the current distributed storage system, Hadoop is generally used as a storage technology, and Hadoop is an open source distributed system infrastructure. Each file stored in Hadoop Distributed File System (HDFS) needs to correspond to a block (Block), and the master node (NameNode) in HDFS needs to establish a cache mapping for each file and its corresponding block. The more files stored in HDFS, the more cache maps need to be established, and the more memory of the master node is occupied.

[0003] Therefore, HDFS is only suitable for storing large files (such as files with a data volume greater than 64M), when HDFS stores a large number of small files (such as files with a data volume of less than 64M), such as pictures, documents, etc. When storing files, a large number of cache maps need t...

Examples

Embodiment Construction

[0051] A distributed data storage method provided by the present invention, such as figure 1 As shown, the method includes:

[0052] Step 101, the small file processing module receives the file uploaded by the client;

[0053] Here, the small file processing module is a device added between the client and HDFS, and the small file processing module can process small file merging asynchronously.

[0054] Step 102, when it is determined that the data volume of the file is less than a preset data volume threshold, save the file in the local system;

[0055] Here, the data volume threshold may be 64M, or it may be set according to actual conditions.

[0056] The local system refers to a storage space other than HDFS.

[0057] Step 103, when the sum of the data volumes of the files stored in the local system reaches the preset merging threshold, the files are merged, and the merged large file is sent to HDFS.

[0058] Here, the merging threshold may be 64M, or it may be set acco...

technical field

[0001] The present invention relates to data access technology, in particular to a distributed data access method, device and system. Background technique

[0002] In the current distributed storage system, Hadoop is generally used as a storage technology, and Hadoop is an open source distributed system infrastructure. Each file stored in Hadoop Distributed File System (HDFS) needs to correspond to a block (Block), and the master node (NameNode) in HDFS needs to establish a cache mapping for each file and its corresponding block. The more files stored in HDFS, the more cache maps need to be established, and the more memory of the master node is occupied.

[0003] Therefore, HDFS is only suitable for storing large files (such as files with a data volume greater than 64M), when HDFS stores a large number of small files (such as files with a data volume of less than 64M), such as pictures, documents, etc. When storing files, a large number of cache maps need t...

Examples

Embodiment Construction

[0051] A distributed data storage method provided by the present invention, such as figure 1 As shown, the method includes:

[0052] Step 101, the small file processing module receives the file uploaded by the client;

[0053] Here, the small file processing module is a device added between the client and HDFS, and the small file processing module can process small file merging asynchronously.

[0054] Step 102, when it is determined that the data volume of the file is less than a preset data volume threshold, save the file in the local system;

[0055] Here, the data volume threshold may be 64M, or it may be set according to actual conditions.

[0056] The local system refers to a storage space other than HDFS.

[0057] Step 103, when the sum of the data volumes of the files stored in the local system reaches the preset merging threshold, the files are merged, and the merged large file is sent to HDFS.

[0058] Here, the merging threshold may be 64M, or it may be set acco...

Images