Data storage and reading method based on HDFS system and device of data storage and reading method

Abstract

The invention discloses a data storage and reading method based on an HDFS system and a device of the data storage and reading method. The method comprises the steps of receiving a data stream of a file to be stored and encapsulating the data stream into a data package; if the file to be stored is a big file, aiming at each data package, storing the data package into a data block of a data storage node which is allocated by a name service node for the data package, and caching the data package into a data block of an encoder; establishing the same index for every N data blocks in which data packages are cached, wherein aiming at the same data package, the index of the data block of the data storage node in which the data package is cached is identical to that of the data block of the encoder; encoding N data blocks of the encoder with the same index to obtain M verification encoding blocks corresponding to the index, wherein N and M are positive integers, and M is smaller than N; determining and storing the data block in which the verification encoding block is stored on the data storage node. The data storage and reading method based on the HDFS system and the device of the data storage and reading method are used for solving the problem that a traditional HDFS system is big in occupied storage volume and high in cost when the data amount is big.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a method for storing and reading data based on an HDFS system and a device thereof. Background technique [0002] In HDFS (Hadoop Distributed File System), n-way replication is generally used for data redundancy. The file system manages file content in units of data blocks (Block), and a file is divided and saved on several Blocks. When an application writes a file, every time a block is written, HDFS will automatically copy the data of the block to other backup servers to ensure that each block has multiple copies (the default value is 3), even if two servers are down , the data is still accessible, the backup method is as follows figure 1 shown. [0003] Such as figure 2 As shown, a typical HDFS cluster consists of a Namenode (name service node), a Secondary Namenode (standby name service node) and several Datanodes (data storage nodes). Namenode provides metadata servi...

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present invention provides a method and device for storing and reading data based on the HDFS system to solve the problems of large storage capacity and high cost when the traditional HDFS system has a large amount of data

Images