Storage optimization method for Hadoop distributed file system

Abstract

The invention discloses a storage optimization method for a Hadoop distributed file system, and the method specifically comprises the following steps: firstly, selecting an INFO-level log file which comprises a specific execution timestamp and file name information, and obtaining an access record and a deletion record of the INFO-level log file; extracting and sorting all information containing keywords in the IFNO level log, and then sorting and numbering according to timestamps; then determining feature tags, selecting features, constructing feature vectors, and forming a sample set of a training file elimination model; selecting three feature values of the feature vector as three classification nodes of a decision tree in sequence, establishing the decision tree by adopting an ID3 algorithm, and constructing a file elimination model by the decision tree; and finally, predicting the reusability of the file by using the established file elimination model. According to the method, the storage efficiency of the distributed file system is optimized, the data storage scale is reduced, and the storage efficiency of the HDFS is improved.

Description

technical field [0001] The invention belongs to the technical field of data storage, and in particular relates to a storage optimization method for a Hadoop distributed file system. Background technique [0002] With the increasing application of big data computing engines (such as Apache Hadoop and Apache Spark), a large amount of new data needs to be stored in the Hadoop distributed file system HDFS, which puts a lot of pressure on HDFS storage. The traditional method continues to expand the capacity of HDFS by increasing hardware investment, so as to store massively increased data, but the cost of doing so is high, and most of the data stored in HDFS has low utilization value, and the probability of being used or accessed by other devices Low, wasting a lot of hardware resources and software costs. [0003] In the era of cloud computing, the problem of storage optimization for large-scale distributed file systems is getting more and more attention. For example, Kirsten ...

AI Technical Summary

Problems solved by technology

However, this method mainly eliminates and deletes duplicate data in the file system. However, in a real distributed file system, the proportion of duplicate data is not large, and the proportion of stored data is large because a large number of them have only been used once or several times. low-value data, and the above method cannot effectively identify and delete a large amount of low-reuse data in distributed storage systems, especially HDFS

Images