File management method and device

Abstract

The invention discloses a file management method and device, and belongs to the field of distributed technology. The method includes: acquiring a text length value of text data matching an to-be-stored audio-frequency file; determining a first index file in at least one created index file according to the text length value, wherein the first index file is used for storing index data of audio-frequency files matching the text length value; writing a hash identifier of the text data into the first index file; and writing the audio-frequency file into a data file corresponding to the first indexfile, and writing a storage address of the audio-frequency file into the first index file. According to the method, storage management, based on the text length value and the hash identifier of the text data, on the audio-frequency file is realized through dividing the index files according to text length values of text data and carrying out hash operating on the text data of the audio-frequency file, programming operations are convenient, distributed accessing is supported, and file processing efficiency is higher.

Description

technical field [0001] The invention relates to the field of distributed technology, in particular to a file management method and device. Background technique [0002] Today, with the rapid development of Internet technology, more and more information is processed digitally, which leads to geometric growth of small files such as pictures, emails, e-books, and Internet files. For example, a large social network provides as much as 60TB of image data per week alone. In the face of a large number of small files, in order to ensure the disk access efficiency of the server, how to safely and effectively manage the large number of small files has become an important issue that those skilled in the art must face. [0003] In HDFS (Hadoop Distributed File System, Hadoop Distributed File System), file archiving technology builds a hierarchical file system on HDFS, and manages storage by packaging small files into HAR files to reduce the number of files in HDFS. Quantity; and Seque...

AI Technical Summary

Problems solved by technology

[0005] Hadoop is limited by its own architectural factors, and its efficiency is low when processing small files, and programming operations are inconvenient, which limits its implementation only in java.

For ReiserFS, it does not support distributed access, and due to its own architectural factors, the processing effect is poor when the number of small files is too large

Images