Data processing method and device applied to distributed system

Abstract

The present invention provides a data processing method and apparatus applied to a distributed system. The method comprises: performing erasure code coding on written file data, and generating redundant data of the file data; storing the file data and the redundant data into a first memory; transferring redundant data of file data with access popularity lower than a preset value into a second memory; and when it is required to read data information corresponding to the file data, directly reading the file data from the first memory, to obtain the data information. According to the data processing method of the present invention, a response speed of access data can be effectively ensured, and storage costs and maintenance costs are significantly saved.

Description

technical field [0001] The present invention relates to the field of computers, in particular to a data processing method and device applied to a distributed system. Background technique [0002] A distributed file system generally includes a client, a metadata server and a data server. The client is responsible for formulating the access interface of the file data, the metadata server handles the layout and attributes of the file, and the data server stores the data content of the file. [0003] For a distributed file system, it can store massive amounts of data and has high reliability. When a large number of files are stored in the system, a large amount of disk storage is required, but the cost of disks is much higher than that of tape libraries. Therefore, use Tape libraries and disks are necessary to store data hierarchically. [0004] The traditional method is to directly store all the data of the files with less access times in a period of time on the tape library, ...

AI Technical Summary

Problems solved by technology

[0004] The traditional method is to directly store all the data of the files with less access times in a period of time on the tape library, and the files with more access times are stored on the sata disk. The disadvantage of this is that when the files that need to be accessed are located in the tape library When uploading, the access speed becomes much slower than the sata disk, the user experience is poor, and multiple accesses to the tape library will also cause accelerated damage to the tape library, resulting in files that cannot be repaired

Images