File random writing method and system suitable for distributed file system

Abstract

The invention provides a file random writing method and system suitable for a distributed file system. The method comprises the following steps: 1, updating a writing interface of an HDFS to support arandom writing mode, and enabling a random writing operation to only affect a directly modified block or a limited adjacent block; 2, an HDFS data transmission protocol is expanded, so that a clientcan access any data block; and step 3, processing an update server data packet, calculating an update check value, obtaining data copy update, and updating the check value. In the present invention, when the data content of the file stored in the distributed file system is updated; according to the method and the system, only the current data block and the adjacent data block are influenced, a large amount of unnecessary data transmission caused by writing back the whole file like a native HDFS (Hadoop Distributed File System) is avoided, the data transmission and movement are effectively reduced, the network bandwidth pressure is relieved, and the processing resource consumption of a client and a server is reduced.

Description

technical field [0001] The invention relates to the field of distributed storage, in particular to a file random writing method and system applicable to a distributed file system. Background technique [0002] Distributed file storage systems are influenced by traditional disk file systems, most of which use fixed-size data blocks to organize and manage files. The currently popular distributed file systems HDFS (Hadoop Distributed File System) and GFS (Google File System) both divide large files into fixed-size blocks for storage, usually 64MB. After these files are created, most of them are appended to the end of the file, and almost no random write operations are involved. This fixed-length block design is not suitable for random writing, because the writing overhead is high and the performance is poor. But usually about 25% of the user's file operations are random writes. Aiming at the above status quo, a HDFS-based file random writing method is proposed, which can wri...

AI Technical Summary

Problems solved by technology

This patent does not have a file random writing method suitable for distributed file systems

Images