File writing method and device in a distributed system

Abstract

A file write-in method in a distributed system and an apparatus thereof are disclosed. The file write-in method comprises the following steps that after receiving a write request for a first file, a client sends a data write-in request of the first file to each storage node corresponding to the first file respectively, wherein the data write-in request at least includes a write request identification corresponding to the first file data write-in request and a storage node address; the storage node address includes an address of a main storage node and / or an address of a secondary storage node in each storage node corresponding to the first file; the storage node address is used for interaction between the main storage node and the secondary storage node; and the client receives an execution result of the data write-in request from each storage node corresponding to the first file respectively. In the invention, under the condition that multiple copy byte orders of the data can be guaranteed to be consistent, data sending hop counts are reduced so that write-in time delay is decreased.

Description

technical field [0001] The invention relates to the field of distributed storage, in particular to a method and device for writing files in a distributed system. Background technique [0002] The underlying distributed storage system is the basis of cloud computing technology, such as Google File System (Google FileSystem, GFS), Hadoop Distributed File System (Hadoop Distributed File System, HDFS), etc., based on which many different functions are established. Distributed storage products, a large part of which realize the distributed storage of data by establishing multiple partitions (Partition), and the data of each Partition is served by a separate process to ensure consistency. However, this design makes storage products need to solve the hotspot problem of Partition, because the service capability of a single process is always limited by the physical resources of the machine where it resides, such as the central processing unit (Central Processing Unit, CPU), network b...

AI Technical Summary

Problems solved by technology

[0007] When the data storage layer completes the write operation, it adopts a chained data flow, and the data to be written has to go through multiple hops to reach all data nodes, such as figure 1 In the system shown, when there are two secondary expansion nodes, the data to be written is first sent from the client to the primary expansion node (the first hop), and then sent from the primary expansion node to the first secondary expansion node (the second hop). hop), and finally sent from the first sub-extension node to the second sub-extension node (the third hop), a total of 3 hops have passed; correspondingly, the execution result of the second sub-extension node also needs to go through the first The secondary expansion node and the main expansion node return to the client; for the client, the total delay is the sum of 3 hop delays; when the amount of data written each time is large, too many hops will lead to long delay

Images