File reading-writing method for distributed file system

A technology of distributed files, read and write methods, applied in the file system, transmission system, file access structure, etc., can solve the problem of reducing the efficiency of client access to files, increasing the access pressure of the metadata server, and the client reading and writing data server, etc. problem, to achieve the effect of reducing communication overhead, fast file reading and writing, and reducing burst load

CN107832423AActive Publication Date: 2018-03-23SUN YAT SEN UNIV +1
4 Cites 9 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2018-03-23

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
Patent Text Reader

Abstract

The invention discloses a file reading-writing method for a distributed file system. The file reading-writing method includes the following steps: in file reading, a file reading IO path of a client-ametadata data server-a data server-a client is adopted; in file writing, the client acquires the number of files to be written on which file writing needs to be performed; if the number of the file to be written exceeds a preset threshold, a high performance computing scene is determined, file writing is performed for mass threads in the high performance computing scene at the same time, a strategy that data is written firstly and metadata is created secondly is adopted to reduce emergency load of a metadata server; otherwise, a file write-in IO path of a client-a data server-a metadata server-a client is adopted for each target file to be written. The file reading-writing method is fast in file reading-writing speed, is high in efficiency, can reduce the number of times of interactions between the client and the metadata server, and can reduce the communication cost.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of distributed storage systems, in particular to a method for reading and writing files used in a distributed file system. Background technique

[0002] With the popularization and deepening of big data applications, the basic computing framework poses higher challenges to the storage system in terms of scale and performance requirements. High-performance computers have higher and higher requirements for the performance of distributed file systems. In the application scenarios of frequent creation and deletion of massive small files and large-scale concurrent I / O operations, the read and write efficiency of file systems has become a key factor limiting its performance. For example, for applications such as health big data, traffic big data, and financial big data, the amount of data is usually in the order of TB, PB or even EB, so a large amount of storage resources are required to store and manage these data. In add...

Examples

Embodiment Construction

[0029] Such as figure 1 As shown, the implementation steps of the file reading and writing method file reading for the distributed file system in this embodiment include:

[0030] A1) The client communicates with the metadata server of the distributed file system to send a request to read the file;

[0031] A2) After receiving the client's request, the metadata server returns the communication query metadata information to the client, and sends the client request information and communication address to the data server where the file block of the read file is located, and the client returns according to the metadata server Information to find the data server where the file block of the read file is located;

[0032] A3) After the data server receives the client request information and communication address, it establishes a connection with the client and starts sending the file block data of the read file to the client;

[0033] A4) The client receives data in units of file ...