Data processing method, client, node server and distributed file system

Abstract

Embodiments of the invention disclose a data processing method. The method comprises the following steps of: receiving a predetermined data writing request sent by a client, wherein the predetermineddata writing request is a data writing request which satisfies a first predetermined condition in more than one data writing request received by the client; and executing a corresponding predetermineddata writing operation according to the predetermined data writing request and string corresponding written data. The embodiments of the invention furthermore disclose the client, a node server and adistributed file system.

Description

technical field [0001] The invention relates to data processing technology, in particular to a data processing method, a client, a node server and a distributed file system. Background technique [0002] With the rapid development of information technology, massive amounts of information are used to store data and the investment is increasing. There is an urgent need for new storage solutions to change the status quo, save storage costs, and reduce storage investment. Cloud storage has emerged as the times require. Hadoop is an open source project of Apache, the purpose is to establish a stable and scalable distributed computing architecture running on the basis of cheap hardware devices. Among them, the Hadoop file system (HDFS, Hadoop Distributed File System) is one of the Hadoop sub-projects. As an open source implementation of Google's distributed file system (GFS, Google File System), HDFS is a cloud storage solution for major institutions and companies. The scheme pro...

AI Technical Summary

Problems solved by technology

[0007] However, the inventor has found through in-depth research that the following problems still exist in the prior art 1: in the prior art 1, an implementation method of letting the DataNode node cache some small file metadata information is adopted. Although this method can at least partially solve the small file writing problem, the The confusion of mechanism and strategy has severely damaged the semantics of the HDFS system, which will inevitably have a huge impact on other operations such as sequential writing and large file processing, and will inevitably seriously affect the original intention of the HDFS system design

[0011] The disadvantage of prior art 3 is that the file threshold used is 16M, and the file granularity is too large, which will inevitably lead to a large number of file mergers during the processing, resulting in a relatively high I / O write operation time delay in some write operations, which is not suitable for High I / O performance data access requirements; moreover, this method still adopts sequential write operations, and cannot realize random write operations; finally, this solution adds machines outside the HDFS system, which has actually destroyed the interface semantics of the HDFS system. The interface relationship between the application system and the HDFS system has been changed, and there is no universality

Images