Method for accelerating data reading of HDFS (Hadoop Distributed File System) based on large IO pre-reading

Abstract

The invention discloses a method for accelerating data reading of an HDFS (Hadoop Distributed File System) based on large IO (Input / Output) pre-reading, which belongs to the field of big data storage and comprises two parts, namely a client Clien and a data node DracosNode, the method comprises the following steps of: introducing a DracosNode to replace a data reading function of a DataNode; according to the method, the DracosNode server interacts with the disk through a fixed reading thread, the data is pre-read at the DracosNode server, the amount of data read each time is large, and the phenomenon of IO preemption of the disk is improved; when a client request arrives, the DracosNode only needs to return the 1MB data read in advance, and the IO wait phenomenon of the CPU is also greatly improved; and meanwhile, the throughput of the disk is also improved, so that the reading performance of the DataNode is integrally accelerated.

Description

Technical Field

[0001] The present invention belongs to the field of big data storage, and particularly relates to a method for accelerating HDFS data reading based on large I / O read-ahead. Background Art

[0002] The Hadoop Distributed File System HDFS (Hadoop Distributed File System) is in the form of a master / backup / slave. A Hadoop cluster consists of two NameNodes (name nodes) and several DataNodes (data nodes). The NameNode has a management role and is responsible for operating the namespace of the file system. It is responsible for maintaining the mapping from file paths to data blocks, the mapping from data blocks to DataNodes, etc. The DataNode is responsible for handling read and write requests from clients. It creates, deletes, and replicates data blocks under the unified scheduling of the NameNode.

[0003] When a Client reads data, it needs to establish a data channel with the DataNode. When the concurrency is high, the phenomenon of multiple threads competing for disk I / O is obvious. The disk is in the IO Busy state for a long time, but the overall throughput of the disk is relatively low. At the same time, due to the busy disk I / O, the CPU has a high IO wait, further exacerbating the performance burden. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for accelerating HDFS data reading based on large I / O read-ahead in view of the deficiencies of the background art.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] A method for accelerating HDFS data reading based on large I / O read-ahead includes two parts: the client Client and the data node DracosNode;

[0007] Among them, the client Client is a program and component for reading data from HDFS;

[0008] The data node DracosNode is used to cooperate with the DataNode and replace the process of the DataNode's read task;

[0009] Based on the principle understanding and role situation of DracosNode, the specific steps of the data reading process are as follows:

[0010] Step 1: The DracosNode service starts and initializes the data transfer framework FlightServer(), the file reading thread pool, and the read plan readplan scheduler;

[0011] Step 2: The Client initiates a flight link request to the DracosNode and brings the read plan readplan for this read task; among them, Flight is the transmission framework for data interaction between the Java client and the C++ server;

[0012] Step 3: After the DracosNode obtains the read plan, it schedules and processes the readplan, and updates and manages the offset and length attributes of the readplan as the data reading progresses;

[0013] Step 4: After the scheduling task discovers an available readplan, it starts a read thread task through the file reading thread pool to read the file;

[0014] Step 5: A cache Cache queue is constructed at the DracosNode end for each file to store the buffer object buffer;

[0015] Step 6: The client initiates a read request to the DracosNode through the Flight framework;

[0016] Step 7: The DracosNode looks for the pre-read buf of the file in the Cache queue. If there is one, it returns directly; if not, it reads in a blocking manner;

[0017] Step 8: The DracosNode delivers the buf to the Client end through a Flight request.

[0018] As a further preferred solution of the method for accelerating HDFS data reading based on large IO pre-reading of the present invention, the client Client is implemented in Java.

[0019] As a further preferred solution of the method for accelerating HDFS data reading based on large IO pre-reading of the present invention, the data node DracosNode is implemented in C++.

[0020] As a further preferred solution of the method for accelerating HDFS data reading based on large IO pre-reading of the present invention, in Step 2, the read plan readplan includes the file name, offset, and length information to be read.

[0021] As a further preferred solution of the method for accelerating HDFS data reading based on large IO pre-reading of the present invention, in Step 4, the fixed size of the buf read each time is 1MB.

[0022] A method for accelerating HDFS data reading based on large IO read-ahead, as described in claim 1, is characterized in that: in step 5, a file can store at most 5 buffers.

[0023] Compared with the prior art by adopting the above technical solution, the present invention has the following technical effects:

[0024] The method for accelerating HDFS data reading based on large IO read-ahead in the present invention introduces DracosNode to replace the data reading function of DataNode; the DracosNode server interacts with the disk with fixed reading threads, and pre-reads data on the DracosNode server, and the amount of data read each time is relatively large, improving the disk IO preemption phenomenon; when a client request arrives, DracosNode only needs to return the pre-read 1MB data, and the IO wait phenomenon of the CPU is also greatly improved; at the same time, the throughput of the disk is also increased, thereby overall accelerating the reading performance of DataNode. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other embodiments can also be obtained based on these drawings.

[0026] Figure 1 is the overall structure diagram of the method for accelerating HDFS data reading based on large IO read-ahead in the present invention;

[0027] Figure 2 is the IO wait situation of the node CPU during the DataNode reading process in the present invention;

[0028] Figure 3 is the IO wait situation of the node CPU during the DracosNode reading process in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] The following further elaborates on the technical solutions of the present invention with reference to the drawings:

[0030] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. The present invention will be described in detail below according to the accompanying drawings and preferred embodiments, and the purpose and effect of the present invention will become clearer. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0032] In most scenarios, the performance of DataNode in dealing with high-concurrency read scenarios is poor, which is also the main reason affecting the read performance of HDFS. To solve this problem, the present invention proposes a method for accelerating HDFS read data based on large IO prefetching, which mainly consists of two parts: Client (client) and DracosNode (data node). The overall structure diagram is as Figure 1 shown.

[0033] Client: A program and component (implemented in Java) for reading data from HDFS.

[0034] DracosNode: A process (implemented in C++) that combines with DataNode and replaces the DataNode read task.

[0035] Flight: A transmission framework for data interaction between Java clients and C++ servers.

[0036] Based on the understanding of the principle of DracosNode and its role, the specific steps of the data reading process are as follows:

[0037] Step 1, the racosNode service starts, and initializes the FlightServer (data transmission framework), the read file thread pool, and the read plan (readplan) scheduler;

[0038] Step 2, the Client sends a flight link request to the DracosNode, and brings the read plan readplan for this read task. The readplan contains information such as the file name, offset, and length of this read;

[0039] Step 3: After DracosNode obtains the read plan, it schedules the read plan and updates the offset and length attributes of the read plan as the data reading progresses.

[0040] Step 4: After the scheduling task finds an available readplan, it starts the reading thread task through the file reading thread pool to read the file, reading a fixed-size buf each time, which is 1MB by default.

[0041] Step 5: For each file, a Cache queue is constructed on the DracosNode side to store the buffer.

[0042] Step 6: The client initiates a read request to DracosNode through the Flight framework.

[0043] Step 7: DracosNode searches for the pre-read buf of the file in the cache queue. If it exists, it returns directly. If not, it performs blocking reading.

[0044] Step 8: DracosNode sends buf to the Client through Flight request.

[0045] A method based on large I / O pre-reading to accelerate HDFS data reads. DracosNode autonomously pre-reads data and manages read plan updates. Upon receiving a client request, data is directly retrieved from the DracosNode cache queue, just like a read cache. Overall, this method can be considered as converting the original synchronous channel into an asynchronous channel, thereby improving read efficiency. Table 1 compares the performance of HDFS native DataNode reads and DracosNode reads:

[0046] Table 1

[0047]

[0048] Table 1 shows that when data is not cached, DracosNode is 10% faster than native HDFS. When all data is cached, DracosNode is 10% slower than native HDFS (the full cache avoids the biggest disadvantage of DataNode's interaction with disks).

[0049] DracosNode significantly optimizes the CPU IO wait phenomenon. Figure 2 It is the CPU usage of DataNode when reading natively. Figure 3 This is the CPU usage when DracosNode is reading.

[0050] The present invention introduces DracosNode to replace the data reading function of DataNode. The DracosNode server interacts with the disk with a fixed number of reading threads (default 2), pre-reads data on the DracosNode server, and reads a relatively large amount of data each time (default 1MB), which improves the disk I / O preemption phenomenon. When a client request arrives, DracosNode only needs to return the pre-read 1MB of data, and the I / O wait phenomenon of the CPU is also greatly improved. At the same time, the throughput of the disk is also increased, thus overall accelerating the reading performance of DataNode.

[0051] Those of ordinary skill in the art can understand that the above are only preferred examples of the invention and are not used to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, for those skilled in the art, they can still modify the technical solutions described in the foregoing examples, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, etc. made within the spirit and principle of the invention shall be included within the protection scope of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0052] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for accelerating HDFS data reading based on large IO read-ahead, characterized in that: It consists of two parts: the client Client and the data node DracosNode; Among them, the client Client is a program and component for reading data from HDFS; The data node DracosNode is a process that combines with the DataNode and replaces the DataNode's read task; Based on the understanding of the DracosNode principle and the role situation, the specific steps of the data reading process are as follows: Step 1, the DracosNode service starts, and initializes the data transmission framework FlightServer, the file reading thread pool, and the read plan readplan scheduler; Step 2, the Client initiates a flight link request to the DracosNode and brings the read plan readplan for this reading task; among them, Flight is a transmission framework for data interaction between the Java client and the C++ server; Step 3, after the DracosNode obtains the read plan, it schedules and processes the readplan, and updates and manages the offset and length attributes of the readplan as the data reading progress; Step 4, after the scheduling task discovers an available readplan, it starts a read thread task through the file reading thread pool to read the file; Step 5, a cache Cache queue is constructed for each file at the DracosNode end to store the buffer object buffer; Step 6, the client initiates a read request to the DracosNode through the Flight framework; Step 7, the DracosNode searches for the pre-read buf of the file in the Cache queue. If there is one, it returns directly. If not, it reads in a blocking manner; Step 8, the DracosNode delivers the buf to the Client end through a Flight request.

2. The method for accelerating HDFS data reading based on large IO prefetching according to claim 1, wherein: The client Client is implemented in Java.

3. A method for accelerating HDFS read data based on large IO prefetching according to claim 1, characterized in that: The data node DracosNode is implemented in C++.

4. A method for accelerating HDFS read data based on large IO prefetching according to claim 1, characterized in that: In step 2, the read plan readplan includes the file name, offset, and length information to be read.

5. A method for accelerating HDFS read data based on large IO prefetching according to claim 1, characterized in that: In step 4, the fixed size of buf read each time is 1MB.

6. A method for accelerating HDFS read data based on large IO prefetching according to claim 1, characterized in that: In step 5, a file can store at most 5 buf.

Images