A method for supporting HDFS replica mode parallel disk landing

Abstract

The application relates to the computer technical field, and particularly discloses a method for supporting HDFS copy mode parallel disk landing, which first defines a client data parallel sending mode, constructs a group of senders in an OutPutStream, the number of the senders is consistent with the number of copies, that is, each copy has an independent data sender, each sender independently runs, does not interfere with each other, and parallelly sends data; the sender comprises a to-be-processed data queue module, each sender contains a data queue, the data queue saves to-be-sent data and data which has been sent but has not received ACK, and the like; the application can greatly improve the response rate of file writing, so that the data disk landing efficiency can be ensured during the writing process, the capacity of HDFS as a bottom file system is improved, and the HDFS writing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer, in particular to a method for supporting HDFS copy mode parallel disk landing. BACKGROUND

[0002] The data landing ability provided by the file system is crucial when writing a file, and the landing operation can ensure that the written file data has been safely written to the disk, and on this basis, the response time of the system is also a very important indicator of system efficiency. In the scenario of massive data and high concurrency, the time of each interaction also agrees to affect the throughput of the file system. This is very important for improving the performance of the file system.

[0003] In order to ensure the high availability of data, HDFS usually adopts multiple copy mode to realize this function, that is, for the same data, it needs to be saved in the same format on different data nodes (DataNode). Even if it is not readable because of some exceptions, it does not affect the normal use of the entire HDFS file. By default, the number of copies is 3.

[0004] HDFS writes multiple copies through the PipeLine mode.

[0005] PipeLine, that is, the data path of the HDFS client transmitting data (Packet) to the data node (DataNode) and receiving the reply (ACK) of the DataNode. The entire PipeLine is composed of a client and several DataNodes in series, and the data flows from the client to the DataNode. In the PipeLine, if DataNodeA is closer to the PipeLine than DataNodeB, then A is in the upstream of B (Upstream), and B is in the downstream of A (Downstream).

[0006] Due to the structural characteristics of PipeLine, it is necessary to receive the data packet in each DN before returning the ACK in turn, so there is a large unstable factor in the entire landing process, and the landing ability of the entire file depends on the slowest DN node. And the requirement for the internal network condition of DN is relatively strict. SUMMARY

[0007] The application aims to provide a method for supporting HDFS copy mode parallel disk landing, aiming to improve the data writing efficiency, and solve the problem of affecting the writing speed due to the presence of a short board node. The application is aimed at the problem of unstable disk landing capacity of HDFS multi-copy PipeLine writing mode and the restriction of short board node. The application is a method for efficient parallel disk landing in the scenario of sufficient Client bandwidth, which can reduce the dependence on network conditions and the limitation of node short board in the multi-copy writing process, and is not perceived by the upper application, and can improve the multi-copy writing data efficiency.

[0008] To achieve the above object, the application provides the following technical scheme.

[0009] A method for supporting HDFS copy mode parallel disk landing, the method comprising the following contents:

[0010] Firstly, a client data parallel sending mode is defined, a group of senders are constructed in the OutPutStream, the number of senders is consistent with the number of copies, that is, each copy has an independent data sender, each sender runs independently and does not interfere with each other, and data is sent in parallel;

[0011] The sender comprises:

[0012] A to-be-processed data queue module, each sender contains a data queue, which saves to-be-sent data and data that has been sent but has not received ACK, when receiving ACK, the related Packet in the data queue is cleared, indicating that the Packet data has been processed;

[0013] An InnerSender module for sending Packet data packets and command packets;

[0014] A Responder module for receiving Ack data packets;

[0015] A sender control module for controlling the type of packet sent by InnerSender.

[0016] As a preferred embodiment of the application, the data queue adopts a RingBuffer mode, adopts a lock-free processing, and maintains three cursors inside: writeCursor, readCursor and clearCursor; wherein writeCursor is a client writing position, readCursor is a sender reading position maintained by the Sender module, and clearCursor is a cleaning position, which is moved when the Responder receives Ack; all the three flags are incremented, and satisfy the following formula:

[0017] ClearCursor <= readCursor <= writeCursor.

[0018] As a preferred embodiment of the present application, the Cursor needs to be taken modulo operation to get the real index of the RingBuffer.

[0019] As a preferred embodiment of the present application, the InnerSender module user maintains the TCP connection with the DataNode, and sends data packets or control command packets to the DataNode through the connection, wherein the data packets are obtained from the data queue, and the control command packets are sent to the InnerSender module through the controller; after the data packets are sent, the readCursor in the Buffer is updated.

[0020] As a preferred embodiment of the present application, the Responder module maintains a copy of the TCP connection of the InnerSender DataNode, and receives the ACK of the data packet Packet through the connection; for the data receiving the ACK, it indicates that the data packet has been written to the disk, and the cache in the data queue is deleted by modifying the ClearCursor.

[0021] As a preferred embodiment of the present application, the data types sent by the Sender control module include data packets and command packets, wherein the main commands include: Create, Append, Recover, Sync, Close; such command packets are consistent with the commands used in the three-copy PipeLine writing mode.

[0022] As a preferred embodiment of the present application, the minimum copy mechanism and copy lag detection mechanism are used when calling the OutPutStream to perform the writing operation, that is, two parameters are configured:

[0023] The minimum number of copies, when the number of writing copies meets the configuration, it indicates that the data writing is successful;

[0024] The maximum number of lags, that is, when the lag of the data packet sent by a Sender reaches the configuration, the Sender is marked as abnormal; and the Sender is excluded from the SenderGroup.

[0025] As a preferred embodiment of the application, if an abnormal Sender is found during transmission, the Recover command is sent by the current normal Sender to update the version of the normal copy, so as to distinguish the abnormal copy from the copy corresponding to the abnormal Sender, and the data of the abnormal copy is expired.

[0026] As a preferred embodiment of the application, after the file writing is completed, the Close operation is performed on the SenderGroup, if the file meets the minimum number of copies but does not meet the expected number of copies, the file can be normally closed, and the subsequent NameNode performs the Reconstruction operation until the expected number of copies is met, if the minimum number of copies cannot be met when the Close operation is performed, it indicates that the file writing is abnormal, and the CloseException is thrown to the upper layer.

[0027] Compared with the prior art, the application has the beneficial effects that:

[0028] 1. The method solves the problem that the HDFS copy mode has too long link in the writing process, and the short board node and too heavy network dependence may occur.

[0029] 2. The method reduces the bandwidth between the DataNodes, can greatly simplify the data processing logic on the DataNode, and increases the stability of the DataNode.

[0030] 3. The method can greatly improve the response rate of the written file, so that the data landing efficiency in the writing process can be ensured, the ability of the HDFS as the underlying file system is improved, and the HDFS writing efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the application.

[0032] Figure 1 The OutPutStream structure diagram of the method for supporting the HDFS copy mode parallel landing of the application;

[0033] Figure 2 The sender structure diagram used in the method for supporting the HDFS copy mode parallel landing of the application;

[0034] Figure 3 The Cursor modulo operation schematic diagram of the method for supporting the HDFS copy mode parallel landing of the application. DETAILED DESCRIPTION

[0035] In order to make the technical problems to be solved by the present application, technical solutions and beneficial effects clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0036] Please refer to Figures 1-3 , the present application is to realize the above-mentioned purpose to the technical solutions described in the present application are described in detail.

[0037] Example one

[0038] In the study of HDFS multi-copy parallel writing problem, first define a client data parallel sending mode, that is, in the OutPutStream inside a group of senders are constructed, the number of senders is consistent with the number of copies, that is, each copy has an independent data sender (Sender), each sender runs independently, and does not interfere with each other, and sends data in parallel. The OutPutStream structure is shown in Figure 1

[0039] Each sender (Sender) contains four modules, and the relationship is shown in Figure 2

[0040] 1, the data queue (Buffer) to be processed.

[0041] Each sender contains a data queue, which saves the data to be sent and the data that has been sent but has not received ACK. When receiving ACK, the relevant Packet in the data queue is cleared, indicating that the Packet data has been processed.

[0042] The data queue adopts RingBuffer mode, adopts lock-free processing to ensure higher writing and reading efficiency, and maintains three input positions inside, readCursor is the Sender reading position, maintained by the Sender module, clearCursor is the cleaning position, which will move when the Responder receives Ack. The three flags are self-incrementing and satisfy the following formula: Cursor: writeCursor, readCursor, clearCursor. Among them, writeCursor is the client writing position, readCursor is the Sender reading position, and clearCursor is the Responder cleaning position.

[0043] ClearCursor<=readCursor<=writeCursor

[0044] In actual use, the Cursor needs to be taken modulo operation to obtain the real index of RingBuffer, as shown in Figure 3 ​​Shown:

[0045] 2、InnerSender module. Used to send Packet data packets and command packets

[0046] This module user maintains and DataNode between the TCP connection, and through this connection to DataNode send data packets, or control command packets, where the data packets from the data queue, control command packets are sent to the InnerSender module through the controller. Data packet transmission is complete, then update Buffer readCursor.

[0047] 3、Responder module, used to receive Ack data packets.

[0048] This module maintains a copy of the TCP connection between InnerSender DataNode, and through this connection to receive data packets Packet ACK, for the received ACK data, which indicates that the data packet has been on disk, by modifying ClearCursor delete data queue cache.

[0049] 4、Sender control module, to control InnerSender send packet type.

[0050] In general, InnerSender and DataNode between the TCP connection only one, data packets and command packets are sent through this TCP connection, so you need a control module for control, indicating that the current data packets or command packets are sent. The main commands include: Create, Append, Recover, Sync, Close. This type of command packet with three copies of the PipeLine write method adopted by the command is consistent.

[0051] Client in the call OutPutStream Write operation, you need to encapsulate the metadata into Packet, and this Packet is delivered to the three sender cache, followed by Sender asynchronous sending. Because of the load and environment of the data node exist differences, resulting in data parallel sending rate differences, that is, there are some data packets for some nodes, have received ACK, but in another node has not been on disk. We use the minimum copy mechanism and copy lag detection mechanism to ensure the efficiency of the write and redundancy. That is, we need to configure two parameters:

[0052] 1、Replica minimum number (dfs.namenode.replication.min), when the number of copies written to meet this configuration, which indicates that the data write success.

[0053] 2. Maximum replica lag (replica.packet.lag.max): When the lag of data packets sent by a Sender reaches this configuration, the Sender is marked as abnormal and removed from the SenderGroup.

[0054] Furthermore, if an abnormal Sender is detected during the transmission process, the currently normal Sender needs to send a Recover command to update the version of the normal replica, thus distinguishing it from the replica corresponding to the abnormal Sender and causing the data of the abnormal replica to expire.

[0055] Furthermore, after the file is written, a Close operation needs to be performed on the SenderGroup. If the file meets the minimum number of replicas but not the expected number of replicas, it can be closed normally, and the NameNode will then perform a Reconstruction operation until the expected number of replicas is met. If the minimum number of replicas cannot be met when closing, it indicates that the file writing is abnormal and a CloseException is thrown to the upper layer.

[0056] Based on the above design approach, we construct the following model for comparison:

[0057] Assumption:

[0058] 1. The network communication time between nodes is the same, and it is t1 for both.

[0059] 2. In Pipeline mode, multiple replicas are written to disk in parallel on different DataNodes, assuming the disk writing time is t2.

[0060] For a replica count of n, the write time is t. (n) The following formula exists:

[0061] t(n) = 2t1n + t2 (pileline method)

[0062] t(n) = 2t1 + t2 (parallel method)

[0063] As shown in the formulas above, when the number of replicas is 1, the write time is the same for both methods. The write time of the Pipeline method increases linearly with the number of replicas. The parallel method maintains a stable write time, independent of the number of replicas. Therefore, the parallel write method improves write efficiency and reduces network dependence.

[0064] The above description represents a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the principles and scope of the present invention should be included within the protection scope of the present invention.

[0065] It is to be understood that the terminology "including", "comprising", or any other variation thereof, is intended to cover a non-exclusive inclusion such that process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0066] The above merely describes the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, made according to the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A method for supporting HDFS replica pattern parallel destaging, characterized in that, The method comprises the following contents: First, a client data parallel sending mode is defined, a group of senders is constructed in the OutPutStream, the number of the senders is consistent with the number of the replicas, that is, each replica has an independent data sender, each sender is independently operated and does not interfere with each other, and data is sent in parallel; The sender comprises: A to-be-processed data queue module, each sender contains a data queue, wherein the to-be-sent data and the data which has been sent but has not received an ACK data packet are saved, when an ACK data packet is received, the relevant Packet data packet in the data queue is cleared, indicating that the Packet data packet data has been processed; A Sender module for sending Packet data packets and command packets; A Responder module for receiving Ack data packets; A sender control module for controlling the type of the packet sent by the Sender; the type of the packet comprises data packets and command packets, wherein the main commands comprise Create, Append, Recover, Sync, and Close; such command packets are consistent with the commands used in the three-replica PipeLine writing mode.

2. The method of claim 1, wherein, The data queue adopts a RingBuffer mode and adopts a lock-free processing, and three cursors are maintained in the RingBuffer, that is, a writeCursor, a readCursor, and a clearCursor; wherein the writeCursor is a client writing position, the readCursor is a Sender reading position and is maintained by the Sender module, and the clearCursor is a cleaning position, when the Responder receives an Ack, the position is moved; the three flags are all incremented and satisfy the following formula: 。 3. The method of claim 2, wherein, The Cursor needs to be taken modulo to obtain the real subscript of the RingBuffer.

4. The method of claim 3, wherein, The Sender module is used for maintaining a TCP connection with a DataNode and sending data packets or command packets to the DataNode through the connection, wherein the data packets are obtained from the data queue, and the command packets are sent to the Sender module through the controller; after the data packets are sent, the readCursor in the RingBuffer is updated.

5. The method of claim 4, wherein, The Responder module maintains a TCP connection replica of the DataNode in the Sender and receives an ACK data packet of a Packet data packet through the connection, for the Packet data packet receiving the ACK data packet, it indicates that the data packet has been landed, and the buffer in the data queue is deleted by modifying the clearCursor.

6. The method of claim 5, wherein, When the OutPutStream is called to perform a writing operation, a minimum replica mechanism and a replica lag detection mechanism are adopted, that is, two parameters are configured: A minimum number of replicas, when the number of writing replicas meets the configuration, it indicates that the data writing is successful; The maximum number of lags of a copy, that is, when the number of lags of a data packet sent by a certain sender reaches the configuration, the sender is marked as abnormal; and the sender is excluded from the SenderGroup.

7. The method of claim 6, wherein, If an abnormal sender is found during sending, the current normal sender needs to send a Recover command to make the normal copy update the version, so as to distinguish the copy corresponding to the abnormal sender, and thus to expire the data of the abnormal copy.

8. The method of claim 7, wherein, After the file writing is completed, the SenderGroup needs to be closed. If the file meets the minimum number of copies but does not meet the expected number of copies, it can be normally closed, and the subsequent NameNode performs a Reconstruction operation until the expected number of copies is met. If the minimum number of copies cannot be met when closed, it indicates that the file writing is abnormal, and a CloseException is thrown to the upper layer.

Images