HDFS protocol data mapping transmission method and device for distributed file system

Abstract

The present invention discloses a HDFS protocol data mapping and transmission method for a distributed file system, comprising: determining an obj size consistent with a block size in advance according to a block size, and establishing a mapping relationship between a block and obj; receiving a read and write request sent by an HDFS client, and determining the current obj according to the current block and the mapping relationship between the block and obj; calculating the target node IP where the main disk corresponding to the current obj is located according to the information of the current obj; returning the target node IP to the HDFS client, so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server; the present invention does not require a distributed storage client to forward data during use, thereby improving data reading and writing efficiency, reducing network bandwidth consumption, and being beneficial to improving system performance.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed file systems, and in particular to a method, device, system and computer-readable storage medium for HDFS protocol data mapping transmission of a distributed file system. Background Art

[0002] For the distributed file system (object storage), when using the HDFS (Hadoop Distributed FileSystem) protocol to write, the HDFS client transmits data to the distributed storage in blocks. After the data reaches the distributed storage client, the distributed storage client divides the block into objects (obj) again according to the distributed storage client's own data requirements, and then determines the disks mapped to each obj. The distributed storage client forwards the data to the node where the primary disk corresponding to the obj is located. In this process, the distributed storage client needs to forward a large amount of data to other nodes. In a high-pressure scenario, the network bandwidth consumption is very large, affecting the system performance.

[0003] In view of this, how to provide a HDFS protocol data mapping transmission method, device, system and computer-readable storage medium for a distributed file system with reduced network bandwidth consumption and higher system performance has become a problem that needs to be solved by those skilled in the art. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a distributed file system HDFS protocol data mapping transmission method, device, system and computer-readable storage medium, which improves data reading and writing efficiency, reduces network bandwidth consumption, and is conducive to improving system performance during use.

[0005] To solve the above technical problems, an embodiment of the present invention provides a distributed file system HDFS protocol data mapping and transmission method, including:

[0006] Determine the size of obj in advance according to the block size, and establish a mapping relationship between the block and obj, wherein the size of obj is consistent with the size of the block;

[0007] Receive read and write requests sent by the HDFS client, and determine the current obj based on the current block and the mapping relationship between the block and obj;

[0008] Calculate the target node IP address of the primary disk corresponding to the current obj according to the information of the current obj;

[0009] The target node IP is returned to the HDFS client so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

[0010] Optionally, before determining the current obj according to the current block and the mapping relationship between the block and obj, the method further includes:

[0011] Get the current read and write position;

[0012] A current block is determined according to the current read / write position.

[0013] Optionally, the naming format of the block is: blk_ino_index, where ino is the inode number of the file;

[0014] Then, the process of determining the current block according to the current read / write position is:

[0015] The index of the current block is determined according to the current read / write position, where index=offset / block_size, where offset is the current read / write position, and block_size is the block size.

[0016] Optionally, the process of calculating the target node IP address of the primary disk corresponding to the current obj according to the information of the current obj is as follows:

[0017] The crush algorithm is used to calculate the ino_index of the current obj to obtain the target node IP where the primary disk corresponding to the current obj is located.

[0018] Optionally, the process of determining the size of obj in advance according to the block size is:

[0019] The block size is obtained from the metadata server in advance, and the obj size is determined according to the block size.

[0020] Optionally, before obtaining the block size from the metadata server in advance, the method further includes:

[0021] Receive block size setting information sent by the HDFS client;

[0022] The block size in the block size setting information is persisted in the metadata server of the file, and the block size is used as the stripe unit for segmenting obj.

[0023] The embodiment of the present invention further provides a HDFS protocol data mapping and transmission device for a distributed file system, comprising:

[0024] An establishment module is used to determine the size of obj in advance according to the block size, and establish a mapping relationship between the block and obj, wherein the size of obj is consistent with the size of the block;

[0025] The receiving module is used to receive the read and write request sent by the HDFS client, and determine the current obj according to the current block and the mapping relationship between the block and obj;

[0026] A calculation module, used to calculate the target node IP where the primary disk corresponding to the current obj is located according to the information of the current obj;

[0027] The return module is used to return the target node IP to the HDFS client so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

[0028] Optionally, the method further comprises:

[0029] The acquisition module is used to obtain the current read and write position;

[0030] A determination module is used to determine a current block according to the current read / write position.

[0031] The embodiment of the present invention provides a HDFS protocol data mapping transmission system for a distributed file system, including:

[0032] Memory for storing computer programs;

[0033] A processor is used to implement the steps of the HDFS protocol data mapping and transmission method of the distributed file system as described above when executing the computer program.

[0034] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the HDFS protocol data mapping and transmission method of the distributed file system as described above are implemented.

[0035] The embodiment of the present invention provides an HDFS protocol data mapping and transmission method for a distributed file system, comprising: determining the size of obj in advance according to the block size, and establishing a mapping relationship between the block and obj, wherein the size of obj is consistent with the block size; receiving a read and write request sent by an HDFS client, and determining the current obj according to the current block and the mapping relationship between the block and obj; calculating the target node IP where the main disk corresponding to the current obj is located according to the information of the current obj; returning the target node IP to the HDFS client, so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

[0036] It can be seen that the embodiment of the present invention determines the size of obj through the block size, makes the size of obj consistent with the block size, and then establishes a mapping relationship between the block and obj. When the HDFS client performs read and write operations, it receives the read and write request sent by the HDFS client, and determines the current block according to the read and write request, determines the current obj according to the current block and the correspondence between the block and obj, and then further calculates the target node IP where the main disk corresponding to the current obj is located according to the information of the current obj, and returns the target node IP to the HDFS client. After receiving the target node IP, the HDFS client directly sends the data to the HDFS file server corresponding to the target node IP, so that the HDFS file server stores the data on the main disk of the target node. During use, the HDFS client can directly send data to the HDFS file server of the node where the main disk is located, without the need for the distributed storage client to forward the data, thereby improving data reading and writing efficiency, reducing network bandwidth consumption, and helping to improve system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the prior art and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A schematic diagram of a flow chart of a HDFS protocol data mapping and transmission method for a distributed file system provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a block-obj mapping relationship provided by an embodiment of the present invention;

[0040] Figure 3A schematic diagram of the structure of an HDFS protocol data mapping and transmission device for a distributed file system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The embodiments of the present invention provide a method, device, system and computer-readable storage medium for HDFS protocol data mapping transmission of a distributed file system, which improves data reading and writing efficiency, reduces network bandwidth consumption, and is conducive to improving system performance during use.

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Please refer to Figure 1 , Figure 1 A flow chart of a method for HDFS protocol data mapping and transmission of a distributed file system provided by an embodiment of the present invention. The method comprises:

[0044] S110: Determine the size of obj in advance according to the block size, and establish a mapping relationship between the block and obj, wherein the size of obj is consistent with the size of the block;

[0045] It should be noted that the method in the embodiment of the present invention can be applied to a distributed file system. Specifically, a mapping relationship between blocks and obj can be established first. The block size block_size can be obtained in advance to determine the size of obj, and a mapping relationship between the two can be established. For example, blk0 corresponds to obj0, blk1 corresponds to obj1, and blkn corresponds to objn. For details, please refer to Figure 2 , and calculate the IP address of the node where the disk corresponding to obj is located based on crush, so as to establish the mapping relationship between obj and disk, for example Figure 2 The obj0 in the table corresponds to disk 1, and disk 2 is the slave disk of disk 1.

[0046] Specifically, when creating a file, you can receive the block size setting information sent by the HDFS client, and then persist the block size block_size in the block size setting information to the file's metadata server, and use the block size as the stripe unit for dividing obj. Then, when establishing the mapping relationship between blocks and obj, obtain the block size from the metadata server in advance, and then determine the obj size based on the block size, thereby establishing a mapping relationship between blocks and obj.

[0047] S120: receiving a read / write request sent by the HDFS client, and determining the current obj according to the current block and the mapping relationship between the block and obj;

[0048] It should be noted that when reading and writing files, the current block can be obtained when receiving a read / write request sent by the HDFS client, and the current obj corresponding to the current block can be determined according to the mapping relationship between the block and obj. Specifically, the current block can be determined according to the current read / write position in the read / write request.

[0049] Furthermore, the naming format of the block may be specifically blk_ino_index, where ino is the inode number of the file; then, the index of the current block may be specifically determined according to the current read / write position, where index=offset / block_size, where offset is the current read / write position, and block_size is the block size.

[0050] S130: Calculate the target node IP where the primary disk corresponding to the current obj is located according to the information of the current obj;

[0051] It should be noted that after determining the current obj, the node IP of the node of the main disk corresponding to it can be calculated by calculating the current obj, and the node IP can be used as the target node IP. Specifically, the crush algorithm can be used to calculate the ino_index of the current obj to obtain the target node IP where the main disk corresponding to the current obj is located.

[0052] S140: Return the target node IP to the HDFS client so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

[0053] Specifically, after determining the target node IP, the target node IP is directly returned to the HDFS client. After receiving the target node IP, the HDFS client sends the corresponding data directly to the HDFS file server corresponding to the target node IP according to the block size in S110. After receiving the data, the HDFS file server sends the data to the primary disk of the node corresponding to the HDFS file server for storage, that is, stores the data on the primary disk of the node corresponding to the target node IP, and processes the data reading and writing operations of the secondary disk corresponding to the primary disk, wherein each HDFS file server corresponds to a storage node.

[0054] It can be seen that the embodiment of the present invention determines the size of obj through the block size, makes the size of obj consistent with the block size, and then establishes a mapping relationship between the block and obj. When the HDFS client performs read and write operations, it receives the read and write request sent by the HDFS client, and determines the current block according to the read and write request, determines the current obj according to the current block and the correspondence between the block and obj, and then further calculates the target node IP where the main disk corresponding to the current obj is located according to the information of the current obj, and returns the target node IP to the HDFS client. After receiving the target node IP, the HDFS client directly sends the data to the HDFS file server corresponding to the target node IP, so that the HDFS file server stores the data on the main disk of the target node. During use, the HDFS client can directly send data to the HDFS file server of the node where the main disk is located, without the need for the distributed storage client to forward the data, thereby improving data reading and writing efficiency, reducing network bandwidth consumption, and helping to improve system performance.

[0055] Based on the above embodiments, the present invention also provides a distributed file system HDFS protocol data mapping transmission device, for details, please refer to Figure 3 The device comprises:

[0056] Establishing module 21, used to determine the size of obj according to the block size in advance, and establish a mapping relationship between the block and obj, wherein the size of obj is consistent with the size of the block;

[0057] The receiving module 22 is used to receive the read and write request sent by the HDFS client, and determine the current obj according to the current block and the mapping relationship between the block and obj;

[0058] The calculation module 23 is used to calculate the target node IP where the primary disk corresponding to the current obj is located according to the information of the current obj;

[0059] The return module 24 is used to return the target node IP to the HDFS client so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

[0060] Furthermore, the device may also include:

[0061] The acquisition module is used to obtain the current read and write position;

[0062] The determination module is used to determine the current block according to the current read and write position.

[0063] It should be noted that the HDFS protocol data mapping and transmission device of the distributed file system in the embodiment of the present invention has the same beneficial effects as the HDFS protocol data mapping and transmission method of the distributed file system provided in the above embodiment, and for the HDFS protocol data mapping and transmission method of the distributed file system involved in the embodiment of the present invention, please refer to the above embodiment, and the present invention will not be repeated here.

[0064] Based on the above embodiments, an embodiment of the present invention provides a HDFS protocol data mapping and transmission system for a distributed file system, the system comprising:

[0065] Memory for storing computer programs;

[0066] The processor is used to implement the steps of the HDFS protocol data mapping and transmission method of the distributed file system as described above when executing a computer program.

[0067] For example, the processor in the embodiment of the present invention can be specifically used to implement pre-determining the obj size based on the block size, and establishing a mapping relationship between the block and obj, wherein the obj size is consistent with the block size; receiving a read and write request sent by the HDFS client, and determining the current obj based on the current block and the mapping relationship between the block and obj; calculating the target node IP where the main disk corresponding to the current obj is located based on the information of the current obj; returning the target node IP to the HDFS client, so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

[0068] Based on the above embodiments, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the HDFS protocol data mapping and transmission method of the above-mentioned distributed file system are implemented.

[0069] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0070] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0071] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0072] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed file system HDFS protocol data mapping transmission method, characterized in that: include: Receive block size setting information sent by the HDFS client; The block size in the block size setting information is persisted to the metadata server of the file, and the block size is used as the stripe unit for segmenting obj; Obtaining a block size from the metadata server in advance, determining an obj size according to the block size, and establishing a mapping relationship between a block and obj, wherein the obj size is consistent with the block size; Receive read and write requests sent by the HDFS client, and determine the current obj based on the current block and the mapping relationship between the block and obj; Calculate the target node IP of the primary disk corresponding to the current obj according to the information of the current obj, so as to establish a mapping relationship between the current obj and the primary disk; The target node IP is returned to the HDFS client so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

2. The HDFS protocol data mapping and transmission method of the distributed file system according to claim 1 is characterized in that: Before determining the current obj according to the current block and the mapping relationship between the block and obj, the method further includes: Get the current read and write position; A current block is determined according to the current read / write position.

3. The HDFS protocol data mapping and transmission method of the distributed file system according to claim 2 is characterized in that: The naming format of the block is: blk_ino_index, where ino is the inode number of the file; Then, the process of determining the current block according to the current read / write position is: The index of the current block is determined according to the current read / write position, where index=offset / block_size, where offset is the current read / write position, and block_size is the block size.

4. The HDFS protocol data mapping and transmission method of the distributed file system according to claim 1, characterized in that: The process of calculating the target node IP of the primary disk corresponding to the current obj according to the information of the current obj is as follows: The crush algorithm is used to calculate the ino_index of the current obj to obtain the target node IP where the primary disk corresponding to the current obj is located.

5. A distributed file system HDFS protocol data mapping transmission device, characterized in that: include: Establish a module to receive block size setting information sent by HDFS client; Persisting the block size in the block size setting information in the metadata server of the file, and using the block size as the stripe unit for dividing obj; obtaining the block size from the metadata server in advance, determining the obj size according to the block size, and establishing a mapping relationship between the block and obj, wherein the obj size is consistent with the block size; The receiving module is used to receive the read and write request sent by the HDFS client, and determine the current obj according to the current block and the mapping relationship between the block and obj; A calculation module, used to calculate the target node IP where the primary disk corresponding to the current obj is located according to the information of the current obj, so as to establish a mapping relationship between the current obj and the primary disk; The return module is used to return the target node IP to the HDFS client so that the HDFS client sends the corresponding data to the HDFS file server corresponding to the target node IP, and stores the data on the main disk through the HDFS file server.

6. The HDFS protocol data mapping and transmission device of the distributed file system according to claim 5, characterized in that: Also includes: The acquisition module is used to obtain the current read and write position; A determination module is used to determine a current block according to the current read / write position.

7. A distributed file system HDFS protocol data mapping transmission system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the HDFS protocol data mapping and transmission method of the distributed file system as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the HDFS protocol data mapping and transmission method of the distributed file system as claimed in any one of claims 1 to 5.

Images