System and method for realizing communication between HDFS (Hadoop Distributed File System) and S3 object storage system

Abstract

The invention provides a system and method for achieving communication between an HDFS file system and an S3 object storage system, and belongs to the technical field of computer data storage, and an HDFS agent module of the system comprises an HDFS NameNodeGateway module used for receiving a user request carrying a name space sent by an external HDFS client; the Distribter request gate module is used for receiving the user request sent by the HDFS NameNodeGateway module, and calling the corresponding name space module according to the received name space, and the Distribter request gate module is used for receiving the user request sent by the HDFS NameNodeGateway module; and the name space module is used for performing judgment according to the requested name space and forwarding the request information to an external HDFS file system or an external S3 object storage system according to a judgment result. According to the method, the intercommunication, interconnection and communication between the HDFS file system and the S3 object storage system are realized on the premise of not changing the access semantics of the original file system.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data storage, and in particular relates to a system and method for realizing communication between an HDFS file system and an S3 object storage system. Background Art

[0002] Hadoop is an open source distributed computing framework designed to process large-scale data sets. It provides reliable and efficient distributed storage and processing capabilities for big data, and is suitable for running large-scale applications on clusters.

[0003] The core components of Hadoop include Hadoop Distributed File System (HDFS). HDFS is Hadoop's distributed file system, which is used to store large-scale data sets and provides high fault tolerance and high throughput. HDFS adopts a master / slave architecture and is mainly composed of the NameNodeGateway module that stores file information and a certain number of DataNodes that store data. The NameNodeGateway module is a central server that manages the file system's namespace and client access to files. The Datanode in the cluster is generally one node per node and is responsible for managing the storage on the node where it is located. HDFS exposes the file system's namespace, and users can store data on it in the form of files. Internally, a file is actually divided into one or more data blocks, which are stored on a group of Datanodes. The NameNodeGateway module performs file system namespace operations, such as opening, closing, and renaming files or directories. It is also responsible for determining the mapping of data blocks to specific Datanode nodes. Datanode is responsible for processing read and write requests from file system clients. Data blocks are created, deleted, and replicated under the unified scheduling of the NameNodeGateway module. It divides data into blocks and then distributes them on multiple nodes in the cluster to ensure data reliability and efficient access. Hadoop is usually used in the following scenarios: big data storage and analysis; log processing; text analysis; data mining and machine learning; image processing and computer vision, etc., which require a lot of scientific and technological scenarios. In general, Hadoop is suitable for scenarios that require processing big data, distributed computing, fault tolerance and high reliability, especially those fields that require storage, processing and analysis of massive data.

[0004] Processing and storing data through HDFS usually requires a standard processing process, which is: extracting the source data to be processed, processing the data, saving the processing results, and repeating this process over and over again, which will generate a large amount of result data. This data includes hot data, warm data, and cold data. Although HDFS can store some data, it has the problem of high storage cost and complex management and maintenance for the results generated by massive data calculations.

[0005] HDFS cannot store all the results. The S3 (Simple Storage Service) object storage system, as the current mainstream distributed file system, is a massive unstructured storage system that is more suitable for storing such warm and cold data. The basic unit of object storage is an object, which usually contains data, metadata, and a unique identifier (ObjectKey). Objects can be dynamically resized as needed and can be stored in multiple nodes for redundant replication to ensure data reliability and persistence. Therefore, object storage can provide highly scalable storage capacity, high-performance data access, and highly secure data protection.

[0006] However, since computing platforms usually access corresponding processing data and store processing results through HDFS, how to open up the access channel between the HDFS file system and the S3 object storage system with different file access and reading interfaces is a technical problem that needs to be solved. Summary of the invention

[0007] In order to solve the problem of how to open up an access channel between an HDFS file system and an object storage system, the present invention provides a system for realizing communication between an HDFS file system and an S3 object storage system.

[0008] In order to achieve the above object, the present invention provides the following technical solutions: A system for realizing communication between an HDFS file system and an S3 object storage system, wherein the communication between the HDFS file system and the S3 object storage system is realized by setting an HDFS proxy module, wherein the HDFS proxy module comprises: The HDFS NameNodeGateway module is used to receive user requests with namespaces sent by external HDFS clients; The Distributer request gateway module is used to receive user requests sent by the HDFS NameNodeGateway module and call the corresponding NamespaceHandler namespace module according to the received namespace; The NamespaceHandler namespace module is used to make judgments based on the namespace of the request and forward the request information to the external HDFS file system or external S3 object storage system based on the judgment results.

[0009] Preferably, the HDFS proxy module also includes a ClientProtocol execution module; if the request information needs to be forwarded to an external HDFS file system, the NamespaceHandler namespace module communicates with the external HDFS file system through the ClientProtocol execution module to achieve data forwarding; at the same time, the request result is returned to the HDFS NameNodeGateway module through the ClientProtocol execution module.

[0010] Preferably, the HDFS proxy module also includes an S3Interpreter S3 command conversion module, an S3RequestS3 instruction execution module and a ResponseConvert result conversion module; If the request information needs to be forwarded to an external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module. The S3Interpreter S3 command conversion module reorganizes the received request information into object access interface request information supported by the S3 protocol, and sends the reorganized request information to the external S3 object storage system through the S3Request S3 command execution module; at the same time, the request result is returned to the HDFS NameNodeGateway module through the S3Request S3 command execution module, the ResponseConvert result conversion module and the S3Interpreter S3 command conversion module in sequence.

[0011] Preferably, the S3Interpreter S3 command conversion module reorganizes the received request information to include: object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject and directory creation Mkdir.

[0012] Preferably, the HDFS proxy module further includes an Authorization module, and the Authorization module is arranged between the HDFS NameNodeGateway module and the Distributer request gate module; Before the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module, the Authorization authentication module verifies the legitimacy of the user authentication information based on the carried verification information; when the request verification is incorrect, the error information is returned to the external HDFS client through the NameNodeGateway module; when the requesting user is legitimate, the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module.

[0013] The present invention also provides a method for realizing communication between an HDFS file system and an S3 object storage system, comprising: The HDFS NameNodeGateway module receives user requests carrying namespaces sent by external HDFS clients; The Distributer request gateway module receives user requests sent by the HDFS NameNodeGateway module and calls the corresponding NamespaceHandler namespace module according to the received namespace; The NamespaceHandler namespace module makes a judgment based on the requested namespace and forwards the request information to the external HDFS file system or external S3 object storage system based on the judgment result.

[0014] Preferably, before the HDFS NameNodeGateway module receives a user request carrying a namespace sent by an external HDFS client, it also includes configuring the namespace mapping of the HDFS file system, specifically: mapping a part of the namespace to the external HDFS NameNode service node, and mapping another part of the namespace to the external object storage bucket Bucket1.

[0015] The present invention also provides a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement any one of the steps in the system for implementing communication between an HDFS file system and an S3 object storage system.

[0016] The present invention also provides a computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is loaded by a processor, it can execute any one of the steps in the system for implementing communication between the HDFS file system and the S3 object storage system.

[0017] The system for realizing communication between the HDFS file system and the S3 object storage system provided by the present invention has the following beneficial effects: The present invention constructs an HDFS proxy module through a NameNodeGateway module, a Distributer request gate module and a NamespaceHandler namespace processing module, and connects the HDFS file system and the S3 object storage system as an intermediate proxy layer. The NameNodeGateway module simulates the NameNodeGateway module service in the original HDFS, provides RPC access to the outside, and receives user requests carrying namespaces sent by external HDFS clients. At the same time, the Distributer request gate module calls the corresponding NamespaceHandler namespace module according to the received namespace, and makes a judgment according to the requested namespace through the NamespaceHandler namespace module, and forwards the request information to the external HDFS file system or the external S3 object storage system according to the judgment result.

[0018] The client requests are processed and transmitted through the above structure, and the interconnection between the HDFS file system and the S3 object storage system is realized without changing the access semantics of the original file system. The data storage cost can be reduced by migrating cold data to S3 storage, and the massive computing results based on the HDFS file system can be stored in low-cost object storage, which improves the application scenarios of object storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a principle block diagram of a system for implementing communication between an HDFS file system and an S3 object storage system according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of a method for implementing communication between an HDFS file system and an S3 object storage system according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0022] Example 1 The present invention provides a system for realizing communication between an HDFS file system and an S3 object storage system, specifically a system for realizing communication between an HDFS file system and an S3 object storage system by setting an HDFS proxy module, such as Figure 1 As shown, the HDFS proxy module includes an HDFS NameNodeGateway module, an Authorization authentication module, a Distributer request gate module, a NamespaceHandler namespace module, an S3Interpreter S3 command conversion module, a ClientProtocol execution module, an S3Request S3 instruction execution module and a ResponseConvert result conversion module.

[0023] Specifically, before the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module, the Authorization module verifies the legitimacy of the user authentication information based on the carried verification information; when the request verification is incorrect, the error information is returned to the external HDFS client through the NameNodeGateway module; when the requesting user is legitimate, the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module. The Distributer request gateway module is used to receive the user request sent by the Authorization module, and call the corresponding NamespaceHandler namespace module according to the received namespace; the NamespaceHandler namespace module is used to make judgments based on the requested namespace, and forward the request information to the external HDFS file system based on the judgment result ( Figure 1 External HDFS system NameNode service in the Figure 1 external S3 service in .

[0024] Furthermore, if the request information needs to be forwarded to an external HDFS file system, the NamespaceHandler namespace module communicates with the external HDFS file system through the ClientProtocol execution module to achieve data forwarding; at the same time, the request result is returned to the HDFS NameNodeGateway module through the ClientProtocol execution module.

[0025] If the request information needs to be forwarded to an external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module, which reorganizes the received request information into object access interface request information supported by the S3 protocol, and sends the reorganized request information to the external S3 object storage system through the S3Request S3 command execution module; at the same time, the request result is returned to the HDFS NameNodeGateway module through the S3Request S3 command execution module, the ResponseConvert result conversion module, and the S3Interpreter S3 command conversion module in sequence. Among them, the S3Interpreter S3 command conversion module reorganizes the received request information into: object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject, and directory creation Mkdir.

[0026] The present invention can access two storage systems through the original HDFS interface through the namespace information in the request, shielding the complexity of S3 interface access; it realizes the interconnection between the HDFS file system and the S3 object storage system without changing the original file system access semantics, so that the massive computing results based on the HDFS file system can be stored in low-cost object storage as the storage medium, reducing storage costs and improving the application scenarios of object storage.

[0027] The present invention completely simulates all protocol processing in HDFS access through the HDFS proxy module, and can access object storage and the original HDFS system without modifying the original HDFS client access protocol, avoiding incompatibility between systems and seamlessly taking over HDFS services; and greatly expands the capacity of the original HDFS storage system, and can reduce data storage costs by migrating cold data to S3 storage.

[0028] Each module in the system for realizing communication between the HDFS file system and the S3 object storage system can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0029] Example 2 Based on the same inventive concept, the present invention also provides a method for realizing communication between an HDFS file system and an S3 object storage system, specifically a system for realizing communication between an HDFS file system and an S3 object storage system by setting an HDFS proxy module, such as Figure 2 As shown, the method comprises the following steps: S1. Configure the namespace mapping of the HDFS file system. For example, map namespace A to the external HDFSNameNode service node and namespace B to the external object storage bucket Bucket1.

[0030] S2. Start the NameNodeGateway module and RPC access. At this time, the NameNodeGateway module receives user requests with namespaces sent by external HDFS clients, and processes HDFS client requests (read and write requests, etc.) with namespaces according to the [Namespace Mapping Configuration of the HDFS File System]. Client requests usually carry namespace information, such as request commands such as Read Write File, etc. After receiving the request, the NameNodeGateway module will carry the request information and hand it over to the Distributer Request Gate module for processing.

[0031] S3. Before the Distributer request gateway module processes the request sent from the NameNodeGateway module, the Authorization authentication module verifies the legitimacy of the user authentication information based on the carried authentication information (using username and password authentication verification). If the request verification is incorrect, the error message is returned to the external HDFS client through the NameNodeGateway module. If the requesting user is legitimate, the Distributer request gateway module is triggered to process and forward the request information to the Distributer request gateway module.

[0032] S4. The Distributer request gateway module receives the user request sent by the HDFS NameNodeGateway module and calls the corresponding NamespaceHandler namespace module according to the received namespace.

[0033] S5, NamespaceHandler namespace processing module finds which system should handle the corresponding namespace according to the requested namespace (Namespace) and [HDFS file system namespace mapping configuration], that is, it makes a judgment based on the requested namespace and forwards the request information to the external HDFS file system or external S3 object storage system based on the judgment result.

[0034] S51. If forwarding to an external HDFS file system is required, the same interface in the ClientProtocol execution module of the external HDFS file system is called. The ClientProtocol execution module forwards the request to the external HDFS file system and returns the request result to the HDFS NameNodeGateway module through the ClientProtocol execution module.

[0035] S52. If the request information needs to be forwarded to an external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module. The S3Interpreter S3 command conversion module reorganizes the received request information into object access interface request information supported by the S3 protocol, including object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject and directory creation Mkdir; and sends the reorganized request information to the external S3 object storage system through the S3Request S3 command execution module; at the same time, the request result is forwarded in turn through the S3Request S3 command execution module, escaped by the ResponseConvert module, and fed back to the NameNodeGateway module through the S3Interpreter command conversion module. The NameNodeGateway module then feeds back the request result to the external S3 object storage system to complete the communication processing.

[0036] The method for realizing communication between the HDFS file system and the S3 object storage system provided by the present invention achieves the following objectives: 1. The present invention realizes the intercommunication between the HDFS file system and the S3 object storage system by constructing an HDFS proxy service.

[0037] 2. Use the NameNodeGateway module to simulate the NameNodeGateway module service in the original HDFS and provide RPC access to the outside world.

[0038] 3. The proxy implements the protocol interface involved in ClientProtocol in the external HDFS client.

[0039] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to execute the steps in the method embodiment for implementing communication between the HDFS file system and the S3 object storage system. The specific implementation method can be found in the method embodiment, which will not be described in detail here.

[0040] Furthermore, the present invention also provides a non-temporary computer-readable storage medium containing instructions, and a computer program is stored on the storage medium. For example, a memory containing instructions, the above instructions can be executed by a processor of a computer device to complete the above method. For example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in the method embodiment for implementing the communication between the HDFS file system and the S3 object storage system can be executed. The specific implementation method can be referred to the method embodiment, which will not be repeated here.

[0041] It should be understood by those skilled in the art that embodiments of the present invention may provide methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0042] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0043] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0044] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0045] It should be pointed out that the specific implementation methods described above can enable those skilled in the art to understand the invention more comprehensively, but do not limit the invention in any way. Therefore, although the invention has been described in detail in this specification and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents; and all technical solutions and improvements that do not deviate from the spirit and scope of the invention are included in the protection scope of the patent for the invention. Any figure mark in the claims should not be regarded as limiting the claims involved. Any simple change or equivalent replacement of the technical solution that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belongs to the protection scope of the present invention.

Claims

1. A system for realizing communication between an HDFS file system and an S3 object storage system, characterized in that: The communication between the HDFS file system and the S3 object storage system is realized by setting an HDFS proxy module, and the HDFS proxy module includes: The HDFS NameNodeGateway module is used to receive user requests with namespaces sent by external HDFS clients; The Distributer request gateway module is used to receive user requests sent by the HDFS NameNodeGateway module and call the corresponding NamespaceHandler namespace module according to the received namespace; NamespaceHandler namespace module, used to make judgments based on the requested namespace and forward the request information to the external HDFS file system or external S3 object storage system based on the judgment results; The HDFS proxy module also includes a ClientProtocol execution module; if the request information needs to be forwarded to an external HDFS file system, the NamespaceHandler namespace module communicates with the external HDFS file system through the ClientProtocol execution module to achieve data forwarding; at the same time, the request result is returned to the HDFS NameNodeGateway module through the ClientProtocol execution module.

2. The system for realizing communication between HDFS file system and S3 object storage system according to claim 1, characterized in that: The HDFS proxy module also includes an S3Interpreter S3 command conversion module, an S3Request S3 instruction execution module, and a ResponseConvert result conversion module; If the request information needs to be forwarded to an external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module. The S3Interpreter S3 command conversion module reorganizes the received request information into object access interface request information supported by the S3 protocol, and sends the reorganized request information to the external S3 object storage system through the S3Request S3 command execution module; at the same time, the request result is returned to the HDFS NameNodeGateway module through the S3Request S3 command execution module, the ResponseConvert result conversion module and the S3Interpreter S3 command conversion module in sequence.

3. The system for realizing communication between HDFS file system and S3 object storage system according to claim 2, characterized in that: The S3Interpreter S3 command conversion module reorganizes the received request information to include: object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject and directory creation Mkdir.

4. The system for realizing communication between HDFS file system and S3 object storage system according to claim 1, characterized in that: The HDFS proxy module also includes an Authorization module, which is arranged between the HDFS NameNodeGateway module and the Distributer request gate module; Before the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module, the Authorization authentication module verifies the legitimacy of the user authentication information according to the carried verification information; When the request verification is incorrect, the error information is returned to the external HDFS client through the NameNodeGateway module; when the requesting user is legitimate, the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module.

5. A method for realizing communication between an HDFS file system and an S3 object storage system, characterized in that: include: The HDFS NameNodeGateway module receives user requests carrying namespaces sent by external HDFS clients; The Distributer request gateway module receives user requests sent by the HDFS NameNodeGateway module and calls the corresponding NamespaceHandler namespace module according to the received namespace; The NamespaceHandler namespace module makes a judgment based on the requested namespace and forwards the request information to the external HDFS file system or external S3 object storage system based on the judgment result.

6. The method for realizing communication between the HDFS file system and the S3 object storage system according to claim 5, characterized in that: Before the HDFS NameNodeGateway module receives a user request carrying a namespace sent by an external HDFS client, it also includes configuring the namespace mapping of the HDFS file system, specifically: mapping part of the namespace to the external HDFS NameNode service node, and mapping the other part of the namespace to the external object storage bucket Bucket1.

7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method for implementing communication between the HDFS file system and the S3 object storage system as described in claim 5 or 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is loaded by a processor, it can execute the steps of the method for realizing communication between the HDFS file system and the S3 object storage system as described in claim 5 or 6.

Images