A system and method for implementing communication between the HDFS file system and the S3 object storage system

Abstract

The present invention provides a system and method for implementing communication between an HDFS file system and an S3 object storage system, belonging to the technical field of computer data storage. The HDFS proxy module of the system includes: an HDFS NameNodeGateway module for receiving a user request carrying a namespace sent by an external HDFS client; a Distributer request gate module for receiving the user request sent by the HDFS NameNodeGateway module and calling a corresponding namespace module according to the received namespace; and a namespace module for judging according to the requested namespace and forwarding the request information to an external HDFS file system or an external S3 object storage system according to the judgment result. The present invention realizes the interconnection and communication between the HDFS file system and the S3 object storage system without changing the access semantics of the original file system.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data storage, and particularly relates to a system and method for implementing communication between an HDFS file system and an S3 object storage system. Background Art

[0002] Hadoop is an open-source distributed computing framework designed to process large-scale datasets. It provides reliable and efficient distributed storage and the ability to process big data, and is suitable for running large-scale applications on a cluster.

[0003] The core components of Hadoop include Hadoop Distributed File System (HDFS). HDFS is the distributed file system of Hadoop, used to store large-scale datasets and provide high fault tolerance and high throughput. HDFS adopts a master / slave architecture, mainly composed of the NameNodeGateway module for storing file information and a certain number of DataNodes for storing data. The NameNodeGateway module is a central server responsible for managing the namespace of the file system and client access to files. Generally, there is one Datanode per node in the cluster, which is responsible for managing the storage on the node it is on. HDFS exposes the namespace of the file system, and users can store data on it in the form of files. Internally, a file is actually divided into one or more data blocks, and these blocks are stored on a group of Datanodes. The NameNodeGateway module performs namespace operations on the file system, such as opening, closing, renaming files or directories. It is also responsible for determining the mapping of data blocks to specific Datanode nodes. The Datanode is responsible for handling read and write requests from the file system client. Under the unified scheduling of the NameNodeGateway module, data blocks are created, deleted, and replicated. It divides the data into blocks and then distributes them across multiple nodes in the cluster to ensure data reliability and efficient access. Hadoop is usually applied to the following scenarios: big data storage and analysis; log processing; text analysis; data mining and machine learning; image processing and computer vision, etc., which require a large number of scientific and technological scenarios. Generally speaking, Hadoop is suitable for scenarios that require big data processing, distributed computing, fault tolerance, and high reliability, especially those fields that need to store, process, and analyze massive amounts of data.

[0004] Processing and storing data through HDFS usually requires a standard processing process, which is: extracting the source data to be processed, processing the data, saving the processing results, and repeating this process over and over again, which will generate a large amount of result data. This data includes hot data, warm data, and cold data. Although HDFS can store some data, it has the problem of high storage cost and complex management and maintenance for the results generated by massive data calculations.

[0005] HDFS cannot store all the results. The S3 (Simple Storage Service) object storage system, as the current mainstream distributed file system, is a massive unstructured storage system that is more suitable for storing such warm and cold data. The basic unit of object storage is an object, which usually contains data, metadata, and a unique identifier (ObjectKey). Objects can be dynamically resized as needed and can be stored in multiple nodes for redundant replication to ensure data reliability and persistence. Therefore, object storage can provide highly scalable storage capacity, high-performance data access, and highly secure data protection.

[0006] However, since computing platforms usually access corresponding processing data and store processing results through HDFS, how to open up the access channel between the HDFS file system and the S3 object storage system with different file access and reading interfaces is a technical problem that needs to be solved. Summary of the invention

[0007] In order to solve the problem of how to open up an access channel between an HDFS file system and an object storage system, the present invention provides a system for realizing communication between an HDFS file system and an S3 object storage system.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A system for realizing communication between an HDFS file system and an S3 object storage system, wherein the communication between the HDFS file system and the S3 object storage system is realized by setting an HDFS proxy module, wherein the HDFS proxy module comprises:

[0010] The HDFS NameNodeGateway module is used to receive user requests with namespaces sent by external HDFS clients;

[0011] The Distributer request gateway module is used to receive user requests sent by the HDFS NameNodeGateway module and call the corresponding NamespaceHandler namespace module according to the received namespace;

[0012] The NamespaceHandler namespace module is used to make judgments based on the requested namespace and forward the request information to the external HDFS file system or the external S3 object storage system according to the judgment results.

[0013] Preferably, the HDFS proxy module further includes a ClientProtocol execution module; if the request information needs to be forwarded to the external HDFS file system, the NamespaceHandler namespace module communicates with the external HDFS file system through the ClientProtocol execution module to implement data forwarding; at the same time, the request result is returned to the HDFS NameNodeGateway module through the ClientProtocol execution module.

[0014] Preferably, the HDFS proxy module further includes an S3Interpreter S3 command conversion module, an S3Request S3 instruction execution module, and a ResponseConvert result conversion module;

[0015] If the request information needs to be forwarded to the external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module. The S3Interpreter S3 command conversion module reorganizes the received request information into an object access interface request information supported by the S3 protocol and sends the reorganized request information to the external S3 object storage system through the S3Request S3 instruction execution module; at the same time, the request result is returned to the HDFS NameNodeGateway module through the S3Request S3 instruction execution module, the ResponseConvert result conversion module, and the S3Interpreter S3 command conversion module in sequence.

[0016] Preferably, the reorganization of the request information received by the S3Interpreter S3 command conversion module includes: object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject, and directory creation Mkdir.

[0017] Preferably, the HDFS proxy module further includes an Authorization authentication module, and the Authorization authentication module is set between the HDFS NameNodeGateway module and the Distributer request gate module;

[0018] Before the HDFS NameNodeGateway module sends the user request carrying the namespace received to the Distributer request gate module, the Authorization authentication module verifies the legality of the user authentication information according to the carried verification information; when the request verification is incorrect, the error information is returned to the external HDFS client through the NameNodeGateway module; when the requested user is legal, the HDFS NameNodeGateway module sends the user request carrying the namespace received to the Distributer request gate module.

[0019] The present invention also provides a method for implementing communication between an HDFS file system and an S3 object storage system, including:

[0020] The HDFS NameNodeGateway module receives a user request carrying a namespace sent by an external HDFS client;

[0021] The Distributer request gate module receives the user request sent by the HDFS NameNodeGateway module and calls the corresponding NamespaceHandler namespace module according to the received namespace;

[0022] The NamespaceHandler namespace module makes a judgment according to the requested namespace and forwards the request information to the external HDFS file system or the external S3 object storage system according to the judgment result.

[0023] Preferably, before the HDFS NameNodeGateway module receives the user request carrying the namespace sent by the external HDFS client, it further includes configuring the namespace mapping of the HDFS file system, specifically: mapping a part of the namespace to an external HDFS NameNode service node and mapping another part of the namespace to an external object storage bucket Bucket1.

[0024] The present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of any one of the systems for implementing communication between an HDFS file system and an S3 object storage system.

[0025] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and characterized in that when the computer program is loaded by a processor, it can execute the steps of any one of the systems for implementing communication between an HDFS file system and an S3 object storage system.

[0026] The system provided by the present invention for implementing communication between the HDFS file system and the S3 object storage system has the following beneficial effects:

[0027] The present invention constructs an HDFS proxy module through the NameNodeGateway module, the Distributer request gate module, and the NamespaceHandler namespace processing module. As an intermediate proxy layer, it connects the HDFS file system and the S3 object storage system. The NameNodeGateway module simulates the NameNodeGateway module service in the original HDFS, provides RPC access externally, and receives user requests carrying the namespace sent by external HDFS clients. At the same time, the Distributer request gate module calls the corresponding NamespaceHandler namespace module according to the received namespace, and the NamespaceHandler namespace module makes a judgment based on the requested namespace, and forwards the request information to the external HDFS file system or the external S3 object storage system according to the judgment result.

[0028] By processing and transmitting client requests through the above structure, the interconnection between the HDFS file system and the S3 object storage system is realized without changing the access semantics of the original file system. The data storage cost can be reduced by migrating cold data to S3 storage, enabling massive computing results based on the HDFS file system to use low-cost object storage as the storage medium, and enhancing the application scenarios of object storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention and their design schemes, the accompanying drawings required for this embodiment will be briefly introduced below. The drawings described below are only partial embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 It is a schematic block diagram of the system for implementing communication between the HDFS file system and the S3 object storage system in Embodiment 1 of the present invention;

[0031] Figure 2 It is a flowchart of the method for implementing communication between the HDFS file system and the S3 object storage system in Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] In order to enable those skilled in the art to better understand the technical solutions of the present invention and implement them, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be used to limit the protection scope of the present invention.

[0033] Example 1

[0034] The present invention provides a system for implementing communication between an HDFS file system and an S3 object storage system, specifically a system for implementing communication between an HDFS file system and an S3 object storage system by setting up an HDFS proxy module. As Figure 1 shown, the HDFS proxy module includes an HDFS NameNodeGateway module, an Authorization authentication module, a Distributer request gate module, a NamespaceHandler namespace module, an S3Interpreter S3 command conversion module, a ClientProtocol execution module, an S3Request S3 instruction execution module, and a ResponseConvert result conversion module.

[0035] Specifically, before the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gate module, the Authorization authentication module verifies the legality of the user authentication information according to the carried verification information; when the request verification is incorrect, the error information is returned to the external HDFS client through the NameNodeGateway module; when the requested user is legal, the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gate module. The Distributer request gate module is used to receive the user request sent by the Authorization authentication module and call the corresponding NamespaceHandler namespace module according to the received namespace; the NamespaceHandler namespace module is used to make a judgment according to the requested namespace and forward the request information to the external HDFS file system ( Figure 1 the external HDFS system NameNode service in Figure 1 ) or the external S3 object storage system (

[0036] the external S3 service in

[0037] If the request information needs to be forwarded to an external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module. The S3Interpreter S3 command conversion module reorganizes the received request information into an object access interface request information supported by the S3 protocol, and sends the reorganized request information to the external S3 object storage system through the S3Request S3 instruction execution module. At the same time, the request result is returned to the HDFS NameNodeGateway module through the S3Request S3 instruction execution module, the ResponseConvert result conversion module, and the S3Interpreter S3 command conversion module in sequence. Among them, the reorganization of the request information received by the S3Interpreter S3 command conversion module includes: object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject, and directory creation Mkdir.

[0038] The present invention can realize the access to two storage systems through the original HDFS interface by the namespace information in the request, shielding the complexity of S3 interface access; realizing the interconnection and interoperability between the HDFS file system and the S3 object storage system without changing the original file system access semantics, enabling the massive computing results based on the HDFS file system to use the low-cost object storage as the storage medium, reducing the storage cost, and enhancing the application scenario of the object storage.

[0039] The present invention can completely simulate all protocol processing in HDFS access through the HDFS proxy module, and can access the object storage and the original HDFS system without modifying the original HDFS client access protocol, avoiding system incompatibility and seamlessly taking over the HDFS service; and greatly expanding the capacity of the original HDFS storage system, and reducing the data storage cost by migrating cold data to the S3 storage.

[0040] Each module in the above system for realizing the communication between the HDFS file system and the S3 object storage system can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in the processor in the computer device in the form of hardware or independent of the processor, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0041] Embodiment 2

[0042] Based on the same inventive concept, the present invention also provides a method for implementing communication between the HDFS file system and the S3 object storage system, specifically a system for implementing communication between the HDFS file system and the S3 object storage system by setting up an HDFS proxy module, as Figure 2 shown. The method includes the following steps:

[0043] S1. Configure the namespace mapping of the HDFS file system. For example, map namespace A to the external HDFS NameNode service section, and map namespace B to the external object storage bucket Bucket1.

[0044] S2. Start the NameNodeGateway module and RPC access. At this time, the NameNodeGateway module receives user requests carrying namespaces sent by external HDFS clients, and at the same time processes HDFS client requests (such as read and write requests) carrying namespaces according to the

Namespace mapping configuration of the HDFS file system

[0045] S3. Before the Distributer request gate module processes the request sent from the NameNodeGateway module, the Authorization authentication module will verify the legality of the user authentication information according to the carried verification information (using username and password authentication verification). When the request verification is incorrect, the error information will be returned to the external HDFS client through the NameNodeGateway module. When the requesting user is legal, it triggers the Distributer request gate module to process and forwards the request information to the Distributer request gate module.

[0046] S4. The Distributer request gate module receives the user request sent by the HDFS NameNodeGateway module and calls the corresponding NamespaceHandler namespace module according to the received namespace.

[0047] S5. The NamespaceHandler namespace processing module looks up which system the corresponding namespace should be handed over to according to the requested namespace (Namespace) and the

Namespace mapping configuration of the HDFS file system

[0048] S51. If it is necessary to forward to an external HDFS file system, call the same interface in the ClientProtocol execution module of the external HDFS file system. The ClientProtocol execution module forwards the request to the external HDFS file system and returns the request result to the HDFS NameNodeGateway module through the ClientProtocol execution module at the same time.

[0049] S52. If it is necessary to forward the request information to an external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module. The S3Interpreter S3 command conversion module reorganizes the received request information into an object access interface request information supported by the S3 protocol, including object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject, and directory creation Mkdir; and sends the reorganized request information to the external S3 object storage system through the S3Request S3 instruction execution module; at the same time, the request result is forwarded through the S3Request S3 instruction execution module in turn, and after being escaped by the ResponseConvert module, it is fed back to the NameNodeGateway module through the S3Interpreter command conversion module, and the NameNodeGateway module then feeds back the request result to the external S3 object storage system to complete the communication process.

[0050] The method for implementing communication between the HDFS file system and the S3 object storage system provided by the present invention achieves the following objectives:

[0051] 1. The present invention realizes the intercommunication between the HDFS file system and the S3 object storage system by constructing an HDFS proxy service.

[0052] 2. The NameNodeGateway module simulates the NameNodeGateway module service in the original HDFS and provides RPC access externally.

[0053] 3. The proxy implements the protocol interfaces involved in ClientProtocol in the external HDFS client.

[0054] The present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to execute the steps in the method embodiment for implementing communication between the HDFS file system and the S3 object storage system. For the specific implementation method, refer to the method embodiment and will not be elaborated here.

[0055] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions, on which a computer program is stored. For example, a memory containing instructions, and the above instructions can be executed by a processor of a computer device to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc. When the computer program is executed by the processor, it can execute the steps in the method embodiment for implementing the communication between the HDFS file system and the S3 object storage system. The specific implementation method can be referred to the method embodiment and will not be elaborated here.

[0056] Those skilled in the art should understand that the embodiments of the present invention can provide a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0058] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing the functions in one Figure 1 one flow or multiple flows and / or blocksFigure 1 Steps of functions specified in one or more boxes.

[0060] It should be noted that the above - described specific embodiments can enable those skilled in the art to understand the present invention - creation more comprehensively, but do not limit the present invention - creation in any way. Therefore, although the present specification and embodiments have described the present invention - creation in detail, those skilled in the art should understand that the present invention - creation can still be modified or equivalently replaced; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention - creation are covered by the protection scope of the patent of the present invention - creation. Any reference numeral in the claims should not be regarded as limiting the claimed claim. Any simple change or equivalent replacement of the technical solutions that can be obviously obtained by any person skilled in the art within the technical scope disclosed by the present invention belongs to the protection scope of the present invention.

Claims

1. A system for realizing communication between an HDFS file system and an S3 object storage system, characterized in that: The communication between the HDFS file system and the S3 object storage system is realized by setting an HDFS proxy module, and the HDFS proxy module includes: The HDFS NameNodeGateway module is used to receive user requests with namespaces sent by external HDFS clients; The Distributer request gateway module is used to receive user requests sent by the HDFS NameNodeGateway module and call the corresponding NamespaceHandler namespace module according to the received namespace; NamespaceHandler namespace module, used to make judgments based on the requested namespace and forward the request information to the external HDFS file system or external S3 object storage system based on the judgment results; The HDFS proxy module also includes a ClientProtocol execution module; if the request information needs to be forwarded to an external HDFS file system, the NamespaceHandler namespace module communicates with the external HDFS file system through the ClientProtocol execution module to achieve data forwarding; at the same time, the request result is returned to the HDFS NameNodeGateway module through the ClientProtocol execution module; The HDFS proxy module also includes an S3Interpreter S3 command conversion module, an S3Request S3 instruction execution module, and a ResponseConvert result conversion module; If the request information needs to be forwarded to an external S3 object storage system, the NamespaceHandler namespace module first forwards the request information to the S3Interpreter S3 command conversion module. The S3Interpreter S3 command conversion module reorganizes the received request information into object access interface request information supported by the S3 protocol, and sends the reorganized request information to the external S3 object storage system through the S3Request S3 command execution module; at the same time, the request result is returned to the HDFS NameNodeGateway module through the S3Request S3 command execution module, the ResponseConvert result conversion module and the S3Interpreter S3 command conversion module in sequence.

2. The system for realizing communication between HDFS file system and S3 object storage system according to claim 1, characterized in that: The S3Interpreter S3 command conversion module reorganizes the received request information to include: object reading ReadObject, object writing PutObject, object deletion DeleteObject, object listing ListObject and directory creation Mkdir.

3. The system for realizing communication between HDFS file system and S3 object storage system according to claim 1, characterized in that: The HDFS proxy module also includes an Authorization module, which is arranged between the HDFS NameNodeGateway module and the Distributer request gate module; Before the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module, the Authorization authentication module verifies the legitimacy of the user authentication information according to the carried verification information; When the request verification is incorrect, the error information is returned to the external HDFS client through the NameNodeGateway module; when the requesting user is legitimate, the HDFS NameNodeGateway module sends the received user request carrying the namespace to the Distributer request gateway module.

4. A method applied to the system for realizing communication between the HDFS file system and the S3 object storage system as claimed in claim 1, characterized in that: include: The HDFS NameNodeGateway module receives user requests carrying namespaces sent by external HDFS clients; The Distributer request gateway module receives user requests sent by the HDFS NameNodeGateway module and calls the corresponding NamespaceHandler namespace module according to the received namespace; The NamespaceHandler namespace module makes a judgment based on the requested namespace and forwards the request information to the external HDFS file system or external S3 object storage system based on the judgment result.

5. The method for realizing communication between the HDFS file system and the S3 object storage system according to claim 4, characterized in that: Before the HDFS NameNodeGateway module receives a user request carrying a namespace sent by an external HDFS client, it also includes configuring the namespace mapping of the HDFS file system, specifically: mapping part of the namespace to the external HDFS NameNode service node, and mapping the other part of the namespace to the external object storage bucket Bucket1.

6. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method for implementing communication between the HDFS file system and the S3 object storage system as described in claim 4 or 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is loaded by a processor, it can execute the steps of the method for realizing communication between the HDFS file system and the S3 object storage system as described in claim 4 or 5.

Images