A method for interconnecting data lake and Ceph RGW based on Swift protocol

Abstract

The present invention relates to a method for interconnecting a data lake and Ceph RGW based on the Swift protocol. This method addresses the prior art issue of Delta Lake consuming a long time and incurring high hardware costs when systematically operating data stored in Ceph RGW. The method includes S1: creating a Hadoop-Ceph RGW file system module; S2: constructing and adapting a transmission interface; S3: creating a class inheritance and implementing a FileSystem abstract class; S4: connecting Ceph RGW and Delta Lake using the adapted interface; and S5: accessing and systematically operating data stored in the Ceph RGW storage repository through the FileSystem abstract class. The present invention has the advantages of effectively improving work efficiency and reducing hardware costs.

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a method for interconnecting a data lake and CephRGW based on the Swift protocol. Background Art

[0002] The current data synchronization needs to obtain structural update data from the data lake, which causes the server to call the configuration file or data update table, obtain the structural update data from the data lake, and synchronize it to the client. This method easily leads to low efficiency of data synchronization operations; and usually, when a large amount of data is stored in Ceph RGW and Delta Lake is used to analyze this data and mine useful information, the traditional approach is to first export the data in the Ceph RGW cluster to the intermediate server, and then import this data into Delta Lake to realize the data entry and subsequent MapReduce jobs. If the amount of data is very large, it will increase the time of the data import process and require additional storage space, which not only delays work efficiency but also increases hardware costs and is a poor usage method.

[0003] To address the shortcomings of existing technologies, researchers have conducted extensive research and proposed various solutions. For example, a Chinese patent document discloses a cloud-based data synchronization method, device, equipment, and storage medium [CN202210369249.4]. The method includes defining updated data in a data lake as updated data and identifying the update type of the updated data; performing data analysis on the updated data based on the update type to obtain analysis data, integrating the analysis data with the update type to obtain structural update data; performing data preprocessing on the structural update data to obtain data to be updated, and storing the data to be updated in a cache; sending an update announcement to a synchronization client, receiving an update request from the synchronization client, and authenticating the synchronization client based on the update request to obtain authentication permission information; and using the data to be updated in the cache corresponding to the authentication permission information as the target update data.

[0004] The above solution has solved the problem of inefficient data synchronization operations in the existing technology to a certain extent, but the solution still has many shortcomings. For example, when analyzing and mining a large amount of data stored on the Ceph RGW side, not only does the data import time take a long time, which delays work efficiency, but it also requires additional storage space, increases hardware costs, and is not a good way to use it. Summary of the Invention

[0005] The purpose of this invention is to address the above problems and provide a method for interconnecting a data lake and a Ceph RGW based on the Swift protocol.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions: a method for interconnecting a data lake and a Ceph RGW based on the Swift protocol, the method comprising the following steps:

[0007] S1. Create Hadoop-Ceph RGW file system module;

[0008] S2. Build the transmission interface and adapt it;

[0009] S3. Create a class that inherits and implements the FileSystem abstract class;

[0010] S4. Connect Ceph RGW and Delta Lake using the adapted interface.

[0011] S5. Access and systematize data stored in the Ceph RGW storage warehouse through the FileSystem abstract class.

[0012] This method actively builds a Hadoop-Ceph RGW file system module to receive request commands from Delta Lake, and builds a transmission interface adapted to Ceph RGW to establish a data connection between Ceph RGW and Delta Lake. When in use, the pre-created FileSystem abstract class is used to access and systematically operate the data stored in the Ceph RGW storage warehouse, eliminating the data transfer step on a third-party platform, improving work efficiency, and saving hardware costs without the need for additional storage space.

[0013] In the above-mentioned method for interconnecting a data lake and Ceph RGW based on the Swift protocol, in step S1, the Hadoop-Ceph RGW file system module receives a request command from Delta Lake.

[0014] In the aforementioned method for interconnecting a data lake with Ceph RGW based on the Swift protocol, the Hadoop-CephRGW file system module includes a CephObjectSystem subclass module, which serves as the central entry point and adaptation layer for receiving Delta Lake request commands. The CephObjectSystem subclass module is the intermediate module implementing this method, receiving request commands, searching the Ceph RGW storage repository for requested files, and responding with data to Delta Lake.

[0015] In the above-mentioned method for interconnecting a data lake and a Ceph RGW based on the Swift protocol, in step S2, the transmission interface is a RESTful-style object storage interface, which is adapted through the Java client API of the Swift protocol, and the Swift protocol parent interface is the above-mentioned FileSystem abstract class, its file input stream is the FSDataInputStream class, and the file output stream is the FSDataOutputStream class.

[0016] Ceph RGW's object storage interface is compatible with both S3 and OpenStack Swift. This method uses the OpenStack Swift protocol to integrate the Hadoop storage layer with Ceph RGW, enabling Delta Lake to directly access the Ceph RGW storage warehouse, improving access efficiency.

[0017] In the above-mentioned method for interconnecting a data lake and a Ceph RGW based on the Swift protocol, in step S3, the FileSystem abstract class is an abstract base class of a general file system in the Hadoop storage layer. The abstracted project content includes interactive operations on data, files, and directories in the data system. The interactive operation projects specifically include creating, copying, moving, renaming, deleting files and directories, reading and writing files, reading and writing file metadata, and other systematic general operation steps that can be performed on data files and data content.

[0018] In the aforementioned method for interconnecting a data lake and a Ceph RGW using the Swift protocol, in step S5, a MapReduce job is run to analyze the data stored in the Ceph RGW before accessing and performing systematic operations on the data stored in the Ceph RGW storage warehouse. This pre-processed MapReduce job analysis allows for analysis and scheduling of requested data tasks, improving command response speed.

[0019] In the above-mentioned method of interconnecting a data lake with Ceph RGW based on the Swift protocol, the steps for accessing and systematically operating the data stored in the Ceph RGW storage warehouse are as follows:

[0020] S51, Delta Lake sends a request command to the Hadoop-Ceph RGW file system module;

[0021] S52, the Hadoop-Ceph RGW file system module processes commands and matches corresponding submodule functions according to different requests;

[0022] S53. Use the abstract class CephObjectSystem that inherits FileSystem within the Hadoop-Ceph RGW file system to respond to the incoming Delta Lake request for data. It sends a request command to the Ceph RGW storage warehouse according to the Swift protocol standard and performs systematic operations such as cropping, merging, and caching on the returned data.

[0023] S54. The Hadoop-Ceph RGW file system module sends the data after the systematization operation is completed to the Ceph RGW storage warehouse according to the Swift protocol standard between the Hadoop-Ceph RGW storage warehouse and the Ceph RGW storage warehouse.

[0024] The operation process eliminates the transit landing warehouse between the Ceph RGW storage warehouse and Delta Lake, avoiding the secondary transmission of data and command requests and improving work efficiency.

[0025] In the above-mentioned method of interconnecting the data lake and Ceph RGW based on the Swift protocol, when the Hadoop-Ceph RGW file system module receives the request command from Delta Lake, it searches for the corresponding data processing function module according to the different commands.

[0026] In the above-mentioned method of interconnecting a data lake and Ceph RGW based on the Swift protocol, the Ceph RGWstorage Ceph object storage warehouse is used to store the real data entity and provide object data for the request command of the Swift protocol.

[0027] In the above-mentioned method of interconnecting a data lake and Ceph RGW based on the Swift protocol, in the Hadoop storage layer, the input stream class must inherit and implement the FSInputStream abstract class to implement the read and seek methods; the output stream class must be a subclass of the OutputStream abstract class to use the write and flush methods.

[0028] Compared with the existing technology, the advantages of the present invention are: integrating Delta Lake and CephRGW through the Swift protocol, and establishing a Hadoop-Ceph RGW file system module with a CephObjectSystem subclass module in Hadoop, using the CephObjectSystem subclass module to receive request commands from Delta Lake and obtain data from the Ceph RGW storage warehouse through an adapted transmission interface and send it back to Delta Lake, thereby realizing direct connection access between Delta Lake and CephRGW, improving work efficiency and saving hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the method of the present invention;

[0030] Figure 2 This is a hierarchical flow chart of data acquisition of the present invention

[0031] Figure 3 This is a hierarchical flow chart of data acquisition in the traditional transit landing warehouse;

[0032] Figure 4 This is a flowchart of the authentication process for accessing the Ceph RGW storage warehouse in the present invention;

[0033] Figure 5 This is a speed test chart of Delta Lake writing data directly to and from the Ceph RGW storage warehouse in the present invention;

[0034] Figure 6 This is a test chart of the speed of writing and writing data to and from the transit landing warehouse by Delta Lake in the present invention;

[0035] Figure 7 This is a comparison chart of the writing speed of Delta Lake for Ceph RGW storage warehouse and transit landing warehouse in this invention; DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, a method for interconnecting a data lake and a Ceph RGW based on the Swift protocol includes the following steps:

[0038] S1. Create Hadoop-Ceph RGW file system module;

[0039] S2. Build the transmission interface and adapt it;

[0040] S3. Create a class that inherits and implements the FileSystem abstract class;

[0041] S4. Connect Ceph RGW and Delta Lake using the adapted interface.

[0042] S5. Access and systematize data stored in the Ceph RGW storage warehouse through the FileSystem abstract class.

[0043] In step S1, the Hadoop-Ceph RGW file system module receives a request command from Delta Lake.

[0044] This method creates a Hadoop-Ceph RGW file system module in the Hadoop storage layer to receive Delta Lake's request commands, builds a transmission interface to enable direct data connection between Delta Lake and the Ceph RGW storage warehouse, and uses the FileSystem abstract class to implement direct operations on the data stored in the Ceph RGW storage warehouse.

[0045] Furthermore, a CephObjectSystem subclass module is provided in the Hadoop-Ceph RGW file system module, and the CephObjectSystem subclass module is used as the total entry and adaptation layer for receiving Delta Lake request commands.

[0046] In step S2, the transmission interface is a RESTful type object storage interface, which is adapted through the Java client API of the Swift protocol, and the Swift protocol parent interface is the above-mentioned FileSystem abstract class, its file input stream is the FSDataInputStream class, and the file output stream is the FSDataOutputStream class.

[0047] The Swift Java client API is called java-cloudfiles. The FilesClient class provides various operations for object storage in the Swift protocol, including logging into Swift, creating and deleting accounts, containers, and objects, obtaining metadata about accounts, containers, and objects, and reading and writing objects.

[0048] Use SwiftFileSystem, which inherits the hadoop-FileSystem abstract class, as an adapter to perform interface adaptation, that is, converting the interface of a class into another interface that the user program needs to use.

[0049] In step S3, the FileSystem abstract class is an abstract base class of a general file system in the Hadoop storage layer. The abstracted project content includes interactive operations on data, files and directories in the data system, and the interactive operation projects specifically include creating, copying, moving, renaming, deleting files and directories, reading and writing files, reading and writing file metadata, and other systematic general operation steps that can be executed on data files and data content.

[0050] In step S5, the data stored in the Ceph RGW storage warehouse is analyzed by running a MapReduce job before accessing and systematizing the data stored in the Ceph RGW storage warehouse.

[0051] Users can specify the URI of the Amazon S3 storage system for the fs.default.name property in the core-site.xml configuration file of the Hadoop storage layer, so that the Hadoop storage layer can access Amazon S3 and run MapReduce jobs on it.

[0052] like Figure 2 As shown in the figure, the steps to access and systematically operate the data stored in the Ceph RGW storage warehouse are:

[0053] S51, Delta Lake sends a request command to the Hadoop-Ceph RGW file system module;

[0054] S52, Hadoop-Ceph RGW file system performs command processing and matches corresponding submodule functions according to different requests;

[0055] S53. Use the abstract class CephObjectSystem that inherits FileSystem within the Hadoop-Ceph RGW file system module to respond to the incoming Delta Lake request for data. It sends a request command to the Ceph RGW storage warehouse according to the Swift protocol standard and performs systematic operations such as cropping, merging, and caching on the returned data.

[0056] S54. The Hadoop-Ceph RGW file system module sends the data after the systematization operation is completed to the Ceph RGW storage warehouse according to the Swift protocol standard between the Hadoop-Ceph RGW storage warehouse and the Ceph RGW storage warehouse.

[0057] When transferring access to the Ceph RGW storage warehouse, file transfer storage depends largely on the platform's excellence, and in some cases, different platforms may cause path matching errors. The operation flow chart is as follows Figure 3 As shown;

[0058] When accessing the Ceph RGW storage warehouse, identity authentication and key verification are required. The verification process is as follows: Figure 4 As shown;

[0059] Delta Lake directly writes data to the Ceph RGW storage warehouse. This is only used as a test comparison under the same environment, not actual production performance. The speed test results are as follows: Figure 5 As shown;

[0060] Delta Lake writes and writes data to and from the transit warehouse (using the Windows 11 local file system as an example) as follows: Figure 6 shown.

[0061] according to Figure 7 As shown, the speed at which Delta Lake directly writes data from the Ceph RGW storage warehouse and the amount of data read in the same time unit are greater than the speed at which Delta Lake writes data from the transit landing warehouse. This shows that when data from the Ceph RGW cluster is exported to the transit landing warehouse, and then imported into Delta Lake for data entry and subsequent MapReduce operations, the time required is significantly greater than the time required for Delta Lake to directly read and write data from the Ceph RGW storage warehouse.

[0062] In detail, when the Hadoop-Ceph RGW file system module receives a request command from Delta Lake, it searches for the corresponding data processing function module based on the command.

[0063] The data types between the object storage warehouse and Delta Lake are in a one-to-many relationship, so the data types not supported by Delta Lake are uniformly transmitted as files, such as Text and Parquet files.

[0064] Specifically, Ceph RGW storage Ceph's object storage warehouse is used to store the real data entity and provide object data for Swift protocol request commands.

[0065] In the Hadoop storage layer, input stream classes must inherit and implement the FSInputStream abstract class to implement the read and seek methods; output stream classes must be subclasses of the OutputStream abstract class to implement the write and flush methods.

[0066] To sum up, the principle of this embodiment is that Delta Lake mainly uses the storage service in the Hadoop storage layer to provide it with the ability to land on other object storage platforms. Therefore, this method adopts the method of adding support for Ceph RGW to the Hadoop storage layer to directly connect Delta Lake and Ceph RGW to realize data systematization operation, that is, creating a Hadoop-Ceph RGW file system module with a CephObj ectSystem subclass module in the Hadoop storage layer, using the CephObj ectSystem subclass module to receive request commands from Delta Lake and obtain data from the Ceph RGW storage warehouse through the adapted transmission interface and send it back to Delta Lake, thereby realizing direct connection access between Delta Lake and Ceph RGW.

[0067] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A method for interconnecting a data lake and a Ceph RGW based on the Swift protocol, characterized in that: This method comprises the following steps: S1. Create Hadoop-Ceph RGW file system module; S2. Build the transmission interface and adapt it; S3. Create a class that inherits and implements the FileSystem abstract class; S4. Connect Ceph RGW and Delta Lake using the adapted interface. S5. Access and systematize data stored in the Ceph RGW storage warehouse through the FileSystem abstract class; The Hadoop-Ceph RGW file system module is provided with a CephObject System subclass module, which serves as the main entry and adaptation layer for receiving Delta Lake request commands; In step S2, the transmission interface is a RESTful object storage interface, which is adapted through the Java client API of the Swift protocol, and the Swift protocol parent interface is the above-mentioned FileSystem abstract class, whose file input stream is the FSDataInputStream class, and whose file output stream is the FSDataOutputStream class; In step S3, the FileSystem abstract class is an abstract base class of a general file system in the Hadoop storage layer. The abstracted project content includes interactive operations on data, files and directories in the data system. The interactive operation projects specifically include creating, copying, moving, renaming, deleting files and directories, reading and writing files, reading and writing file metadata, and other systematic general operation steps that can be performed on data files and data content; The steps to access and systematically operate on data stored in Ceph RGW storage warehouse are: S51, Delta Lake sends a request command to the Hadoop-Ceph RGW file system module; S52, the Hadoop-Ceph RGW file system module processes commands and matches corresponding submodule functions according to different requests; S53. Use the abstract class CephObjectSystem that inherits FileSystem within the Hadoop-Ceph RGW file system module to respond to the incoming Delta Lake request for data, issue a request command to the CephRGW warehouse according to the Swift protocol standard, and perform systematic operations on the returned data such as cropping, merging, and caching. S54, the Hadoop-Ceph RGW file system module sends the data after the systematization operation is completed to the Ceph RGW storage warehouse according to the Swift protocol standard between the Hadoop-Ceph RGW storage warehouse and the Ceph RGW storage warehouse.

2. The method for interconnecting a data lake and a Ceph RGW based on the Swift protocol according to claim 1, characterized in that: In step S1, the Hadoop-Ceph RGW file system module receives a request command from Delta Lake.

3. The method for interconnecting a data lake and a Ceph RGW based on the Swift protocol according to claim 1, characterized in that: In step S5, before accessing and systematically operating the data stored in the Ceph RGW storage warehouse, the data stored in the Ceph RGW is analyzed by running a MapReduce job.

4. The method for interconnecting a data lake and a Ceph RGW based on the Swift protocol according to claim 1, characterized in that: When the Hadoop-Ceph RGW file system module receives a request command from Delta Lake, it searches for the corresponding data processing function module based on the command.

5. The method for interconnecting a data lake and a Ceph RGW based on the Swift protocol according to claim 1, characterized in that: The object storage warehouse of Ceph RGW storage ceph is used to store the real data entity and provide object data for the request command of the Swift protocol.

6. The method for interconnecting a data lake and a Ceph RGW based on the Swift protocol according to claim 5, characterized in that: In the Hadoop storage layer, the input stream classes need to inherit and implement the FSInputStream abstract class to implement the read and seek methods; the output stream classes need to be subclasses of the OutputStream abstract class to implement the write and flush methods.

Images