Method for synchronizing hbase data to qianbase mpp in real time

Abstract

The application discloses a kind of HBase data real-time synchronization to QianBase MPP method, belong to database real-time synchronization field, obtain Snapshot metadata in HBase database and real Snapshot data, and the change data of WAL in HBase database, then it is converted into the data format supported by QianBase MPP database.The QianBase MPP data file obtained by conversion is sent to the corresponding Segment according to the corresponding relationship.The data is synchronized from HBase database to QianBase MPP database in real time by using the combination of HBase snapshot and WAL pre-write log, and the synchronization process is not affected by HBase Compaction and hotspot problem, and the performance loss caused by the fact that data writing must pass through QianBase MPP Master is also avoided, so that the server of database synchronization is obviously improved in ease of use and performance.

Description

Technical Field

[0001] This invention relates to the field of real-time database synchronization, and more specifically, to a method for real-time synchronization of HBase data to QianBase MPP. Background Technology

[0002] QianBase MPP is a relational database designed for data warehouse applications, boasting significant advantages in data storage, high concurrency, high availability, linear scalability, response speed, ease of use, and cost-effectiveness. The database architecture comprises three layers: the client service layer, the SQL database service layer, and the storage engine layer. The first layer is the client service layer, where the application resides. Applications can be written by users or implemented using third-party ISV tools / solutions. You can access the QianBase database service layer using the Windows or Linux client drivers provided by QianBase, through standard ODBC / JDBC interfaces. QianBase supports type2 JDBC, type4 JDBC, and ADO.NET drivers. You can choose the appropriate driver type based on your specific needs (response time, number of connections, security requirements, and other factors). The second layer is the SQL database engine layer. This layer includes all QianBase services, encapsulating all services for managing QianBase objects and efficiently executing SQL database requests. Services include connection management, SQL statement compilation and creation of optimal execution plans, SQL execution (serial and parallel), transaction management, and workload management. The third layer is the storage engine layer, which includes the standard Hadoop services (HDFS and Zookeeper) used by QianBase. QianBase objects are stored in a native Hadoop database structure, including HBase, cached text files, and key-value sequence files. QianBase processes SQL requests from applications and transparently translates these requests into native interface calls required for the underlying data format. QianBase provides a relational schema abstraction on top of HBase, so it can support traditional relational database objects (tables, views, secondary indexes) using familiar DDL / DML syntax (object naming, column definitions, and data type support). Additionally, QianBase supports using native HBase and Hive tables as its external tables.

[0003] HBase is a column-oriented distributed database. When synchronizing data from HBase to QianBase MPP, the mainstream approach is to call the HBase API to read the data and then use JDBC to write it to QianBase MPP. Real-time HBase synchronization solutions typically call the HBase API when retrieving data, acting as HBase clients. In extreme cases, if all read requests fall on a few regions of a single RegionServer, it can cause significant resource consumption on that RegionServer (e.g., IO exhaustion, handler exhaustion), severely impacting other services on that RegionServer. HBase's LSM architecture results in each Store containing a small number of HFile files. More files mean more IO operations are required for retrieval, reducing read efficiency. Too many small files can trigger compaction, affecting or even blocking client data reading. Data submission to QianBase MPP primarily uses JDBC. Improper database connection usage can waste database resources and impact performance. When writing data to QianBase MPP, it is necessary to first connect to the Master, and then the Master distributes the data to each Segment. This can create a bottleneck in some scenarios where timeliness is critical. Summary of the Invention

[0004] One of the objectives of this invention is to address the aforementioned shortcomings by providing a method for real-time synchronization of HBase data to QianBaseMPP, in order to solve the technical problems that similar data synchronization methods can easily lead to significant consumption of server resources and bottlenecks in certain scenarios where timeliness is required.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] The present invention provides a method for real-time synchronization of HBase data to QianBase MPP, the method comprising the following steps:

[0007] Step A: Obtain the Snapshot metadata from the HBase database, then obtain the actual Snapshot data from the HBase database's HDFS storage system, and finally convert the Snapshot metadata and the Snapshot data into a data format supported by the QianBase MPP database.

[0008] Step B: Obtain changes to the WAL in the HBase database. Whenever a WAL changes, read the information in the WAL and then convert the real-time change information of the WAL into a data format supported by the QianBase MPP database.

[0009] Step C: Obtain the metadata of the QianBase MPP database, and combine the Snapshot metadata with the Snapshot data and the real-time change information of the WAL to generate a correspondence between the QianBase MPP database Segment and the corresponding Segment. Then, according to the correspondence, send the converted QianBase MPP data file to the corresponding Segment.

[0010] As a preferred option, a further technical solution is that the aforementioned Snapshot metadata includes table schema information, RegionInfo information for all Regions of the original table, and HFile information for all Regions.

[0011] A further technical solution is: the above correspondence is generated from the metadata information of the QianBase MPP database, the real-time change information of the Snapshot metadata, Snapshot data and WAL, and the converted QianBase MPP data.

[0012] A further technical solution is as follows: The above method also includes storing Snapshot data source queue information, WAL real-time change information, and the running status of synchronization tools in Zookeeper.

[0013] A further technical solution is that the above method also includes caching the Snapshot metadata in the HBase database and the metadata in the QianBase MPP database.

[0014] A further technical solution is that the method is executed by a synchronization tool, which is used to execute the SnapshotSource thread, the WALSource thread, and the data conversion thread.

[0015] Compared with existing technologies, one of the advantages of this invention is that by using a combination of HBase snapshots and WAL write-ahead logs, data is synchronized from the HBase database to the QianBase MPP database in real time. The synchronization process is not affected by HBase compaction and hotspot issues, and the performance loss caused by data writing having to go through QianBase MPPMaster is also avoided. This results in a significant improvement in both the ease of use and performance of the database synchronization server. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method according to an embodiment of the present invention. Detailed Implementation

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] like Figure 1 As shown, HBase is a column-oriented distributed database that uses snapshots to migrate and copy HBase data. HDFS is a distributed file system suitable for running on commodity hardware. WAL (Write-Ahead Log) is a log used by HBase RegionServers to record operations during data insertion and deletion, which can be used for data recovery in case of node failure. QianBase MPP is a relational database for data warehouse applications, with outstanding advantages in data storage, high concurrency, high availability, linear scalability, response speed, ease of use, and cost-effectiveness.

[0019] As described in the background section of this invention, existing HBase real-time synchronization solutions generally call the APIs provided by HBase when retrieving HBase data, acting as HBase clients. In extreme cases, if all read requests fall on a few regions of a single RegionServer, it can cause significant resource consumption on that RegionServer (e.g., IO exhaustion, handler exhaustion), severely impacting other services on that RegionServer. HBase's LSM architecture results in each Store containing a small number of HFile files; the more files, the more IO operations are required for retrieval, reducing read efficiency. Too many small files can trigger compaction, affecting or even blocking client data reading. Furthermore, data submission to QianBase MPP often uses JDBC. Improper database connection usage can waste database resources and impact performance. Data writing to QianBase MPP requires first connecting to the Master, which then distributes the data to each Segment, creating bottlenecks in scenarios with time-sensitive requirements.

[0020] Therefore, in this embodiment, the aim is to obtain data without calling the HBase client API, process the data, and then directly write the data into the corresponding Segment of QianBase MPP. To achieve the aforementioned technical objective, the technical solution adopted is as follows: First, generate the data file of the data supported by QianBase MPP. This step mainly targets the HBase Snapshot information and WAL write-ahead log information.

[0021] Specifically, the module parses the snapshot into a data file supported by QianBase MPP and sends it to the corresponding segment. This module retrieves all existing data after task submission. The synchronization tool starts a SnapshotSource thread, which first obtains the snapshot's metadata (table schema information, RegionInfo information for all Regions of the original table, and information for all HFiles under the Region (such as file location on HDFS, file block size, file permissions, etc.)). Based on this information, it then retrieves the actual snapshot data from HDFS. The synchronization tool then starts a dedicated Snapshot file parsing and conversion thread to parse and convert this data into a data format supported by QianBase MPP.

[0022] Correspondingly, the WAL is parsed into a data file supported by QianBase MPP and sent to the corresponding Segment. This module is responsible for obtaining real-time changes to the HBase WAL during task execution. The synchronization tool starts a WALSource daemon thread to monitor changes in the HBase WAL. It records the WAL queue that needs to be synchronized, and continuously reads the contents of the WAL whenever a change occurs. The WALSource can also be configured with filtering conditions to exclude changes to certain specific data. A dedicated WAL file parsing and conversion thread is responsible for converting the received WAL into a data format supported by QianBase MPP.

[0023] Secondly, the transformed data files are submitted to QianBase MPP. The synchronization tool obtains the metadata information from QianBase MPP, and then combines it with the data source information and the processed data information to generate the mapping relationship between the data files and segments, and sends the data files to the corresponding segments.

[0024] Secondly, the synchronization tool stores various types of information in ZooKeeper. To prevent rereads or incorrect reads that might occur if the synchronization tool unexpectedly exits and restarts during the reading of snapshots and WAL data, the tool stores its read offset and data source queue information in ZooKeeper. It also writes the synchronization tool's own state, data write status, and QianBase MPP status to ZooKeeper to avoid duplicate or missing writes.

[0025] Finally, metadata caching.

[0026] To avoid reading metadata multiple times and speed up metadata access, HBase and QianBaseMPP metadata can be read from Zookeeper and cached in the memory allocated by the synchronization tool. When the metadata changes, the changed metadata information only needs to be updated in Zookeeper.

[0027] In addition to the above, it should be noted that the terms "one embodiment," "another embodiment," and "embodiment" used in this specification refer to specific features, structures, or characteristics described in connection with that embodiment, which are included in at least one embodiment described in the general description of this application. The appearance of the same expression in multiple places in the specification does not necessarily refer to the same embodiment. Furthermore, when a specific feature, structure, or characteristic is described in connection with any embodiment, the intention is to suggest that implementing such a feature, structure, or characteristic in conjunction with other embodiments also falls within the scope of this invention.

[0028] Although the invention has been described herein with reference to several illustrative embodiments, it should be understood that many other modifications and implementations can be devised by those skilled in the art, which will fall within the scope and spirit of the principles disclosed herein. More specifically, various variations and modifications can be made to the components and / or layout of the subject matter arrangement within the scope of the disclosure, drawings, and claims. Besides variations and modifications to the components and / or layout, other uses will be apparent to those skilled in the art.

Claims

1. A method for real-time synchronization of HBase data to QianBase MPP, characterized in that... The method includes the following steps: Obtain the Snapshot metadata from the HBase database, then obtain the actual Snapshot data from the HBase database's HDFS storage system, and then convert the Snapshot metadata and the Snapshot data into a data format supported by the QianBase MPP database. The changes in the WAL in the HBase database are obtained. Whenever the WAL changes, the information in the WAL is read, and then the real-time change information of the WAL is converted into a data format supported by the QianBase MPP database. Obtain the metadata of the QianBase MPP database, and combine the Snapshot metadata with the Snapshot data and the real-time change information of the WAL to generate a correspondence with the QianBase MPP database Segment. Then, send the converted QianBase MPP data file to the corresponding Segment according to the correspondence. The correspondence is generated from the metadata information of the QianBase MPP database, the real-time change information of the Snapshot metadata, the Snapshot data and WAL, and the converted QianBase MPP data; The metadata of the HBase database and QianBase MPP is read from Zookeeper and cached in the memory allocated by the synchronization tool. When the metadata of the HBase database and QianBase MPP changes, it is updated in Zookeeper.

2. The method for real-time synchronization of HBase data to QianBase MPP according to claim 1, characterized in that: The snapshot metadata includes table schema information, RegionInfo information for all Regions in the original table, and HFile information for all Regions.

3. The method for real-time synchronization of HBase data to QianBase MPP according to claim 1, characterized in that... The method further includes storing Snapshot data source queue information, WAL real-time change information, and the running status of synchronization tools in Zookeeper.

4. The method for real-time synchronization of HBase data to QianBase MPP according to claim 1 or 3, characterized in that... The method also includes caching the Snapshot metadata from the acquired HBase database and the metadata from the QianBase MPP database.

5. The method for real-time synchronization of HBase data to QianBase MPP according to claim 1, characterized in that: The method is performed by a synchronization tool that executes the threads of SnapshotSource, WALSource, and data conversion.

Images