A data synchronization method and device based on HBase and a storage medium

Abstract

The application discloses a kind of based on HBase's data synchronization method, device and storage medium, it is related to big data field.The method comprises the following steps: receiving user operation index list corresponding to index update;Receive HBase data and synchronize to public region server;Indexer indexes data in the public region server according to index list, and obtains index data;According to the index list after updating, the index data is filtered, and effective data stream is output, and the effective data stream is written into ES.The application is synchronized to data by public region server, and it is conducive to the index of data, can substantially reduce memory, CPU and thread resource occupation, improve the robustness of indexer component under high load scenario, improve product competitiveness.

Description

Technical Field

[0001] This invention relates to the field of big data technology, specifically to a data synchronization method, apparatus, and storage medium based on HBase. Background Technology

[0002] Currently, HBase, as a commonly used distributed database in the big data field, supports data storage of hundreds of billions of rows and millions of columns. However, HBase itself only provides queries based on row keys and full table scans. When multiple conditions are combined in a query, it can only be done through a full table scan, which is inefficient. Multiple condition combined queries are quite common in big data applications. Our product's Indexer is based on the open-source hbase-indexer, which modifies certain columns from the HBase table to index into Elasticsearch. This does not introduce intrusive code into HBase and does not affect write performance. This requirement can improve the efficiency of HBase secondary indexes and meet the needs of full-text indexing scenarios. In scenarios with large volumes of data and multiple query combinations, query time can be reduced from hours to seconds, significantly improving the performance of combined queries. However, due to the relatively inactive open-source HBase-indexer community in recent years, some core design logic of the Indexer component has not been optimized for a long time, limiting the performance of the Indexer component.

[0003] In practical use, the following problems exist: Each table's index simulates a regionserver service to parse the WAL files of all tables written to in HBase, but only retains the write records of the table corresponding to that index. The number of simulated regionservers for data synchronization and parsing is equal to the number of secondary index tables. Under high load, this can exhaust machine resources, leading to the Indexer component becoming unavailable and significantly impacting business operations. Summary of the Invention

[0004] To address at least one of the problems mentioned in the background art, this invention provides a data synchronization method, apparatus, and storage medium based on HBase. By synchronizing data through a public region server, it facilitates data indexing, significantly reduces memory, CPU, and thread resource consumption, improves the robustness of the Indexer component under high-load scenarios, and enhances product competitiveness.

[0005] The specific technical solutions provided by the embodiments of the present invention are as follows:

[0006] Firstly, an HBase-based data synchronization method is provided, the method comprising:

[0007] Receive user-operated indexes and update the corresponding index list;

[0008] Receive HBase data and synchronize it to the public region server;

[0009] The indexer indexes the data in the public region server according to the index list to obtain index data;

[0010] The index data is filtered according to the updated index list, a valid data stream is output, and the valid data stream is written into Elasticsearch (ES).

[0011] Furthermore, a public region server is constructed, and the HBase data is synchronized using the public region server.

[0012] Furthermore, before writing the valid data stream into the ES, the method further includes converting the valid data stream into a data format recognizable by the ES.

[0013] Furthermore, it also includes: the user operation index includes at least one or more of the following: adding, deleting, and updating the index.

[0014] Furthermore, it receives the index list corresponding to user operation index update, specifically including:

[0015] Receive user operations on the index and automatically record the index information in ZooKeeper;

[0016] The index information in ZooKeeper is obtained through the Indexer, and the index information is updated in the index list.

[0017] Furthermore, receiving HBase data and synchronizing it to the public region server specifically includes:

[0018] Receive all WAL logs from HBase in real time and synchronize the WAL log data to the public region server.

[0019] Furthermore, the index data is filtered according to the updated index list to output a valid data stream, specifically including:

[0020] The WAL log data is parsed to obtain identifiable data;

[0021] The identifiable data is filtered based on the table names and column names in the index list;

[0022] Output the valid data stream.

[0023] Furthermore, the identifiable data is a single data entry.

[0024] Secondly, an HBase-based data synchronization device is provided, the device comprising:

[0025] The index registration module is used to receive the list of indexes corresponding to user operations on index updates.

[0026] The data synchronization module is used to receive HBase data and synchronize it to the public region server;

[0027] The data indexing module is used by the Indexer to index the data in the public region server according to the index list to obtain index data;

[0028] The data filtering module is used to filter the index data according to the updated index list, output a valid data stream, and write the valid data stream into ES.

[0029] Thirdly, a computer-readable storage medium is provided, storing computer-executable instructions, wherein the instructions are used to execute the HBase-based data synchronization method described above.

[0030] The embodiments of the present invention have the following beneficial effects:

[0031] 1. In this embodiment, a public region server is constructed. Leveraging the characteristics of the public region server, it receives all WAL log data from HBase. Then, based on user operations such as adding, deleting, and updating indexes, the corresponding index list is updated. The data in the public region server is indexed according to the index list to obtain index data. Then, using the updated index list as a filtering condition, the index data is parsed and filtered, outputting a valid data stream, which is then written to Elasticsearch (ES). This approach significantly reduces memory, CPU, and thread resource consumption, improves the robustness of the Indexer component under high load scenarios, and enhances product competitiveness.

[0032] 2. When a user performs an indexing operation, the index information is automatically recorded in ZooKeeper. The indexer retrieves the index information from ZooKeeper and updates the index list with the index information, thereby registering the filtering information.

[0033] 3. Filter the index data according to the updated index list. That is, parse the synchronized WAL log data into recognizable single data entries, filter it according to the table name and column name in the index list, output only the data stream that needs to be indexed, and convert the index data into a data format that Elasticsearch can recognize and write it into Elasticsearch. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A schematic diagram illustrating the HBase-based data query method in this application;

[0036] Figure 2 A schematic diagram illustrating the parsing and filtering process in step S4 of this application is shown.

[0037] Figure 3 Exemplary system diagrams used to illustrate the various embodiments described in this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0039] HBase is an open-source NoSQL database suitable for storing and managing various unstructured and semi-structured loose data. Using HBase database technology, large-scale storage clusters can be built on server clusters to meet the storage needs of big data.

[0040] Elasticsearch (ES) is an open-source, distributed RESTful search engine built on Lucene that supports indexing data using JSON over HTTP.

[0041] Indexer: An HBase secondary indexing tool.

[0042] Secondary indexes: a solution to compensate for the limitations of HBase's own multi-condition combined queries.

[0043] Currently, HBase, as a commonly used distributed database in the big data field, supports data storage of hundreds of billions of rows and millions of columns. However, HBase itself only provides queries based on row keys and full table scans. When multiple conditions are combined in a query, it can only be done through a full table scan, which is inefficient. Multiple condition combined queries are quite common in big data applications. Our product's Indexer is based on the open-source hbase-indexer, which modifies certain columns from the HBase table to index into Elasticsearch. This does not introduce intrusive code into HBase and does not affect write performance. This requirement can improve the efficiency of HBase secondary indexes and meet the needs of full-text indexing scenarios. In scenarios with large volumes of data and multiple query combinations, query time can be reduced from hours to seconds, significantly improving the performance of combined queries. However, due to the relatively inactive open-source HBase-indexer community in recent years, some core design logic of the Indexer component has not been optimized for a long time, limiting the performance of the Indexer component.

[0044] In practical use, the following problems exist: Each table's index simulates a regionserver service to parse the WAL files of all tables written to in HBase, but only retains the write records of the table corresponding to that index. The number of simulated regionservers for data synchronization and parsing is equal to the number of secondary index tables. Under high load, this can exhaust machine resources, rendering the Indexer component unavailable and significantly impacting business operations. To address these issues, this application proposes an HBase-based data synchronization method, device, and storage medium that significantly reduces memory, CPU, and thread resource consumption, improves the robustness of the Indexer component under high load scenarios, and enhances product competitiveness.

[0045] Example 1

[0046] like Figure 1 This is a flowchart of a data query method based on HBase. This HBase data synchronization method can be executed on the server side; optionally, in this embodiment, it can be executed by a computer device. The method specifically includes the following steps:

[0047] Step S1:

[0048] Receive the list of indexes corresponding to user operations that update the indexes.

[0049] Specifically, the Indexer receives user-operated indexes, automatically records the index information in ZooKeeper, retrieves the index information from ZooKeeper through the Indexer, and updates the index list with the index information. User-operated indexes include at least one or more of the following: adding, deleting, and updating indexes.

[0050] Specifically, the Indexer uses ZooKeeper as its metadata store; all index metadata information is stored in ZooKeeper. Based on user operations on the indexes, the index information is automatically recorded in ZooKeeper. The Indexer retrieves the index information from ZooKeeper and updates the index list accordingly. When indexes are added, deleted, or updated—meaning that some tables or columns in HBase need to be indexed or removed—these operations must be updated in real-time to the service, thereby updating the filtering conditions.

[0051] Furthermore, when users add, delete, or update the index list, the backend automatically records the corresponding index information in ZooKeeper. For example, table t1 is filtered for column f1; table t2 is filtered for column f2; table t3 is added and filtered for column f2, etc. The indexer obtains the index information from ZooKeeper in real time and updates the filtering conditions. After obtaining the index information from ZooKeeper, it queries the real-time WAL log data to obtain: the f1 data of table t1; the f1 data of table t2; the f2 data of table t3, etc.

[0052] For example, when a user adds a new index for "gender", a column corresponding to "gender" will be added to the index list. When a user modifies one of the indexes, such as changing "gender" to "name", the column corresponding to "gender" will be deleted from the index list, and a column corresponding to "name" will be added.

[0053] The Indexer in this embodiment is a secondary indexing tool based on HBase. It is a modification of HBase-indexer. While HBase-indexer in related technologies uses the Solr search engine for secondary index queries, the Indexer provided in this embodiment builds secondary index queries based on the Elasticsearch (ES) search engine. It utilizes the ES search engine for real-time distributed search and analysis of the HBase database, improving the data query efficiency of multi-condition combined queries on the HBase database. In this embodiment, the Indexer receives and parses user-input combined query commands.

[0054] In this embodiment, the user inputs a query statement through the client's display interface. Optionally, the display interface has a query statement input window where the user can input a query statement. This query statement can be a single-condition query statement or a combined query command line with multiple conditions. This embodiment uses an index list as an example for illustration. After the user inputs the index list, a confirmation command is triggered, sending the index list to the Indexer. The index list may include: logical operators for combined queries, fields to be queried, specified output columns, and other information.

[0055] Step S2:

[0056] The system receives HBase data and synchronizes it to a public region server. First, a public region server is built, and then used to synchronize the HBase data.

[0057] Specifically, the constructed public region server leverages HBase replication features. HBaseReplication is handled by a separate thread within the RegionServer, operating in a parallel and asynchronous manner. This mechanism makes indexing HBase data more secure. Furthermore, the near real-time index synchronization of replication offers high flexibility and scalability.

[0058] Furthermore, the public region server receives all WAL log data from HBase in real time and synchronizes the WAL log data to the public region server.

[0059] In this open-source component, one index table corresponds to one region server. This application modifies the Indexer architecture and adjusts parameter configurations to ensure all index data is retrieved from a single public region server. Leveraging HBase replication, the public region server receives all WAL logs from HBase in real time and synchronizes them to the public region server. HBase contains multiple region servers, and all tables requiring indexing are evenly distributed across them. By adjusting parameter configurations, WAL logs from all HBase region servers are simultaneously sent to the Indexer's public region server. Based on the maintained index list, the Indexer outputs index data for all indexed tables from the public region server.

[0060] Step S3:

[0061] The indexer indexes the data in the public region server based on the index list to obtain the indexed data.

[0062] Specifically, users index the index list using the Indexer in the client. The client obtains the index information of the index list, sends a data query request to the public region server to index the data according to the index information, and retrieves the corresponding index data from the public region server's WAL log data.

[0063] Step S4:

[0064] like Figure 2 As shown, the index data is filtered based on the updated index list, outputting a valid data stream, which is then written to Elasticsearch (ES). First, the valid data stream is converted into ES-recognizable data; specifically, recognizable data consists of individual data entries. Then, each individual data entry is filtered based on the table and column names in the index list, outputting only the data stream needed for index building. Finally, the index data is converted into an ES-recognizable data format and written to ES.

[0065] Specifically, this includes index lists for companies A, B, C, and D. Taking company A's index list as an example, the list includes multiple indexes such as name, date of birth, gender, and year of employment. "Name" is identified as the index to be filtered. Single records indexed by "Name" are parsed, and then filtered using "Company A" as the table name and "Name" as the column name, outputting a data stream indexed by "Name". This data stream is then converted to a format recognizable by Elasticsearch (ES). A write request is sent to ES, which receives and validates the request. If validation passes, the data stream is written to ES.

[0066] The specific order of steps S1, S2 and S3 is not limited, and those skilled in the art can adjust it according to the actual situation.

[0067] Example 2

[0068] Corresponding to the above embodiments, this application provides a data synchronization device based on HBase, the device comprising:

[0069] The index registration module receives a list of indexes corresponding to user index updates. This list includes at least one or more updates, such as additions, deletions, and general updates. Specifically, the Indexer client is modified to receive user-mandated indexes and synchronize these changes to the index list stored in ZooKeeper.

[0070] The data synchronization module is used to receive HBase data and synchronize it to the public region server to generate synchronized data. Specifically, it first builds the public region server, and then uses the replication feature of the public region server and specific parameter settings to receive and synchronize all WAL log data from HBase.

[0071] The data indexing module is used by the Indexer to parse the data in the public region server based on the index list to obtain index data. Specifically, the user indexes the index list using the Indexer in the client. The client obtains the index information of the index list, sends a data query request to the public region server to index the data according to the index information, and retrieves the corresponding index data from the public region server's WAL log data.

[0072] The data filtering module is used to parse and filter the index data according to the updated index list, output a valid data stream, and write the valid data stream into Elasticsearch (ES). Specifically, the valid data stream is first converted into single data entries recognizable by ES, then filtered according to the table names and column names in the index list, outputting only the data stream needed for index building, and finally converted into a data format recognizable by ES before being written into ES.

[0073] Figure 3 Exemplary systems that can be used to implement the various embodiments described in this application are shown.

[0074] like Figure 3 As shown, in some embodiments, the system can function as any of the aforementioned devices for traffic noise control in the various embodiments. In some embodiments, the system may include one or more computer-readable media (e.g., system memory or NVM / storage device) having instructions and one or more processors (e.g., one or more processors) coupled to the one or more computer-readable media and configured to execute the instructions to implement the module and thus perform the actions described in this application.

[0075] In one embodiment, the system control module may include any suitable interface controller to provide any suitable interface to at least one of the processors(s) and / or any suitable device or component communicating with the system control module.

[0076] The system control module may include a memory controller module to provide an interface to the system memory. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0077] System memory can be used, for example, to load and store data and / or instructions for the system. In one embodiment, system memory may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory may include Double Data Rate Type Quad Synchronous Dynamic Random Access Memory (DDR4 SDRAM).

[0078] In one embodiment, the system control module may include one or more input / output (I / O) controllers to provide interfaces to the NVM / storage device and (one or more) communication interfaces.

[0079] For example, an NVM / storage device can be used to store data and / or instructions. An NVM / storage device may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0080] Example 3

[0081] This invention provides a computer-readable storage medium storing computer-executable instructions for executing the HBase-based data synchronization method described above.

[0082] The computer-executable instructions implement the following steps during execution:

[0083] Receive the list of indices corresponding to user operations that update indices;

[0084] Build a public region server, and receive HBase data through the public region server and perform synchronization with it;

[0085] The indexer parses the data in the public region server based on the index list to obtain the index data;

[0086] The index data is parsed and filtered according to the updated index list, and a valid data stream is output and written into Elasticsearch.

[0087] Specifically, the Indexer uses ZooKeeper as its metadata store; all index metadata information is stored in ZooKeeper. Based on user operations on the indexes, index information is automatically recorded in ZooKeeper. The Indexer retrieves the index information from ZooKeeper and updates the index list accordingly. When indexes are added, deleted, or updated—meaning some tables or columns in HBase need to be indexed or removed—these operations must be updated in real-time to update the filtering conditions. A public region server is built to receive HBase data and synchronize it. The public region server receives all WAL log data from HBase in real-time and synchronizes the WAL log data to itself.

[0088] The indexer indexes the data in the public region server based on the index list to obtain indexed data. Specifically, the user indexes the index list using the indexer in the client. The client obtains the index information of the index list, sends a data query request to the public region server to index the data according to the index information, and retrieves the corresponding indexed data from the WAL log data of the public region server.

[0089] The index data is parsed and filtered according to the updated index list to output a valid data stream. Specifically, the valid data stream is first converted into single data entries recognizable by Elasticsearch (ES), then filtered based on the table and column names in the index list, outputting only the data stream needed for index building. Finally, the index data is converted into a data format recognizable by ES and written to ES.

[0090] In this embodiment, a computer-readable storage medium may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, a computer-readable storage medium includes, but is not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other currently known media or those developed hereafter capable of storing computer-readable information / data for use by a computer system.

[0091] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

[0092] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A data synchronization method based on HBase, characterized in that, The methods include: When the user enters the index list, a confirmation command is triggered, and the index list is sent to the Indexer. The index list includes at least the logical operators for the combined query, the fields to be queried, and the specified output columns. Receive user operation indexes and update the corresponding index list, including: receiving user operation indexes, automatically recording index information in ZooKeeper, obtaining index information in ZooKeeper through the indexer, and updating the index list with the index information. Here, the indexer uses ZooKeeper as the metadata storage, and all index metadata information is stored in ZooKeeper. Build a public region server; Receive HBase data and synchronize it to the public region server, including: receiving all WAL log data from HBase in real time and synchronizing the WAL log data to the public region server; The indexer indexes data in the public region server based on the index list to obtain index data, including: obtaining the index information of the index list, sending a data query request to the public region server to index the data according to the index information, and indexing the corresponding index data from the WAL log data of the public region server. The index data is filtered according to the updated index list, and a valid data stream is output, including: parsing the WAL log data to obtain identifiable data, filtering the identifiable data according to the table name and column name in the index list, and outputting a valid data stream, wherein the identifiable data is a single data record; Convert the valid data stream into a data format that Elasticsearch can recognize, and write the valid data stream into Elasticsearch.

2. The method according to claim 1, characterized in that, Also includes: The user operation index includes at least one or more of the following: adding, deleting, and updating the index.

3. A data synchronization device based on HBase, used to implement the method described in any one of claims 1 or 2, characterized in that, The device includes: The index registration module is used to receive the list of indexes corresponding to user operations on index updates. The data synchronization module is used to receive HBase data and synchronize it to the public region server; The data indexing module is used by the Indexer to index the data in the public region server according to the index list to obtain index data; The data filtering module is used to filter the index data according to the updated index list, output a valid data stream, and write the valid data stream into ES.

4. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer-readable storage medium and executable instructions are used to perform the HBase-based data synchronization method as described in any one of claims 1 or 2.

Images