HBase memory index construction method, system and storage medium

Abstract

The present invention discloses an HBase memory index construction method, system and storage medium. The HBase memory index construction method comprises: an update step: when an update operation is performed on a data table, an update operation request is intercepted by a coprocessor, and an index table is updated to obtain a secondary index table; a search step: when an index query is performed, a search operation request is intercepted by a coprocessor, and index items that meet the conditions are found from the secondary index table, and the original data table is returned to perform the query. The present invention realizes the construction of a memory index, and persistently stores the constructed memory index in an external memory, realizes the rapid construction of a secondary index by a coprocessor, and can automatically update the index according to the changes in the HBase table. When in use, the index data retrieval speed is greatly improved through memory calculation, and the availability and fault tolerance of the index are guaranteed.

Description

Technical Field

[0001] The present invention relates to a method, system and storage medium for constructing an HBase memory index, and in particular to a method, system and storage medium for constructing an HBase memory index based on a coprocessor. Background Art

[0002] With the rapid development of the Internet, data indexing technology is also developing rapidly. In order to achieve efficient storage and query of massive data, many NoSQL databases have been developed, and HBase is one of them. However, the native HBase database only supports primary key indexes when querying data, and non-primary key data can only be queried by full table scanning, which greatly reduces the multi-condition query speed of HBase.

[0003] The native HBase database only supports primary key indexes when querying data, and non-primary key data can only be queried by full table scan, which greatly reduces the multi-condition query speed of HBase. We studied the general solution of HBase distributed secondary index, introduced the distributed index mechanism, and completed the index management in SolrCloud. With the help of the index function provided by the coprocessor, we created and stored indexes for HBase records, where HBase is responsible for storing data and Solr is responsible for indexing and retrieving data.

[0004] In addition, the prior art also uses secondary indexing solutions based on Solr and HiBase, but the retrieval speed of these two solutions is relatively slow, and when a non-coprocessor solution is used, the storage overhead is relatively large, especially when there are many index columns, the space overhead will be even greater.

[0005] Therefore, after intensive research, the inventors have developed a coprocessor-based HBase memory index construction method, system and storage medium that overcome the above-mentioned defects. Summary of the invention

[0006] In view of the above problems, the present invention provides a method for constructing an HBase memory index, wherein the method for constructing an HBase memory index comprises:

[0007] Update step: When updating the data table, the update operation request is intercepted by the coprocessor, and the index table is updated to obtain a secondary index table;

[0008] Search step: When performing an index query, the search operation request is intercepted by the coprocessor, and the index items that meet the conditions are found from the secondary index table, and the query is performed after returning to the original data table.

[0009] The above HBase memory index construction method further includes:

[0010] Index persistence management steps: perform persistence operations on the secondary index table and provide persistent storage for the secondary index table and value table.

[0011] In the above HBase memory index construction method, the updating step includes:

[0012] The first interception step: intercepting the update operation request through the coprocessor;

[0013] Secondary index table construction steps: The secondary index table is constructed through the Observer coprocessor.

[0014] In the above HBase memory index construction method, the first interception step includes:

[0015] Insertion step: When performing a Put operation on the data table, the data table will monitor the Rowkey of the inserted data;

[0016] Positioning steps: Locate the Region on the RegionServer based on the Rowkey;

[0017] Calling steps: The event is intercepted by CoprocessorHost before the Put operation, and the prePut of the Observer listening on this Region is automatically called.

[0018] In the above-mentioned HBase memory index construction method, in the secondary index table construction step, the index details are set in the configuration file, the main server obtains the index information to be established from the configuration file, and then establishes the secondary index table in the IndexingCoprocessor in the corresponding RegionServer and manages the secondary index data at the same time.

[0019] In the above-mentioned HBase memory index construction method, in the index persistence management step, the established secondary index table is mapped from the memory to the external memory, and the external memory area index file used for persistent memory index is composed of an index header and an index structure for storing index information, and the index file and the memory space are kept consistent using memory mapping technology.

[0020] In the above HBase memory index construction method, an address conversion table is used during mapping to convert addresses in the index file into addresses in the memory, and the addresses are in one-to-one correspondence.

[0021] In the above HBase memory index construction method, when the structure of the secondary index table is modified, the external memory index file will also be automatically updated synchronously.

[0022] The present invention also provides an HBase memory index construction system, wherein the HBase memory index construction system comprises:

[0023] Index management module: when updating the data table, the update unit intercepts the update operation request through the coprocessor and updates the index table to obtain the secondary index table; when performing index query, the search unit intercepts the search operation request through the coprocessor and finds the index items that meet the conditions from the secondary index table, returns to the original data table and performs the query;

[0024] The index persistence management module performs persistence operations on the secondary index table and provides persistent storage for the secondary index table and value table.

[0025] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein the program executes the HBase memory index construction algorithm described in any one of the above items when running.

[0026] In summary, compared with the prior art, the present invention has the following effects: the present invention improves the query performance of HBase, especially the coprocessor-based solution proposed in the present invention, which realizes the construction of memory indexes and persistently stores the constructed memory indexes in external memory. The coprocessor is used to realize the rapid construction of secondary indexes and automatically update the indexes according to the changes in the HBase table, thereby greatly improving the index data retrieval speed and ensuring the availability and fault tolerance of the index.

[0027] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 This is a schematic diagram of the application of the HBase memory index construction method of the present invention;

[0030] Figure 2 This is a flowchart of the HBase memory index construction method of the present invention;

[0031] Figure 3 for Figure 2 A step-by-step flow chart of step S1;

[0032] Figure 4 for Figure 3 A step-by-step flow chart of step S11;

[0033] Figure 5 It is a structural diagram of the secondary index table;

[0034] Figure 6 Update the flow chart for the secondary index table;

[0035] Figure 7 for Figure 2 Flow chart of step S3;

[0036] Figure 8 A schematic diagram of the structure of the HBase memory index construction system of the present invention.

[0037] Wherein, the accompanying drawings are marked as follows:

[0038] Index management module: 11;

[0039] A first interception unit 111;

[0040] Secondary index table construction unit: 112;

[0041] Index persistence management module: 12. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0044] The terms “first”, “second”, “S1”, “S2”, etc. used in this document do not specifically refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0045] The directional terms used in this document, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0046] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0047] As used herein, "and / or" includes any or all combinations of the items described.

[0048] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality groups” includes “two groups” and “more than two groups”.

[0049] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0050] Certain terms used to describe the present application are discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art regarding the description of the present application.

[0051] HBase is a distributed, column-oriented open source database. Just as Bigtable uses the distributed data storage provided by Google File System, HBase provides similar capabilities to Bigtable on Hadoop. HBase is a subproject of Apache's Hadoop project. HBase is different from general relational databases. It is a database suitable for unstructured data storage. Another difference is that HBase is column-based rather than row-based. Specifically, HBase is a highly reliable, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC Server. Unlike commercial big data products such as FUJITSU Cliq, HBase is an open source implementation of Google Bigtable. Similar to Google Bigtable using GFS as its file storage system, HBase uses Hadoop HDFS as its file storage system; Google runs MapReduce to process the massive data in Bigtable, and HBase also uses Hadoop MapReduce to process the massive data in HBase; Google Bigtable uses Chubby as a collaborative service, and HBase uses Zookeeper as a counterpart.

[0052] A coprocessor is used to relieve the system microprocessor and assist the central processor in completing processing tasks that it cannot perform or performs inefficiently or ineffectively.

[0053] Please refer to Figure 1 , Figure 1 The following is a schematic diagram of the application of the HBase memory index construction method of the present invention. Figure 1 As shown, in the index mechanism of the present invention, the index management module is the core of the framework. When the user updates the data table, the coprocessor Coprocessor will intercept these requests and perform corresponding operations on the index table, including inserting, deleting and updating the metadata of the index (recording the index table name, index column and other information corresponding to the user table). When executing a query operation, the system will search for the corresponding index position in the cache, which improves the index retrieval speed. When the index in the cache is updated, the index management module will also update the index. The index persistence management step mainly performs persistence operations on the cache index and provides persistent storage of the index table and value table. HBase provides scalability and fault tolerance for persistent storage of data.

[0054] Please refer to Figure 2 , Figure 2This is a flow chart of the HBase memory index construction method of the present invention. Figure 2 As shown, the coprocessor-based HBase memory index construction method of the present invention includes:

[0055] Update step S1: When performing an update operation on the data table, the update operation request is intercepted by the coprocessor, and the index table is updated to obtain a secondary index table.

[0056] Please refer to Figure 3 , Figure 3 for Figure 2 The step-by-step flow chart of step S1 in FIG. Figure 3 As shown, the updating step includes:

[0057] The first interception step: intercepting the update operation request through the coprocessor;

[0058] Secondary index table construction steps: Construct the secondary index table (memory index) through the Observer coprocessor.

[0059] Specifically, the coprocessor secondary index is implemented as follows: the coprocessor mechanism supports users to rewrite Endpoint and Observer codes according to business requirements. The present invention uses the Observer coprocessor to build the index. The Observer coprocessor is similar to a trigger in a relational database. When adding, deleting, modifying, and checking the database, the Observer intercepts these requests and updates the index table in real time according to these requests.

[0060] Furthermore, in the secondary index table construction step, the index details are set in the configuration file, the main server obtains the index information to be established from the configuration file, and then establishes the secondary index table in the IndexingCoprocessor in the corresponding RegionServer and manages the secondary index data at the same time.

[0061] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of the secondary index table. Figure 5As shown in the figure, Hbase Client: Hbase client; Hmaster: Hbase master node, master server; Region Server: regional server; Coprocessor: coprocessor; Indexing: index; Rowkey: primary key, unique identifier; Cf: a: column family: column name, which can be understood as the value corresponding to the column named a under the column family named Cf. The index structure established is shown in the figure. The inverted method is used to exchange the value of the data table with the primary key. The original value serves as the primary key, and the original primary key is placed in the position of the value. A basic index table is constructed. Before the index is established, the data model of HBase can be formally expressed as: R→{Ci→{CQi→{Ti|Vi}}}, i=0,1,2,…,n, where the row key is R, the column family is C, the column qualifier is CQ, the timestamp is T, and the value is V. Then the formal expression of the secondary index is: Vi→{Ci→{CQi→{Ti|R}}}, i=0,1,2,…,n. The framework ensures that the index file and the main table are on the same RegionServer, so that when the index file is needed, it can be completed by establishing a connection with the RegionServer only once, which improves the speed. In this architecture, the user first sets the index details in the configuration file (hbase-site.xml). The main server obtains the index information to be established from the configuration file, and then establishes the index in the IndexingCoprocessor in the corresponding RegionServer and manages the secondary index data. The coprocessor Coprocessor is deployed on each node. After deployment, a regional coprocessor RegionCoprocessorHost instance will be automatically created on each region in the RegionServer. Its main function is to maintain the system-level or table-level regional observer coprocessor RegionObserver. Whenever the RegionObserver is started, the RegionCoprocessorHost will be initialized. During the initialization process, the RegionCoprocessorHost will load all the RegionObservers on the current server. Index management is completed by the extended RegionObserver instance (IndexRegionObserver). The main extended interfaces of IndexRegionObserver are shown in the following table.

[0062]

[0063]

[0064] Please refer to Figure 4 , Figure 4 for Figure 3 The step-by-step flow chart of step S11 in FIG. Figure 4 As shown, the first interception step includes:

[0065] Insertion step: When performing a Put operation on the data table, the data table will monitor the Rowkey of the inserted data;

[0066] Positioning steps: Locate the Region on the RegionServer based on the Rowkey;

[0067] Calling steps: The event is intercepted by CoprocessorHost before the Put operation, and the prePut of the Observer listening on this Region is automatically called.

[0068] Specifically, please refer to Figure 6 , Figure 6 Update the flow chart for the secondary index table. Figure 6 As shown in the figure, the following uses the Put interceptor as an example to illustrate the update process of the index table: Figure 6 As shown in the figure, when the data table performs a Put operation, the data table will monitor the Rowkey of the inserted data and locate the Region on the RegionServer. This event is intercepted by the CoprocessorHost before the Put operation, and the prePut method of the Observer that is listening on this Region is automatically called. Users can rewrite this method according to their needs to update the index. After the index update is completed, that is, after the Put method is executed, the event after Put will be intercepted by the CoprocessorHost again, and the postPut method of the listener Observer will be called. Similarly, this method supports user rewriting to implement the index logic after the Put operation is completed. It can be seen from this that when the Put operation is not completed, the listener will listen to this event, so the postPut method will not be called, and the index logic will not be completed, which ensures the consistency and event nature of the index and the original data table, and there will be no situation where the index cannot match the original data table.

[0069] Search step S2: When performing an index query, the search operation request is intercepted by the coprocessor, and the index items that meet the conditions are found from the secondary index table, and the query is performed after returning to the original data table.

[0070] Specifically, in this step S2, the use of the index: after building the secondary index table, when performing an index query, the client needs to perform a Scan operation, this operation will be intercepted by the coprocessor, the search conditions will be intercepted, and the index items that meet the conditions will be found from the index table, and the Rowkey of the original data table will be returned. The data table can be queried based on the Rowkey.

[0071] Index persistence management step S3: Perform persistence operation on the secondary index table to provide persistent storage of the secondary index table and the value table.

[0072] Please refer to Figure 7 , Figure 7 for Figure 2 Flow chart of step S3 in FIG. Figure 7 As shown, the index persistence management step S3 includes: a start step, a query step, a step of determining whether an index exists, a step of creating an external memory index file, a step of calling mmap to map the external file to the memory, a step of building or updating the memory index, a step of updating the external memory index file, a step of unmapping the internal and external memory, and an end step.

[0073] Specifically, since the maintenance of memory indexes is completed by the coprocessor, the operation of memory persistence is mainly to ensure the consistency of internal and external memory data, to ensure that the index does not need to be rebuilt when the host is powered off, and does not involve index creation operations. A memory index persistence method is proposed to map the established memory index from the memory to the external memory. The external memory area index file used for persistent memory index consists of an index header and an index structure that stores index information. The index file and the memory space use the memory mapping technology mmap to maintain consistency. When mapping, an address translation table is used to convert the address in the index file into the address in the memory, and they correspond one to one. When the memory index structure is modified, the external memory index file will also be automatically updated.

[0074] It should be noted that in the above steps, an important step is how to map the index file into the memory. After the index file is mapped into the memory, the virtual address is returned, and the index structure data is not entered into the memory. In order to ensure that the index structure is completely in the memory, the corresponding method should be used to gradually import the index structure into the memory. The hierarchical traversal method is used to completely traverse the index tree, and all index structures are in the memory.

[0075] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of the HBase memory index construction system of the present invention. Figure 8 As shown, the coprocessor-based HBase memory index construction system of the present invention includes:

[0076] The index management module 11, when updating the data table, the update unit intercepts the update operation request through the coprocessor, and updates the index table to obtain the secondary index table; when performing index query, the search unit intercepts the search operation request through the coprocessor, and finds the index items that meet the conditions from the secondary index table, and returns to the original data table for query;

[0077] The index persistence management module 12 performs persistence operations on the secondary index table and provides persistent storage of the secondary index table and the value table.

[0078] Furthermore, the index management module 11 includes: a first interception unit 111 and a secondary index table construction unit 112; the first interception unit 111 intercepts the update operation request through the coprocessor; the secondary index table construction unit 112 constructs the secondary index table through the Observer coprocessor.

[0079] When the first interception unit 111 performs a Put operation on the data table, the data table monitors the Rowkey of the inserted data; locates the Region on the RegionServer according to the Rowkey; the event is intercepted by the CoprocessorHost before the Put operation, and automatically calls the prePut of the Observer listening on this Region.

[0080] The secondary index table building unit 112 sets the index details in the configuration file. The main server obtains the index information to be built from the configuration file, and then builds the secondary index table in the IndexingCoprocessor in the corresponding RegionServer and manages the secondary index data at the same time.

[0081] The index persistence management module 12 maps the established secondary index table from the internal memory to the external memory. The external memory area index file used for persistent memory index consists of an index header and an index structure for storing index information. The index file and the memory space are consistent using memory mapping technology. Among them, an address conversion table is used during mapping to convert the address in the index file to the address in the memory, and they correspond one to one. In this embodiment, when the structure of the secondary index table is modified, the external memory index file will also be automatically updated synchronously.

[0082] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein the program executes the HBase memory index construction algorithm described in any one of the above items when running.

[0083] In summary, the present invention realizes the construction of memory index based on the solution of coprocessor Coprocessor, and stores the constructed memory index persistently in external memory, realizes the rapid construction of secondary index through coprocessor and can automatically update index according to the change of HBase table. At the same time, the established index is persisted, and memory calculation is performed when it is used, which greatly improves the index data retrieval speed, ensures the availability and fault tolerance of the index, and the retrieval speed is also greatly improved compared with the secondary index solution based on Solr and HiBase.

[0084] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an HBase memory index, characterized in that: The HBase memory index construction method includes: Update step: When updating the data table, the update operation request is intercepted by the coprocessor, and the index table is updated to obtain a secondary index table; Search step: When performing an index query, the coprocessor intercepts the search operation request, finds the index items that meet the conditions from the secondary index table, and returns to the original data table for query; Index persistence management steps: perform persistence operations on the secondary index table and provide persistent storage for the secondary index table and value table; Wherein, the updating step includes: The first interception step: intercepting the update operation request through the coprocessor; Secondary index table construction steps: construct the secondary index table through the Observer coprocessor; Wherein, the first interception step includes: Insertion step: When performing a Put operation on the data table, the data table will monitor the Rowkey of the inserted data; Positioning steps: Locate the Region on the RegionServer based on the Rowkey; Calling steps: The event is intercepted by CoprocessorHost before the Put operation, and the prePut of the Observer listening on this Region is automatically called; Among them, in the index persistence management step, the established secondary index table is mapped from the memory to the external memory, and the external memory area index file used for persistent memory index consists of an index header and an index structure for storing index information. The index file and the memory space are kept consistent using memory mapping technology.

2. The HBase memory index construction method according to claim 1, characterized in that: In the secondary index table construction step, the index details are set in the configuration file, the main server obtains the index information to be established from the configuration file, and then establishes the secondary index table in the IndexingCoprocessor in the corresponding RegionServer and manages the secondary index data at the same time.

3. The HBase memory index construction method according to claim 1, characterized in that: An address translation table is used during mapping to convert addresses in the index file into addresses in memory, with a one-to-one correspondence.

4. The HBase memory index construction method according to claim 3, characterized in that: When the structure of the secondary index table is modified, the external index file will also be automatically updated synchronously.

5. An HBase memory index construction system, characterized in that: The HBase memory index construction system includes: Index management module: when updating the data table, the update unit intercepts the update operation request through the coprocessor and updates the index table to obtain the secondary index table; when performing index query, the search unit intercepts the search operation request through the coprocessor and finds the index items that meet the conditions from the secondary index table, returns to the original data table and performs the query; The index persistence management module performs persistence operations on the secondary index table and provides persistent storage of the secondary index table and value table; Wherein, the index management module includes: A first interception unit: intercepts the update operation request through a coprocessor; Secondary index table construction unit: constructs the secondary index table through the Observer coprocessor; Wherein, the first interception unit includes: When performing a Put operation on a data table, the data table will monitor the Rowkey of the inserted data; Locate the Region on the RegionServer based on the Rowkey; The event is intercepted by CoprocessorHost before the Put operation, and the prePut of the Observer listening on this Region is automatically called; Among them, in the index persistence management module, the established secondary index table is mapped from the memory to the external memory, and the external memory area index file used for persistent memory index consists of an index header and an index structure for storing index information. The index file and the memory space are kept consistent using memory mapping technology.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the HBase memory index construction algorithm described in any one of claims 1 to 4 when running.

Images