An HBase optimization solution based on row-column hybrid storage

Abstract

The present invention discloses an HBase optimization scheme based on row-column hybrid storage, comprising the following steps: S1, constructing a new data table and index table, wherein the table structure of the data table and index is as follows: storing multiple values ​​of the same field continuously, and recording the maximum value, minimum value, and starting offset of the same field of the relevant columns in the data table and index table; S2, constructing a new row-column hybrid storage query scheme based on the new table structure; S3, converting Get to Scan on the data table; S4, rewriting Scan on the index table. This HBase optimization scheme based on row-column hybrid storage performs data modeling based on a row-column hybrid storage mode, can make full use of data similarity, significantly reduce data storage, and is verified by actual measurement to occupy 50% less storage than the original scheme; based on the new data aggregation storage mode, it can reduce the number of random data reads and improve overall query performance.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed databases, and in particular to an HBase optimization solution based on row-column hybrid storage. Background Art

[0002] With the development of big data technology, HBase has gradually become a widely used NoSQL distributed storage system in the industry. It has the characteristics of high reliability, column orientation, and open source, and has been successfully used in production systems by companies such as Facebook and Alibaba. How to efficiently store and query HBase's secondary indexes is a hot topic in the industry. Currently, the widely used solution architectures include Figure 1 As shown in the figure, the solution mainly consists of three modules: query processing module, query execution engine and big data storage module.

[0003] 1) Query processing module

[0004] The query processing module receives the query request sent by the client, obtains the query conditions from the query request, and checks whether the query conditions contain illegal conditions, error conditions, etc. If legal, the query conditions are sent to the query execution engine; if illegal, the error information is returned to the client.

[0005] 2) Query execution engine

[0006] The query execution engine is mainly responsible for matching and querying. Matching: matching conditions, that is, finding the table that the query conditions actually want to query; querying: converting legal and searchable query conditions into requests that HBase can process and performing queries.

[0007] 3) Big data storage module

[0008] The big data storage module is responsible for the storage of raw data and index data, and has the characteristics of distribution, large capacity, and fast response. For example, the column storage distributed big data storage platform HBase provided by the Apache open source community. From the perspective of storage: the data table stores complete data; index tables are formed for different indexes of the data table, stored in SATA disks, and the index table redundantly filters columns according to business needs.

[0009] HBase does not have the secondary index capability. The common HBase secondary index solutions are implemented as follows:

[0010] Raw data:

[0011] ID Name Age Time Address Clothes id1; time1 name1 age1 time1 add1 clo1 id2; time2 name2 age2 time2 add2 clo2 id3; time3 name3 age3 time3 add3 clo3 id4; time4 name4 age4 time4 add4 clo4 id5; time5 name5 age5 time5 add5 clo5

[0012] For the original data, the storage structure of the secondary index is as follows:

[0013] Data table structure:

[0014]

[0015] Index table 1 structure (users want to use the Name field for efficient query):

[0016]

[0017] Index table 2 structure (users want to use the Age field for efficient query):

[0018]

[0019] Based on the above table structure design, the existing query process is as follows:

[0020] 1) If the query condition matches an index area, and the index area contains all the fields of the query request, the index area can be searched, and the query condition is used to construct a Scan query for the index area. If the index area contains all the returned columns, the query result is used to construct a query response; if the index area does not contain all the returned columns, the Rowkey of the data table is obtained from the Scan result, and then the Rowkey is used to accurately get the data table.

[0021] 2) If the query condition matches a certain index area, but the index area does not contain all the filter conditions in the query request, the index area cannot be searched.

[0022] 3) If at least one index area is searchable, the query condition is searchable.

[0023] 4) If all index tables are not searchable, use the query conditions to construct a filter to perform a full table scan on the data table.

[0024] Existing query solutions such as Figure 2 As shown, the Scan construction method is: use the values ​​of all fields that are the Rowkey fields in the query condition, use prefix matching to construct the startkey / endkey of Sacn, use the values ​​of the fields that are not in the RowKey to construct the Filter, and construct the Scan request. The Get construction method is: use the drk found from the index table as the Get Rowkey, and use the Filter in the index table Scan as the Get Filter to initiate a Get request.

[0025] The main problem with this solution is that it concatenates the values ​​of multiple original fields into a string and stores it in a column of the column family. This solution cannot fully utilize the similarity of the values ​​of the same field to achieve a high data compression ratio, occupies a large amount of software storage, has high product costs, and has low query and analysis efficiency.

[0026] Therefore, we make improvements to this and propose an HBase optimization solution based on row-column hybrid storage. Summary of the invention

[0027] The purpose of the present invention is to solve the problems in the existing secondary index storage and query solutions, and propose a storage solution in a row-column mixed format to reduce storage occupancy and improve query performance. The present invention provides the following technical solutions:

[0028] The present invention provides an HBase optimization solution based on row-column hybrid storage, comprising the following steps:

[0029] S1. Construct a new data table and index table. The table structure of the data table and index is as follows: store multiple values ​​of the same field continuously, and record the maximum value, minimum value, and starting offset of the same field of the relevant column in the data table and index table;

[0030] S2. Based on the new table structure, a new row-column hybrid storage query solution is constructed. The specific query process of the new row-column hybrid storage query solution is as follows:

[0031] S2.1. If all index tables are not searchable, use the query conditions to construct a filter to perform a full table scan on the data table;

[0032] S2.2. If the query condition matches a certain index table, and the index table contains the columns that need to be returned, the index table can be searched. Index table queries are all Scan requests. In order to prevent Scan from missing data, Scan needs to be rewritten and the filter is constructed using the original filter condition to scan the index table.

[0033] S2.3. If the query condition can match a certain index table, but the index table does not contain all the fields that the query needs to return, use the F:drk column of the index table to construct a Scan request for the data table to perform data query;

[0034] S3. Convert the data table from Get to Scan;

[0035] S4. Scan and rewrite the index table.

[0036] As a preferred technical solution of the present invention, the time in the Rowkey in step S1 selects the latest time value among all time values.

[0037] As a preferred technical solution of the present invention, the specific query process of the data table Get to Scan in step S3 is as follows:

[0038] S3.1. If the RowKey in the Get request actually exists in the table, the data can be accurately obtained after executing the Get request. In this case, there is no need to switch to Scan, and the Get result can be directly returned;

[0039] S3.2. If the Rowkey in the Get request is not in the table, such as between two adjacent real Rowkeys, then Get needs to be converted to Scan. The construction of Scan takes the data table as an example: the startkey of Sacn is set to empty, the endKey is set to "id+maxTime", and the Get request condition is used as a filter, and finally Scan is executed; if the returned field does not contain all fields, the location of the returned field is used on the client to filter the byte array result after Scan.

[0040] As a preferred technical solution of the present invention, the index table Scan rewriting process in step S4 is specifically as follows:

[0041] S4.1. Get the startKey in the original Scan condition, split the startKey by field, and get the value of the time field;

[0042] S4.2, replace the original time value with the minimum time value and reconstruct the startKey, that is, move the startKey forward to bring in the data packets that may be missed;

[0043] S4.3. Reconstruct Scan using the new startKey, original endKey and original Filter, and return the final query result to the client.

[0044] The beneficial effects of the present invention are:

[0045] 1. This HBase optimization solution based on row-column hybrid storage performs data modeling based on the row-column hybrid storage mode, which can make full use of data similarity and significantly reduce data storage. It has been verified by actual measurements that it occupies 50% less storage than the original solution.

[0046] 2. This HBase optimization solution based on row-column hybrid storage is based on a new data aggregation storage mode, which can reduce the number of random data reads and improve overall query performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1 It is a schematic diagram of the existing solution architecture in the prior art;

[0049] Figure 2 is a schematic diagram of an existing query scheme in the prior art;

[0050] Figure 3 It is a flow chart of a query solution of row-column hybrid storage of an HBase optimization solution based on row-column hybrid storage of the present invention;

[0051] Figure 4 This is a data table Get to Scan flow chart of an HBase optimization solution based on row-column hybrid storage of the present invention;

[0052] Figure 5 The present invention discloses an index table Scan rewriting flow chart of an HBase optimization solution based on row-column hybrid storage. DETAILED DESCRIPTION

[0053] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0054] Embodiment: The present invention provides an HBase optimization solution based on row-column hybrid storage, comprising the following steps:

[0055] S1. Build a new data table and index table. The table structure of the data table and index is as follows: store multiple values ​​of the same field continuously, and record the maximum and minimum values ​​of the relevant columns and the starting offset of the same field in the data table and index table. The time in the rowkey selects the latest (largest) time value among all time values.

[0056] Data table structure:

[0057]

[0058]

[0059] Index table 1 structure (users want to use the Name field for efficient query):

[0060]

[0061] Index table 2 structure (users want to use the Age field for efficient query):

[0062]

[0063] S2, such as Figure 3 As shown, based on the new table structure, a new row-column hybrid storage query solution is constructed. The specific query process of the new row-column hybrid storage query solution is as follows:

[0064] S2.1. If all index tables are not searchable, use the query conditions to construct a filter to perform a full table scan on the data table;

[0065] S2.2. If the query condition matches a certain index table, and the index table contains the columns to be returned, the index table can be searched. Index table queries are all Scan requests. In order to prevent Scan from missing data, Scan needs to be rewritten and the original filter condition is used to construct a filter to scan the index table (see the Scan rewriting process). Figure 5 );

[0066] S2.3. If the query condition can match a certain index table, but the index table does not contain all the fields that the query needs to return, use the F:drk column of the index table to construct a Scan request for the data table to perform data query.

[0067] S3, such as Figure 4 As shown, the data table is converted from Get to Scan. The specific query process is as follows:

[0068] S3.1. If the RowKey in the Get request actually exists in the table, the data can be accurately obtained after executing the Get request. In this case, there is no need to switch to Scan, and the Get result can be directly returned;

[0069] S3.2. If the Rowkey in the Get request is not in the table, such as between two adjacent real Rowkeys, then Get needs to be converted to Scan. The construction of Scan takes the data table as an example: the startkey of Sacn is set to empty, the endKey is set to "id+maxTime", and the Get request condition is used as a filter, and finally Scan is executed; if the returned field does not contain all fields, the location of the returned field is used on the client to filter the byte array result after Scan.

[0070] Example: The requested Rowkey is:<id=001,time=20210101> ,Filter is:<age=10> , if the S:maxTime corresponding to the RowKey in the data table is 20210106, the converted Scan is: [startKey:<””>, endKey:<id=001,time=20210106> ,Filter:<age=10> ].

[0071] S4, such as Figure 5 As shown, the index table is scanned and rewritten. The specific process is as follows:

[0072] S4.1. Get the startKey in the original Scan condition, split the startKey by field, and get the value of the time field;

[0073] S4.2, replace the original time value with the minimum time value and reconstruct the startKey, that is, move the startKey forward to bring in the data packets that may be missed;

[0074] S4.3. Reconstruct Scan using the new startKey, original endKey and original Filter, and return the final query result to the client.

[0075] Example: The original Scan request is:

[0076] [startKey:<name=liu,time=20200101> ,endKey:<name=liu,time=20210301> ,Filter:<age=10> ]

[0077] After the conversion is completed:

[0078] [startKey:<name=liu,time=00000000> ,endKey:<name=liu,time=20210301> ,Filter:<age=10> ].

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An HBase optimization solution based on row-column hybrid storage, characterized in that: The steps include: S1. Construct a new data table and index table. The table structure of the data table and index is as follows: store multiple values ​​of the same field continuously, and record the maximum value, minimum value, and starting offset of the same field of the relevant column in the data table and index table; S2. Based on the new table structure, a new row-column hybrid storage query solution is constructed. The specific query process of the new row-column hybrid storage query solution is as follows: S2.

1. If all index tables are not searchable, use the query conditions to construct a filter to perform a full table scan on the data table; S2.

2. If the query condition matches a certain index table, and the index table contains the columns that need to be returned, the index table can be searched. Index table queries are all Scan requests. In order to prevent Scan from missing data, Scan needs to be rewritten, and the filter is constructed using the original filter condition to rewrite the Scan of the index table; S2.

3. If the query condition matches a certain index table, but the index table does not contain all the fields that need to be returned by the query, use the F:drk column of the index table to construct a Scan request for the data table to perform data query; S3. Convert the data table from Get to Scan; The index table Scan rewriting process in step S2.2 is specifically as follows: Get the startKey in the original Scan condition, split the startKey by field, and get the value of the time field; Replace the original time value with the minimum time value and reconstruct the startKey, that is, move the startKey forward to bring in the data packets that may be missed; Reconstruct Scan with the new startKey, original endKey and original Filter, and return the final query result to the client; The specific query process of the data table Get to Scan in step S3 is as follows: S3.

1. If the RowKey in the Get request actually exists in the table, the data can be accurately obtained after executing the Get request. In this case, there is no need to switch to Scan, and the Get result can be directly returned. S3.

2. If the Rowkey in the Get request does not exist in the table, you need to convert Get to Scan. Set the startkey of Sacn to empty, the endKey to "id+maxTime", and use the Get request condition as a filter. Finally, execute Scan. If not all fields are returned, use the location of the returned field on the client to filter the byte array result after Scan.

2. According to claim 1, an HBase optimization solution based on row-column hybrid storage is characterized in that: The time in the Rowkey in step S1 is the latest one among all time values.

Images