Multi-dimensional streaming full-quantity indexing method

Abstract

The invention discloses a multi-dimensional streaming type full index SFI-HBase (Streaming Full Index HBase) method, which comprises the following steps of: when data is inserted into the SFI-HBase, firstly writing the data into a storage layer, and simultaneously sending a message to a Kafka. And after the Spark Streaming pulls the data from the Kafka, the index is updated in the memory firstly, and then the HBase full-amount index layer is updated. During reading, the range query and the KNN query of the SFI-HBase access the storage layer after obtaining the index through the full index layer, and the point query directly accesses the storage layer. The bipartite mixed space filling curve is combined with a Z curve and a Hilbert curve, multi-dimensional space division is supported, locality is improved by more than 20%, and the degree of aggregation is improved by more than 50%. Based on the dimension reduction data, the SFI-HBase constructs an efficient multi-dimensional index structure. During insertion, the stored data is aggregated by using Spark Streaming, and concurrent insertion is supported; during query, the full index layer stores indexes with different granularities, and the granularities can be selected according to query conditions. Compared with MD-HBase, the SFI-HBase range query efficiency is improved by 10%, the KNN query efficiency is improved by 5 times, and the insertion efficiency is improved by 10 times.

Description

Technical Field

[0001] The present invention relates to HBase database, in particular to a multi-dimensional streaming full indexing method Background Art

[0002] In the information age, the amount of data is growing explosively, and how to maximize the value of big data is the top priority for the development of various industries. The massive, heterogeneous and distributed characteristics of big data make it impossible for traditional relational databases to play their advantages, so non-relational databases are needed to manage and use data.

[0003] Non-relational databases can store different types of data and have high read and write efficiency when there is no predefined table fixed mode. The emergence of non-relational databases solves the application problems of big data.

[0004] HBase is an open source distributed column storage database built on Hadoop and is part of the Hadoop ecosystem. As an outstanding representative of NoSQL databases, HBase has demonstrated superior read and write performance, strong scalability and availability when processing large-scale data. HBase uses LSM tree as the storage structure at the bottom layer, and one of its advantages is that it can provide fast queries based on primary keys. However, for multi-dimensional queries that are not primary keys, HBase's query efficiency may be affected because it uses a full table scan and filter method. Although HBase does not provide native support for auxiliary indexes, some application scenarios still require the use of auxiliary indexes, such as e-commerce, Internet of Things, transportation, earth science and other fields. In these fields, multi-dimensional queries based on non-primary keys are common requirements. For example, a traffic management system may need to query traffic conditions in a specific area based on attributes such as time, location, and vehicle type. For example, traffic flow statistics in a certain area can be quickly retrieved and analyzed through auxiliary indexes. Creating indexes on the distributed database HBase is a complex issue, and it is necessary to consider the data distribution of cluster expansion nodes and the cost of index maintenance.

[0005] Although HBase has certain limitations in querying, which restricts its promotion to a certain extent, due to the popularity of the Hadoop ecosystem, users still hope that HBase can support non-primary key multidimensional queries. To make up for this shortcoming, a common practice is to design an index structure based on the original data table and store and manage the index based on different methods. Reference [5] divides indexing schemes into the following three types: (1) a two-layer indexing scheme that combines local indexes and global indexes; (2) a secondary indexing scheme that creates additional auxiliary index tables; and (3) a global indexing scheme based on linearization technology. Among them, the two-layer indexing scheme has good scalability, but the index implementation is complex and the cost of updating the index is high; the secondary index structure is simple and the maintenance cost is low, but the efficiency is low when executing multidimensional queries and it will occupy more storage space; in contrast, the global indexing scheme based on linearization technology has high write throughput and low maintenance cost, but it is more complex to implement in terms of ensuring data consistency. Therefore, it is of great practical significance to deeply study the existing indexing technologies, solve the shortcomings of these schemes, and propose a simple, easy-to-use, and efficient HBase multidimensional indexing structure. Summary of the invention

[0006] Purpose of the present invention: In order to overcome the problem that bucket splitting in the classic HBase linearization technology solution MD-HBase will cause the data in the bucket to be unreadable and unwritable within a certain period of time, and there will be data concurrency problems. This paper proposes a new indexing scheme to avoid the operation of bucket splitting. At the same time, Kafka and Spark Streaming are introduced to pre-aggregate the data of the full index layer to support concurrent writing of data.

[0007] Technical solution: To achieve the above-mentioned purpose, the present invention proposes the following technical solution.

[0008] Based on a multi-dimensional streaming full indexing method, the method includes the following steps:

[0009] (1) When the client requests to insert data, SFI-HBase first writes the data to the storage layer.

[0010] (2) After SFI-HBase writes data to the storage layer, it simultaneously sends a message to Kafka to insert the data.

[0011] (3) After Spark Streaming pulls data from Kafka, it first updates the index data in memory.

[0012] (4) Spark Streaming pulls data from Kafka and updates the index data in memory first, and then updates the index data to the HBase full index layer.

[0013] (5) When reading data, SFI-HBase range query and KNN query first access the full index layer to obtain index information, and then access the storage layer based on the queried index information.

[0014] (6) When reading data, SFI-HBase point queries directly access the storage layer.

[0015] Furthermore, the multi-dimensional streaming full indexing method further includes the following steps:

[0016] After Spark Streaming pulls a batch of data from Kafka, it parses the data into index information of different granularities. After pre-aggregation calculation, it incrementally updates the key-value pair information of the full index layer to the memory through the MapStateByKey operator, and finally incrementally updates the index information to the HBase full index layer.

[0017] Furthermore, the multi-dimensional streaming full indexing method further includes the following steps:

[0018] SFI-HBase does not have an index layer with a fixed bucket size, and does not need to split the storage bucket. SFI-HBase writes to the storage layer and the full index layer in no particular order. For the storage layer, data is written to the HBase table concurrently. For the index layer, data inserted into the HBase full index layer will first be aggregated and stored by Spark Streaming. SFI-HBase can calculate index information of any bucket size through the full index layer, and can set the appropriate bucket size according to different query conditions (range query, KNN query).

[0019] Furthermore, the multi-dimensional streaming full indexing method further includes the following steps:

[0020] SFI-HBase consists of a string of 0 or 1 as the primary key, such as Figure 2 As shown, the full index table storing the bucket size column of Integer type and the primary key composed of a string concatenated with UUID consisting of 0 or 1, contains a storage table consisting of required index columns and optional data columns. The index column includes the values ​​of each index dimension column, which is of type Integer; the data column stores non-dimensional column values, which can be of any data type supported by HBase. When the preset conditions are: dimension = 3, order = 4, and the variable order of the binary mixed space filling curve = 2. In order to reduce Region splitting, an HBase partition with a binary length of 4 is set, and the index layer includes 16 kd trees with a root node length of 4 bits. At the storage layer, the encoding length is 3×4=12. In order to avoid duplication of the primary key of the data point, a 4-bit UUID is added. The corresponding physical structure is shown in the figure. Figure 3As shown in the figure, the full index layer only needs to store two columns, the binary prefix of each granular storage bucket and the bucket data size, to index the data in the storage layer. The storage layer needs to store dimension columns and other columns, where dimension columns are used for dimension filtering during query and other columns are used for specific data storage.

[0021] Furthermore, the multi-dimensional streaming full indexing method further includes the following steps:

[0022] The process of SFI-HBase querying the full index layer to obtain index information according to the specified bucket size: Assume Figure 4 The kd tree in the figure is a partition of the full index layer, and the query bucket size is set to 12. The red line indicates the path to find the hit index block from top to bottom, and the blue box indicates the index block that meets the conditions. The recursion starts from the partition root node and stops recursing downward after finding an index block that satisfies the data volume less than the optimal bucket size. For the index block represented by the last layer node of the kd tree, although the data volume may not meet the query bucket size requirement, it is already the finest-grained index, so it is also added to the final result set.

[0023] Furthermore, the multi-dimensional streaming full indexing method further includes the following steps:

[0024] Assume that the total cost is TC (total cost), the scan build cost (scanbuild cost) and scan filter build cost (scanfilter build cost) of building a containment block and an intersection block are SB_C and SFB_C, the number of containments (contains) and intersections (intersets) are CN and IN, the cost (filtercost) and number (filter num) of filtering a row of data are FC and FN, and the dimension is D. The total cost calculation formula for the query is: TC = SB_C*CN+SFB_C*IN+FC*FN*D.

[0025] Furthermore, the multi-dimensional streaming full indexing method further includes the following steps:

[0026] The process of range query in SFI-HBase is as follows: ① Set the optimal query bucket size according to the range query conditions. ② Obtain the index information of the area that intersects the range query area from the full index layer according to the optimal query bucket size, and query the data from the storage layer based on the obtained index information. Among them, the pseudo code of the algorithm for implementing the optimal bucket size by binary calculation in the full index layer is as follows:

[0027] Input: HBaseFullIndexTable, hyperCube / / Full index table, hypercube query area

[0028] Output: mid / / Optimal bucket size

[0029] min = SFIRangeScanBucketSize_Min

[0030] max = SFIRangeScanBucketSize_Max

[0031] mid = (min + max) / 2

[0032] leftMid = (min + mid) / 2

[0033] rightMid = (max + mid) / 2

[0034] While (max - min) > SFIRangeScanBucketSize_Precision do

[0035] leftColumns = CalScanColumns(HBaseFullIndexTable, hyperCube, leftMid)

[0036] rightColumns = CalScanColumns(HBaseFullIndexTable, hyperCube, rightMid)

[0037] leftCost = calRangeQueryCost(leftColumns)

[0038] rightCost = calRangeQueryCost(rightColumns)

[0039] if leftCost <= rightCost then

[0040] tmp = mid

[0041] mid = leftMid

[0042] max = tmp

[0043] end

[0044] else

[0045] tmp = mid

[0046] mid = rightMid

[0047] min = tmp

[0048] end if

[0049] leftMid=(min+mid) / 2

[0050] rightMid=(max+mid) / 2

[0051] end while

[0052] return mid

[0053] The input is the full index table HBaseFullIndexTable and the multidimensional query area hyperCube. The start conditions are set in the first to fifth rows. SFIRangeScanBucketSize_Min and SFIRangeScanBucketSize_Max are two hyperparameters, representing the minimum and maximum values ​​of the binary calculation, indicating that there is no optimal bucket size beyond this range. SFIRangeScanBucketSize_Precision in the sixth row is also a hyperparameter, indicating the range accuracy of the optimal bucket size, indicating that after the binary calculation reaches this accuracy, it does not need to be refined and the result can be returned. The CalScanColumns function obtains the CN, IN, and FN information under the specified bucket size by accessing the full index table. The calRangeQueryCost function calculates the total cost based on CN, IN, and FN. After calculating the total cost of the left and right nodes, a binary selection is performed. Finally, after the accuracy requirement of SFIRangeScanBucketSize_Precision is met, the result is returned to obtain the optimal bucket size.

[0054] The pseudo code of the algorithm implementation process of SFI-HBase range query is as follows:

[0055] Input: HBaseFullIndexTable, HBaseStoreTable, hyperCube / / full index table, storage table, hyperspace query area

[0056] Output: Rq / / Data set that meets the query conditions

[0057] bucketSize=getRangeScanBestBucket(HBaseFullIndexTable,hyperCube)

[0058] low,high=encode(hyperCube)

[0059] longestStr=LongestSubString(low,high)

[0060]

[0061] if len(longestStr)>Partition_len then

[0062] rootPrefixs=GetPrePartition_Strategy(Partition_value)

[0063] for each prefix∈rootPrefixs do

[0064] contains,intersects=contains,intersects U GetBucketIndexsByIndex(prefix,HBaseFullIndexTable,bucketSize)

[0065] end for

[0066] else

[0067] contains,intersects=contains,intersects U GetBucketIndexsByIndex(longestStr,HBaseFullIndexTable,bucketSiz)

[0068] end if

[0069] for each contain∈contains do

[0070] Rq=Rq U scanByPrefix(HBaseStoreTable,contain,dim,order)

[0071] end for

[0072] for each interest∈intersects do

[0073] Rq=Rq U scanByPrefixAndFilter(HBaseStoreTable,contain,dim,order)

[0074] end for

[0075] return Rq

[0076] The input of the range query algorithm is the HBase full index table, HBase storage table, and multi-dimensional query range space, and the output is the query result set.

[0077] The present invention also proposes a physical storage structure, an SFI-HBase storage structure, an SFI-HBase paradigm structure, and a kd-tree partition structure of a full index layer based on a multi-dimensional streaming full indexing method.

[0078] The physical storage structure is characterized by: SFI-HBase consists of a string of 0 or 1 as the primary key, a full index table storing an Integer-type bucket size column and a UUID composed of a string of 0 or 1 as the primary key, and a storage table containing a required index column and an optional data column. The index column includes the values ​​of each index dimension column, which is of type Integer; the data column stores non-dimensional column values, which can be any data type supported by HBase. When the preset conditions are: dimension = 3, order = 4, and the variable order of the binary mixed space filling curve = 2. In order to reduce the region splitting, an HBase partition with a binary length of 4 is set, and the index layer includes 16 kd trees with a root node length of 4 bits. In the storage layer, the encoding length is 3×4=12, and a 4-bit UUID is added to avoid duplication of the primary key of the data point. The full index layer only needs to store two columns, the binary prefix of each granular storage bucket and the bucket data size, to index the data in the storage layer. The storage layer needs to store dimension columns and other columns. The dimension columns are used for dimension filtering during query, and other columns are used for specific storage of data.

[0079] The SFI-HBase paradigm structural features are:

[0080] SFI-HBase::=<FullIndexTable,StoreTable>

[0081] FullIndexTable::= <prefix><Bucket_Size>

[0082] StoreTable::= <rowkey><Index_Column*>[Data_Column*]

[0083] Prefix::=<BinaryChar*>

[0084] Bucket_Size::= <integer>

[0085] Rowkey::=<Rowkey_Prefix> <uuid>

[0086] Index_Column::= <integer>

[0087] Data_Column::= <string> | <int> | <double> | <long>

[0088] Rowkey_Prefix::=<BinaryChar*>

[0089] BinaryChar::=‘0’|‘1’

[0090] UUID::= <string>

[0091] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0092] The present invention uses a multi-dimensional streaming full indexing method that not only supports binary partitioning and clipping of multi-dimensional data space, but also has better locality and aggregation. Under experimental conditions, the Blend_Z_Hilbert curve using the Z curve mixed with the Hilbert curve improves the locality index by more than 20% compared with the Z curve, and improves the aggregation index by more than 50%. In terms of data insertion, SFI-HBase uses Spark Streaming to aggregate and store the data inserted into the full index layer in advance, provides support for concurrent insertion, and achieves good insertion efficiency. In terms of data query, the full index layer of SFI-HBase stores index information of different granularities, and can select the index granularity according to the range and KNN query conditions, thereby improving the query efficiency. Compared with MD-HBase, a classic solution for linearization technology, the range query efficiency of SFI-HBase is improved by about 10%, the KNN query efficiency is improved by more than 5 times, and the insertion efficiency is improved by more than ten times. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A flowchart of a multi-dimensional streaming full indexing system according to an embodiment of the present invention

[0094] Figure 2 An example diagram of an SFI-HBase storage structure based on a multi-dimensional streaming full indexing system according to an embodiment of the present invention

[0095] Figure 3 The physical storage structure of SFI-HBase storage based on a multi-dimensional streaming full index system involved in the embodiment of the present invention

[0096] Figure 4 A schematic diagram of a kd tree partitioning of a full index layer based on a multi-dimensional streaming full index system according to an embodiment of the present invention DETAILED DESCRIPTION

[0097] In order to overcome the problem in the classic HBase linearization technology solution MD-HBase that bucket splitting will cause the data in the bucket to be unreadable and unwritable for a certain period of time, and there will be data concurrency problems. This paper proposes a new indexing solution to avoid the bucket splitting operation. At the same time, Kafka and Spark Streaming are introduced to pre-aggregate the data of the full index layer to support concurrent data writing.

[0098] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. However, it should be understood that the present invention can be implemented in various forms, and some exemplary and non-limiting embodiments shown in the accompanying drawings and described below are not intended to limit the present invention to the specific embodiments described.

[0099] Figure 1 This embodiment shows an implementation method based on a multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to the present invention, including the following steps:

[0100] S1: When the client requests to insert data, SFI-HBase first writes the data to the storage layer.

[0101] S2: After SFI-HBase writes data to the storage layer, it synchronously sends a message to Kafka to insert the data.

[0102] S3: After Spark Streaming pulls data from Kafka, it first updates the index data in memory.

[0103] S4: Spark Streaming pulls data from Kafka and updates the index data in memory first, and then updates the index data to the HBase full index layer

[0104] S5: When reading data, SFI-HBase range query and KNN query first access the full index layer to obtain index information, and then access the storage layer based on the queried index information.

[0105] S6: When reading data, SFI-HBase point queries directly access the storage layer.

[0106] In step 1: When the client requests to insert data, SFI-HBase first writes the data to the storage layer. Specifically, when the client initiates an insert operation, the system first submits the received data to the storage layer for persistent storage. In this process, the data is processed by the logic of the application layer and then passed to the database or other storage medium at the backend. In the storage layer, the data is written to the corresponding table or data structure to ensure that it can be retrieved and used in the future. After confirming that the data is successfully written to the storage layer, the system returns a corresponding response to the client, indicating that the insert operation has been successfully completed. If an exception occurs in the storage layer, the system may record an error log and feedback the error information to the client so that the user can take appropriate measures or retry the operation. SFI-HBase consists of a string consisting of 0 or 1 as the primary key, a full index table that stores the bucket size column of the Integer type, and a storage table consisting of a string consisting of 0 or 1 concatenated with a UUID, which contains the required index columns and optional data columns. The index column includes the values ​​of each index dimension column, which is of type Integer; the data column stores the values ​​of non-dimensional columns, which can be of any data type supported by HBase. When the preset conditions are: dimension = 3, order = 4, and variable order of binary mixed space filling curve = 2. In order to reduce Region splitting, HBase partitions with a binary length of 4 are set, and the index layer includes 16 kd trees with root nodes of 4-bit binary numbers. In the storage layer, the encoding length is 3×4=12. In order to avoid duplication of data point primary keys, a 4-bit UUID is added. When stored in each partition, the SFI-HBase storage structure example is as follows: Figure 2 As shown, the corresponding physical storage structure is as follows Figure 3 As shown. The full index layer only needs to store two columns: the binary prefix of each granular storage bucket and the bucket data size, to index the data in the storage layer. The storage layer needs to store dimension columns and other columns, where the dimension columns are used for dimension filtering during query and other columns are used for specific data storage.

[0107] In step 2, after SFI-HBase writes data to the storage layer, it synchronously sends a message to Kafka to insert the data. Specifically, after Spark Streaming pulls data from Kafka, it first updates the index data in memory, and then updates the index data to the HBase full index layer. In terms of data reading, SFI-HBase's range query and KNN query first access the full index layer to obtain index information, and then access the storage layer based on the queried index information, while point query directly accesses the storage layer. SFI-HBase does not have an index layer with a fixed bucket size, and does not need to split the storage bucket. SFI-HBase writes operations to the storage layer and the full index layer in no particular order. For the storage layer, data is written to the HBase table concurrently. For the index layer, data inserted into the HBase full index layer will first be aggregated and stored by Spark Streaming. SFI-HBase can calculate index information of any bucket size through the full index layer, and can set the appropriate bucket size according to different query conditions (range query, KNN query). The paradigm structure of SFI-HBase is as follows:

[0108] SFI-HBase::=<FullIndexTable,StoreTable>

[0109] FullIndexTable::= <prefix><Bucket_Size>

[0110] StoreTable::= <rowkey><Index_Column*>[Data_Column*]

[0111] Prefix::=<BinaryChar*>

[0112] Bucket_Size::= <integer>

[0113] Rowkey::=<Rowkey_Prefix> <uuid>

[0114] Index_Column::= <integer>

[0115] Data_Column::= <string> | <int> | <double> | <long>

[0116] Rowkey_Prefix::=<BinaryChar*>

[0117] BinaryChar::=‘0’|‘1’

[0118] UUID::= <string>

[0119] In step 3, after Spark Streaming pulls data from Kafka, it first updates the index data in memory. Specifically, after Spark Streaming pulls a batch of data from Kafka, it parses the data into index information of different granularities.

[0120] In step 4, Spark Streaming pulls data from Kafka and updates the index data in memory first, and then updates the index data to the HBase full index layer. Specifically, after pre-aggregation calculation, the MapStateByKey operator incrementally updates the key value pair information of the full index layer in memory, and finally incrementally updates the index information to the HBase full index layer.

[0121] In step five, when reading data, SFI-HBase's range query and KNN query first access the full index layer to obtain index information, and then access the storage layer based on the queried index information. Specifically, SFI-HBase's range query relies on the full index layer. For the full index layer that stores index information of each granularity, it is necessary to specify a query bucket size to obtain the index information at the specified granularity. During the study, it was found that there is an optimal value for this query bucket size to maximize the efficiency of range queries. Therefore, the range query algorithm can be divided into two steps: 1. Set the optimal query bucket size based on the range query conditions. 2. Obtain the index information of the area that intersects the range query area from the full index layer based on the optimal query bucket size, and query the storage layer for data based on the obtained index information. Assume Figure 4 The kd tree in is a partition of the full index layer, and the query bucket size is set to 12. The red straight line represents the path to find the hit index block from top to bottom, and the blue box represents the index block that meets the conditions. The recursion starts from the partition root node and stops recursing downward after finding an index block that satisfies the data volume less than the optimal bucket size. For the index block represented by the last layer node of the kd tree, although the data volume may not meet the query bucket size requirements, it is already the finest-grained index, so it is also added to the final result set. Among them, the algorithm for calculating the optimal bucket size by binary division of the full index layer is as follows:

[0122] Pseudo code for the implementation of the optimal bucket size algorithm for binary calculation at the full index layer:

[0123] Input: HBaseFullIndexTable, hyperCube / / Full index table, hypercube query area

[0124] Output: mid / / optimal bucket size

[0125] min=SFIRangeScanBucketSize_Min

[0126] max=SFIRangeScanBucketSize_Max

[0127] mid=(min+max) / 2

[0128] leftMid=(min+mid) / 2

[0129] rightMid=(max+mid) / 2

[0130] While(max-min)>SFIRangeScanBucketSize_Precision do

[0131] leftColumns=CalScanColumns(HBaseFullIndexTable,hyperCube,leftMid)

[0132] rightColumns=CalScanColumns(HBaseFullIndexTable,hyperCube,rightMid)

[0133] leftCost=calRangeQueryCost(leftColumns)

[0134] rightCost=calRangeQueryCost(rightColumns)

[0135] if leftCost<=rightCost then

[0136] tmp=mid

[0137] mid=leftMid

[0138] max=tmp

[0139] end

[0140] else

[0141] tmp=mid

[0142] mid=rightMid

[0143] min=tmp

[0144] end if

[0145] leftMid=(min+mid) / 2

[0146] rightMid=(max+mid) / 2

[0147] end while

[0148] return mid

[0149] The input is the full index table HBaseFullIndexTable and the multidimensional query area hyperCube. The start conditions are set in the first to fifth rows. SFIRangeScanBucketSize_Min and SFIRangeScanBucketSize_Max are two hyperparameters, representing the minimum and maximum values ​​of the binary calculation, indicating that there is no optimal bucket size beyond this range. SFIRangeScanBucketSize_Precision in the sixth row is also a hyperparameter, indicating the range accuracy of the optimal bucket size, indicating that after the binary calculation reaches this accuracy, it does not need to be refined and the result can be returned. The CalScanColumns function obtains the CN, IN, and FN information under the specified bucket size by accessing the full index table. The calRangeQueryCost function calculates the total cost based on CN, IN, and FN. After calculating the total cost of the left and right nodes, a binary selection is performed. Finally, after the accuracy requirement of SFIRangeScanBucketSize_Precision is met, the result is returned to obtain the optimal bucket size.

[0150] The range query process of SFI-HBase is as follows:

[0151] ① Set the optimal query bucket size based on the range query conditions.

[0152] ②According to the optimal query bucket size, the index information of the area that intersects the range query area is obtained from the full index layer, and the storage layer is queried for data based on the obtained index information.

[0153] SFI-HBase range query algorithm:

[0154] SFI-HBase range query algorithm implementation process pseudo code:

[0155] Input: HBaseFullIndexTable, HBaseStoreTable, hyperCube / / full index table, storage table, hyperspace query area

[0156] Output: Rq / / Data set that meets the query conditions

[0157] bucketSize=getRangeScanBestBucket(HBaseFullIndexTable,hyperCube)

[0158] low,high=encode(hyperCube)

[0159] longestStr=LongestSubString(low,high)

[0160]

[0161] if len(longestStr)>Partition_len then

[0162] rootPrefixs=GetPrePartition_Strategy(Partition_value)

[0163] for each prefix∈rootPrefixs do

[0164] contains,intersects=contains,intersects U GetBucketIndexsByIndex(prefix,HBaseFullIndexTable,bucketSize)

[0165] end for

[0166] else

[0167] contains,intersects=contains,intersects U GetBucketIndexsByIndex(longestStr,HBaseFullIndexTable,bucketSiz)

[0168] end if

[0169] for each contain∈contains do

[0170] Rq=Rq U scanByPrefix(HBaseStoreTable,contain,dim,order)

[0171] end for

[0172] for each interest∈intersects do

[0173] Rq=Rq U scanByPrefixAndFilter(HBaseStoreTable,contain,dim,order)

[0174] end for

[0175] return Rq

[0176] The input of the range query algorithm is the HBase full index table, HBase storage table, and multi-dimensional query range space, and the output is the query result set.

[0177] In step 6, when reading data, SFI-HBase's point query directly accesses the storage layer. Specifically, for a certain KNN query, the input point is the center of the sphere, and the nearest Kth point is a point on the sphere to form a hypersphere area, and this hypersphere must be included in an index block of the full index layer, so as long as a scan is initiated to a certain granularity index block where the input point is located, the result of the KNN query can be obtained. Among them, the pseudo code of the algorithm implementation process of SFI-HBase's KNN query is:

[0178] Input: HBaseFullIndexTable, HBaseStoreTable, Input, k / / Full index table, storage table, input point, query neighbor point number

[0179] Output: maxHeap / / The largest root heap that meets the query conditions

[0180] Indexlist=getsIndex(HBaseFullIndexTable,Input,PrePartition_Value,MaxIndexLen)

[0181]

[0182] for each index∈indexList do

[0183] nearestDistance=calNearestDistance(Input,Index)

[0184] SphereVolume=calculateHyperSphereVolume(nearestDistance)

[0185] HyperCubeVolume=calculateHyperCubeVolume(index)

[0186] antiDataNum= SphereVolume / HyperCubeVolume×Index.bucketSize×KNNInclinateRate

[0187] if antiDataNum>=k then

[0188] targetIndex=Index

[0189] break

[0190] end if

[0191] end for

[0192]

[0193] nearestDistance=calNearestDistance(Input,Index)

[0194] maxHeap=scanIndex(targetIndex,HBaseStoreTable,input)

[0195] if maxHeap.peek().Distance<=nearestDistance then

[0196] return maxHeap

[0197] else

[0198] maxHeap=KNNQuery_AllBucket (HBaseFullIndexTable,HBaseStoreTable,Input,k)

[0199] end if

[0200] else

[0201] maxHeap=KNNQuery_AllBucket (HBaseFullIndexTable,HBaseStoreTable,Input,k)

[0202] end if

[0203] return maxHeap

[0204] In the first line of the algorithm, based on the information of the input point, the index information of different bucket sizes where the input point is located is queried from the full index layer, where PrePartition_Value and MaxIndexLen represent the partition binary length and the maximum index length, respectively. targetIndex represents the index block that meets the set conditions. Starting from the third line, the index blocks of different bucket sizes obtained in the first line are traversed from back to front. The fourth line calculates the shortest distance from the input point to the boundary of the current index block. Based on this distance, the fifth line calculates the volume of a hypersphere with this distance as the radius. The hypersphere represents the range of neighboring points that are expected to be queried within this hyperspace. If it exceeds this range, the neighboring points may appear in other index blocks, which deviates from the core idea of ​​the algorithm to only query one index block. Line 6 calculates the volume of the hyperspace of the index block, and line 7 estimates the amount of data in the hypersphere through mathematical calculations, where Index.bucketSize is the amount of data in the index block, and KNNInclinateRate represents the tilt parameter, which is between 0 and 1. The purpose of setting the tilt parameter is to ensure that when the data in the index block is tilted, the data in the hypersphere can still meet the query condition once. Lines 8 to 11 indicate that when the expected amount of data in the hypersphere is greater than K, the best query block is selected, and it is expected to query K nearest neighboring points from it. The judgment logic of line 14 is that if the best query block exists, it is scanned, otherwise the KNN query algorithm based on the sorting of all blocks is used. After calculating the closest distance from the input point to the boundary of the best query block in line 15, line 16 scans the index block and calculates the distance. Line 17 indicates that if the maximum value of the large root heap is less than the closest distance to the boundary, it means that K neighboring points are found in the best query block and the result is returned. If the condition is not met, it means that the Kth neighboring point may appear outside the range of the best query block.

[0205] The technical features involved in the above different embodiments can be combined with each other to form other embodiments within the scope of the present invention. The above embodiments are not restrictive, and any sequence modification, equivalent replacement, etc. made to the present invention should be included in the protection scope of the present invention.

[0206] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Those skilled in the art can make substitutions and changes within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.< / string> < / long> < / double> < / int> < / string> < / integer> < / uuid> < / integer> < / rowkey> < / prefix> < / string> < / long> < / double> < / int> < / string> < / integer> < / uuid> < / integer> < / rowkey> < / prefix>

Claims

1. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method, characterized in that: Includes steps: (1) When the client requests to insert data, SFI-HBase first writes the data to the storage layer. (2) After SFI-HBase writes data to the storage layer, it simultaneously sends a message to Kafka to insert the data. (3) After Spark Streaming pulls data from Kafka, it first updates the index data in memory. (4) Spark Streaming pulls data from Kafka and updates the index data in memory first, and then updates the index data to the HBase full index layer (5) When reading data, SFI-HBase range query and KNN query first access the full index layer to obtain index information, and then access the storage layer based on the queried index information. (6) When reading data, SFI-HBase point queries directly access the storage layer.

2. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: Spark Streaming processing flow: After Spark Streaming pulls a batch of data from Kafka, it parses the data into index information of different granularities. After pre-aggregation calculation, it incrementally updates the key-value pair information of the full index layer to the memory through the MapStateByKey operator, and finally incrementally updates the index information to the HBase full index layer.

3. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: SFI-HBase does not have an index layer with a fixed bucket size, and does not need to split the storage bucket. SFI-HBase writes to the storage layer and the full index layer in no particular order. For the storage layer, data is written to the HBase table concurrently. For the index layer, data inserted into the HBase full index layer will first be aggregated and stored by Spark Streaming. SFI-HBase can calculate index information of any bucket size through the full index layer, and can set the appropriate bucket size according to different query conditions (range query, KNN query).

4. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: SFI-HBase paradigm structure: SFI-HBase::=<FullIndexTable,StoreTable> FullIndexTable::= <prefix><Bucket_Size>< / prefix> StoreTable::= <rowkey><Index_Column*> [Data_Column*]< / rowkey> Prefix::=<BinaryChar*> Bucket_Size::= <integer>< / integer> Rowkey::=<Rowkey_Prefix> <uuid>< / uuid> Index_Column::= <integer>< / integer> Data_Column::= <string> | <int> | <double> | <long>< / long> < / double> < / int> < / string> Rowkey_Prefix::=<BinaryChar*> BinaryChar::='0'|'1' UUID::= <string> 。< / string> 5. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: SFI-HBase consists of a string of 0 or 1 as the primary key, a full index table storing the bucket size column of Integer type and a UUID composed of a string of 0 or 1 as the primary key, and a storage table containing a required index column and an optional data column. The index column includes the values ​​of each index dimension column, which is of type Integer; the data column stores non-dimensional column values, which can be any data type supported by HBase. When the preset conditions are: dimension = 3, order = 4, and the variable order of the binary mixed space filling curve = 2. In order to reduce the region splitting, an HBase partition with a binary length of 4 is set, and the index layer includes 16 kd trees with a root node length of 4 bits. In the storage layer, the encoding length is 3×4=12. In order to avoid duplication of the primary key of the data point, a 4-bit UUID is added. When stored in each partition, the SFI-HBase storage structure is a tree structure. The corresponding physical storage structure is that HBase FullIndex stores Rowkey and bucket_size, HBase Store table stores RowKey and dimension value columns, and other columns are business names. The full index layer only needs to store two columns: the binary prefix of each granular storage bucket and the bucket data size, to index the data in the storage layer. The storage layer needs to store dimension columns and other columns, where dimension columns are used for dimension filtering during query and other columns are used for specific data storage.

6. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: The process of SFI-HBase querying the full index layer to obtain index information based on the specified bucket size: Assume that the kd tree is a partition of the full index layer, and the query bucket size is set to 12. The red straight line represents the path of searching for the hit index block from top to bottom, and the blue box represents the index block that meets the conditions. The recursion starts from the partition root node and stops recursing downward after finding an index block that satisfies the data volume less than the optimal bucket size. For the index block represented by the last layer node of the kd tree, although the data volume may not meet the query bucket size requirement, it is already the finest-grained index, so it is also added to the final result set.

7. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: Cost-based method to estimate the optimal bucket size: Assume that the total cost is TC (total cost), the scan build cost (scanbuildcost) and scan filter build cost (scanfilterbuildcost) of building a containment block and an intersection block are SB_C and SFB_C, the number of containments (contains) and intersections (intersets) are CN and IN, the cost (filtercost) and number (filternum) of filtering a row of data are FC and FN, and the dimension is D. The total cost calculation formula for the query is: TC = SB_C*CN+SFB_C*IN+FC*FN*D.

8. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: The optimal bucket size algorithm for binary calculation of the full index layer: Pseudo code for the implementation of the optimal bucket size algorithm for binary calculation at the full index layer: Input: HBaseFullIndexTable, hyperCube / / Full index table, hypercube query area Output: mid / / optimal bucket size The input is the full index table HBaseFullIndexTable and the multidimensional query area hyperCube. The start conditions are set in the first to fifth rows. SFIRangeScanBucketSize_Min and SFIRangeScanBucketSize_Max are two hyperparameters, representing the minimum and maximum values ​​of the binary calculation, indicating that there is no optimal bucket size beyond this range. SFIRangeScanBucketSize_Precision in the sixth row is also a hyperparameter, indicating the range precision of the optimal bucket size, indicating that after the binary calculation reaches this precision, it does not need to be refined and the result can be returned. The CalScanColumns function obtains the CN, IN, and FN information under the specified bucket size by accessing the full index table. The calRangeQueryCost function calculates the total cost based on CN, IN, and FN. After calculating the total cost of the left and right nodes, a binary selection is performed. Finally, after the precision requirement of SFIRangeScanBucketSize_Precision is met, the result is returned to obtain the optimal bucket size.

9. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: SFI-HBase range query process: ① Set the optimal query bucket size based on the range query conditions. ②According to the optimal query bucket size, the index information of the area that intersects the range query area is obtained from the full index layer, and the storage layer is queried for data based on the obtained index information.

10. A multi-dimensional streaming full index SFI-HBase (Streaming Full Index HBase) method according to claim 1, characterized in that: SFI-HBase range query algorithm: SFI-HBase range query algorithm implementation process pseudo code: Input: HBaseFullIndexTable, HBaseStoreTable, hyperCube / / full index table, storage table, hyperspace query area Output: Rq / / Data set that meets the query conditions The input of the range query algorithm is the HBase full index table, HBase storage table, and multi-dimensional query range space, and the output is the query result set.

11. A multi-dimensional streaming full index SFI-HBase (Streaming FullIndexHBase) method according to claim 1, characterized in that: SFI-HBase KNN query algorithm: Pseudo code of the algorithm implementation process of SFI-HBase's KNN query: Input: HBaseFullIndexTable, HBaseStoreTable, Input, k / / Full index table, storage Table, input point, query the number of neighboring points Output: maxHeap / / The largest root heap that meets the query conditions In the first line of the algorithm, based on the information of the input point, the index information of different bucket sizes where the input point is located is queried from the full index layer, where PrePartition_Value and MaxIndexLen represent the partition binary length and the maximum index length, respectively. targetIndex represents the index block that meets the set conditions. Starting from the third line, the index blocks of different bucket sizes obtained in the first line are traversed from back to front. The fourth line calculates the shortest distance from the input point to the boundary of the current index block. Based on this distance, the fifth line calculates the volume of a hypersphere with this distance as the radius. The hypersphere represents the range of neighboring points that are expected to be queried within this hyperspace. If it exceeds this range, the neighboring points may appear in other index blocks, which deviates from the core idea of ​​the algorithm to only query one index block. Line 6 calculates the volume of the index block hyperspace, and line 7 estimates the amount of data in the hypersphere through mathematical calculations, where Index.bucketSize is the amount of data in the index block, and KNNInclinateRate represents the tilt parameter, which is between 0 and 1. The purpose of setting the tilt parameter is to ensure that when the data in the index block is tilted, the data in the hypersphere can still meet the query condition once. Lines 8 to 11 indicate that when the expected amount of data in the hypersphere is greater than K, the best query block is selected, and it is expected to query K nearest neighboring points from it. The judgment logic of line 14 is that if the best query block exists, it is scanned, otherwise the KNN query algorithm based on the sorting of all blocks is used. After calculating the closest distance from the input point to the boundary of the best query block in line 15, line 16 scans the index block and calculates the distance. Line 17 indicates that if the maximum value of the large root heap is less than the closest distance to the boundary, it means that K neighboring points are found in the best query block and the result is returned. If the condition is not met, it means that the Kth neighboring point may appear outside the range of the best query block.

Images