Hbase second-level query scheme based on solr

Abstract

The invention discloses an Hbase second-level query scheme based on solr. The Hbase second-level query scheme comprises the following steps of inserting raw data into an Hbase column-oriented database; calling a MapReduce increment to update an index in the solr, obtaining the raw data, and storing into a server of the solr with a particular file format of the solr; accessing the server of the solr, and establishing the index; firstly, searching the index, obtaining rowkey from the index, and querying required result data from an Hbase main list. The Hbase second-level query scheme has the advantages that the searching speed is high, and the accuracy is high; by adopting a solr and Hbase combining technique, the massive data can be searched in a second-level way, and the rowkey of data of one page can be returned back by a page separating function of the solr; because the number of data of each page is extremely limited, the response speed is higher when the Hbase query is performed according to the rowkey of the corresponding page, and is controlled to the millisecond level.

Description

technical field [0001] The invention relates to the technical field of hbase, in particular to a solr-based Hbase second-level query scheme. Background technique [0002] Solr is a complete search service based on Lucene under Apache. Solr mainly includes two core components: the index component and the search component. The index component is used to index the data that needs to be indexed in the search program, and the search component is used to query the index in response to the request of the client. Solr is a high-performance, Java5-based, Lucene-based full-text search server. At the same time, it has been extended to provide a richer query language than Lucene. At the same time, it is configurable, scalable, and optimizes query performance, and provides a complete function management interface. It is a very good Full text search engine. Documents are added to a searchable collection using XML via Http. Querying the collection is also achieved by receiving an XML / ...

AI Technical Summary

Problems solved by technology

[0012] (2) Indexing will affect the speed of inserting data

[0013] Since inserting data and building an index is a synchronous process, the operation of building an index will greatly affect the speed of inserting data

[0014] (3) The fields that need to be indexed must be determined before data insertion, and cannot be modified later

[0015] Another problem with inserting data while building an index is that we must determine all the fields that need to be indexed at one time. If an index needs to be built on a new field later, the previously inserted data will not be automatically indexed again

[0016] (4) Each index field corresponds to an index table is not efficient

[0020] But again, this solution also has a problem: the filter still needs to scan the data, which is inefficient

This process will occupy a lot of server-side memory when the amount of original query data is relatively large, and the scanning time will also be very long. The time-consuming of this process alone cannot meet the requirements of second-level query.

[0022] Based on the fact that some features of the above two solutions cannot meet our needs, we propose a solr-based Hbase second-level query solution

Images