Reverse index mixed compression and decompression method based on Hbase database

Abstract

The invention discloses a reverse index mixed compression method based on an Hbase database. The reverse index mixed compression method comprises the steps of processing the Hbase database to obtain an Hbase database reversed index data table including keys and values; compressing the key part by a key dictionary compression method; compressing the value part by a variable bytecode compression method; and writing the compressed content into files. The invention also discloses a decompression method of the compressed file key part after being compressed by the compression method. The decompression method comprises the steps of judging the length of each compressed data item, processing to obtain decompressed data according to two conditions of the length being less than or equal to 13, and the length being more than or equal to 25, otherwise, failing to decompress. According to the method adopts the classification mixed compression and the classification decompression method, the compression ratio is improved on the condition that the high decompression ratio is ensured possibly; the unified considerations of file reading and data decompression can be achieved; and the query efficiency of the reverse index can be improved completely and the storage space can be saved.

Description

Technical field [0001] The invention relates to an inverted index compression and decompression method, in particular to a hybrid compression and decompression method of an inverted index based on an Hbase database. Background technique [0002] Compressing data can speed up the rate at which data is read from disk to memory. The efficient decompression algorithm makes the compressed data block read from the hard disk to the memory and then decompress the time consumed less than the same uncompressed data block read from the disk to the memory. When the amount of data is huge, this time reduction is more obvious. Therefore, in more cases, the search engine system that uses the compressed inverted index table is more efficient than the unused system. Generally speaking, the higher the compression ratio, the lower the decompression efficiency. If you blindly pursue one of compression efficiency, decompression efficiency, and compression ratio, none of them can meet actual needs....

AI Technical Summary

Problems solved by technology

[0007] The above compression methods improve the compression ratio or decompression efficiency from different angles and methods, but the common problem in inverted index compression is that these compression methods either focus on the compression ratio or decompression efficiency, For an inverted index that pays more attention to query efficiency, file reading and data decompression are a unified process, and these two factors should be considered together, rather than being biased towards one side alone

[0008] Hbase database currently only supports Gzip compression and Lzo compression. Gzip compression satisfies a high compression ratio, but the decompression efficiency is low; Lzo has high decompression efficiency, but the compression ratio is low

[0009] To sum up, there is currently no compression and decompression method based on the Hbase database, which can take into account the high decompression rate and high compression ratio, and realize the unified consideration of file reading and data decompression. There are no related reports in the public literature. There is no application precedent in practical application

Images