A data statistics method and system based on hbase database

Abstract

The invention discloses a data statistical method and system based on an Hbase database. According to received data, a storage format of the data is created, wherein a row key RowKey in the storage format is composed of a statistical granularity and a fixed statistical type of the data, a column Column can be formed by splicing of a matching statistical type name and a statistical value, according to the first statistical granularity and the storage format, the data is stored in the Hbase, when a second statistical granularity specified by a user is inconsistent with the first statistical granularity, a combined format is determined according to the second statistical granularity and the storage format, and the combination statistics is conducted on the data stored in the Hbase according to the combined format. The dependency on the specific statistical granularity and the statistical type is reduced in the data statistical process, and the instantaneity of the data statistical result is improved.

Description

technical field [0001] The invention relates to the technical field, in particular to a data statistics method based on an Hbase database, and in particular to a data statistics system based on an Hbase. Background technique [0002] Now is the era of data, where all kinds of data pile up into mountains, how to find regular statistical data based on certain dimensions in these messy data is becoming more and more important. Statistical data can provide an important basis for decision-making, research, analysis, etc. Therefore, a fast, efficient, real-time, and intelligent statistical solution can bring important value. [0003] HBase is a distributed, column-oriented open source database, a distributed storage system for structured data. HBase is different from general relational databases. It is a database suitable for unstructured data storage. In addition, HBase is column-based rather than row-based. [0004] The process of the existing data storage method based on Hba...

AI Technical Summary

Problems solved by technology

[0015] (1) The scalability is poor. Still take the above-mentioned vehicle information as an example, whether it is the time, daily, weekly, monthly, annual report, etc. in the time dimension, or the vehicle type, vehicle color, and violation type in the statistical type dimension, etc., It is necessary to rebuild the table, restart the process or thread to do it, which is relatively expensive for the calculation and memory of the system

Moreover, neither the time dimension nor the statistical type dimension can be configured. Adding a new time period or statistical type requires redevelopment, and the workload is huge;

[0016] (2) poor real-time performance, because the storage method similar to the above-mentioned scheme in the prior art is to query the data of the required time period from a large amount of historical data, then store it in the temporary table first, and then summarize the data to the required time period from the temporary table For reports, the data source is historical data, which determines that the real-time performance of this solution is not high

[0017] (3) The overhead is high. The above-mentioned data storage process is based on the statistical granularity of the time dimension. In practical applications, if there are many statistical types in the statistical granularity of the time dimension and the statistical type dimension, this process will do a lot Repeated work, so the overhead on the system will be relatively large

Images