Monitoring data storage and query method and device

Abstract

The invention discloses a monitoring data storage and query method and device, and relates to the technical field of computers. One specific embodiment of the storage method comprises: encoding to-be-stored monitoring data to obtain encoded to-be-stored monitoring data; when the time dimension information of the to-be-stored monitoring data is at a second level, storing the to-be-stored monitoringdata subjected to encoding processing into a cache and a database; and when the time dimension information of the to-be-stored monitoring data is at a non-second level, storing the to-be-stored monitoring data subjected to encoding processing into a database. According to the embodiment, the monitoring data is encoded, the disk space for data storage can be remarkably reduced, the second-level monitoring data is stored in the cache and the database respectively, the efficiency of different types of query can be met, and the query speed of recent data in the second-level monitoring data is guaranteed.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for storing and querying monitoring data. Background technique [0002] In large-scale and high-concurrency Internet applications, when there are functional or performance problems in the system, it is necessary to be able to quickly and accurately locate the problem. An efficient monitoring system plays a key role in problem location. At present, system monitoring has entered the era of second-level monitoring from minute-level monitoring in the past. Nearly hundreds of times the amount of data and concurrency poses unprecedented challenges to the entire monitoring system, especially the data storage system at the core. [0003] At this stage, for time-series-based data storage such as monitoring data, common solutions are: (1) Use non-relational databases (ie NoSQL databases, such as Hbase, cassandra) to store metrics and timestamps ) splicing as the ro...

AI Technical Summary

Problems solved by technology

[0004] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: 1. The existing non-relational database is a monitoring data storage solution for minute level, facing the magnitude and concurrency of second level data, its The performance of data writing or data query cannot meet business needs; 2. OpenTSDB needs to put source data into memory, and every time a range of data is queried, a large range of data needs to be fetched into memory and then filtered. The demand is very large, which can easily lead to memory overflow; 3. The solution based on native time series data storage has weak support for data replication and linear expansion in design, which is unacceptable for large-scale monitoring applications with rapidly growing data

Images