A columnar storage and query method and system for time series data

Abstract

The present invention provides a columnar storage and query method and system for time-series data. The storage method includes: dividing a column of time-series data into multiple pages, and each page stores a part of data points of the column of time-series data. The sum of the stored data points is all the data points in a column of time series data; set the page header and page body for each page, and for each page, store the aggregated index information of all the data points in the page in the page of the page header, and store the data value information of all data points in the page in the page body of the page. Through the present invention, the time stamp and specific data value of a group of time series data can be stored in a column, which reduces the disk I / O during data query; and divides the data according to pages, and establishes aggregation index information respectively, Speed ​​up the query speed of data.

Description

technical field [0001] The present invention relates to the technical field of data storage, and more specifically, to a columnar storage and query method and system for time series data. Background technique [0002] With the development of computer technology and industrial informatization, the amount of data generated in the industrial field is increasing. Time series data (referred to as "time series data") is widely used in the industrial field and is the main data type in industrial data. As the main data format in industrial big data, its storage and query issues have become a key content in the research of industrial big data. [0003] Time-series data is usually generated by machine sensors at a certain frequency, and once generated, there is almost no need to modify it; secondly, the application of time-series data is more for analytical queries rather than fine-grained additions, deletions, and modifications. Therefore, combined with the structural characteristi...

AI Technical Summary

Problems solved by technology

[0004] Currently, mainstream columnar storage file formats such as Parquet and ORCFile are not optimized for the storage of time-series data. When storing time-series data in a columnar format, they all adopt a more general columnar storage method, which has complex The nested structure of a set of time-series data, and the data value and timestamp of a set of time-series data cannot be stored in a column at the same time, and the timestamp and data value of a set of time-series data cannot be indexed in a column of data at the same time. These problems will slow down the time series Data query speed consumes more system resources

Images