Time series data storage method based on multilayer column-type storage structure

Abstract

The invention provides a time series data storage method based on a multilayer column-type storage structure. The method comprises the steps that all time series data to be stored is stored in a file,wherein the file comprises a first part and a second part; acquisition time of each piece of time series data to be stored and a data value of each piece of time series data to be stored are stored in the first part; the first part is divided into multiple row sets; each row set is divided into multiple column sets; the number of all the row sets, the offset amount of each row set in the file, the number of all the column sets in each row set, an attribute value of each piece of time series data and the offset amount of each column set in the file are used as the second part. According to thetime series data storage method, the time series data is organized according to the columns, the data of the same column in the same row set is continuously stored in the file, queried data is located at the close positions in the file accordingly, the amount of reading the time series data during query can be reduced, the query speed is increased, and the IO performance of a disk is improved.

Description

technical field [0001] The present invention relates to the technical field of computer data storage, and more specifically, to a time series data storage method based on a multi-level columnar storage structure. Background technique [0002] At present, with the continuous development of modern technologies such as cloud computing, the Internet, and the Internet of Things, data is playing an increasingly important role. [0003] Data comes from all aspects of our lives and is rapidly expanding—production and transaction data of enterprises, interaction information between people on the Internet, monitoring data returned by sensors in the Internet of Things, and so on. The speed of data generation is getting faster and faster, and the amount of data is getting bigger and bigger, followed by the problem of how to store data and how to quickly query existing data. [0004] Automation and informatization applications in the industrial field use more and more time-series data. ...

AI Technical Summary

Problems solved by technology

[0006] However, the existing columnar storage system only stores each column separately, without considering the following shortcomings for time series data: the time column and the corresponding value column are operated at the same time, and each column in the existing columnar storage is independently Storage, resulting in a decrease in disk IO performance; several time series data may be generated by the same IoT device, so the probability of these time series data being read and written at the same time is also high. In the existing columnar storage, each time series data is stored separately , leading to disk IO performance degradation

[0007] Based on the above two aspects, the disadvantage of the existing columnar storage system is that the disk IO performance is not high

Images