Storage method based on large redundancy time series data

Abstract

The invention discloses a storage method based on large redundancy time series data. A traditional relational database is high in storage cost, low in write-in throughput and poor in query performancewhen storing time series data. The server provided by the invention uses a memory database as a cache. After the time series data sent by the sensor reaches a set period of measurement and control time or a staged measurement task is completed, the server stores the time series data cached in the memory database in a disk database in a redundancy elimination storage format to become historical data of the period of measurement and control time or the staged measurement task; when historical data needs to be searched and analyzed, calling is performed according to different storage formats. Historical data is stored in a compressed mode, redundancy of the data is removed, space expenditure of a server hard disk is reduced, the number of rows of the data is reduced, and the historical datacan be called rapidly and accurately when calling is needed.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a storage method based on large redundancy time series data. Background technique [0002] As one of the three foundations of information technology, sensor technology is a high-tech that is being developed by developed countries. With the advancement of economy and technology, the popularization of the Internet and the development of the information highway, a large number of sensors for various purposes have been deployed in every corner of society to ensure the safety and convenience of people's lives. [0003] A large number of sensors also generate a large amount of data, which not only occupies a large amount of network bandwidth during transmission, but also requires a large number of hardware systems to store historical data with the accumulation of applications. If the data is not processed accordingly, there will be a lot of redundant data. When re...

AI Technical Summary

Problems solved by technology

[0007] 1. High storage cost: poor compression of time series data requires a lot of machine resources;

[0008] 2. High maintenance cost: stand-alone system requires manual sub-database and sub-table at the upper layer, and the maintenance cost is high;

[0009] 3. Low write throughput: The write throughput of a single machine is low, and it is difficult to meet the write pressure of tens of millions of sequential data;

[0010] 4. Poor query performance: suitable for transaction processing, poor aggregation analysis performance of massive data