Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Multisource time series data compression storage method

A time series, data compression technology, applied in database index, relational database, database model, etc., can solve the problems of limited data reading speed, insignificant increase in compression rate, slow query speed, etc., to improve compression efficiency, Improve data query speed and improve efficiency

Active Publication Date: 2014-07-09
ASAT CHINA TECH
View PDF4 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In the actual system, this kind of method also has restrictive factors, such as the difficulty of compression and decompression and time consumption. The data compression rate has a certain relationship with the data volume, that is, within a certain range, the compression rate increases along with the data volume. However, it increases rapidly. After exceeding a certain range, the compression ratio does not increase significantly.
[0007] In addition, since the above methods are often combined with files in a custom format to store a custom structure, although the custom storage structure can improve storage speed and reduce storage space, its query speed is usually slower than that of traditional relational databases. Limits the speed at which data can be read

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multisource time series data compression storage method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the following will further describe the implementation of the present invention in detail in conjunction with the accompanying drawings.

[0041] A preferred embodiment of the present invention relates to a method for compressing and storing multi-source time series data. The multi-source time series data comes from different measurement points of different deployment objects. The specific flow chart of the method is as follows figure 1 shown.

[0042] In step 101, each deployment object is grouped according to the number of measurement points contained in each deployment object. In this step, two or more deployment objects whose total number of measurement points is less than a preset value are divided into a group, and each remaining deployment object is divided into a group respectively.

[0043] For example, the default value is 20,000, deployment object 1 has 10,000 m...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a multisource time series data compression storage method. The multisource time series data compression storage method comprises the following steps of grouping deployment objects; dividing internal groups of deployment object groups; distributing a memory file which is used for caching for every internal group in the memory; performing first level lossy compression when time series data of a certain measuring point is received, finding the memory file which is corresponding to the measuring point according to the deployment object group and the internal group to which the measuring point belongs and caching the compressed data into the memory file; mapping the memory files to a hard disk and performing second level lossy compression when the internal files are fully filled or achieve the preset time limits and storing the compressed data blocks into a relational database. The multisource time series data compression storage method has the advantages of enabling the corresponding memory files to be found rapidly when data is cached, rapidly positioning storage positions and improving the caching efficiency; improving the compression efficiency and effectively saving hard drive capacity due to the partitioning compression mode; improving the data reading speed due to the relational database.

Description

technical field [0001] The invention relates to a method for compressing and storing data, in particular to a method for compressing and storing multi-source time series data. Background technique [0002] For large-scale enterprises such as electric power, petroleum and chain operations, their sub-stations or business locations are distributed across the country or even around the world, and each of their sub-stations or business locations contains many measurement points. The data centers of these large-scale enterprises, at any time The data generated by these measurement points must be stored at any time, and when necessary, the required data must be read from the stored data. And these industrial data have one thing in common: the data volume is relatively large, and the data sources are relatively extensive. Usually, industrial data is a continuous quantity of each measurement point, and the stored data are all time-stamped. Industrial data has obvious time character...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/22G06F16/24552G06F16/284
Inventor 俞高宇张晓华梁俊
Owner ASAT CHINA TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products