Method and system for storing analytic measurement data format MDF file

Abstract

The invention provides a method and system for storing an analytical measurement data format MDF file. The method comprises the steps of collecting and uploading an MDF data file; sorting the collected MDF data files; extracting metadata from the sorted MDF data files and storing the extracted metadata; converting the electric signal value of each record in the sorted MDF data file into a physicalvalue; generating a self-parsed column type storage format file and storing the generated column type storage format file in a distributed storage system; and calculating the stored column type storage format file by utilizing a distributed calculation system.

Description

Technical field[0001]The present invention relates to the field of data storage and analysis, and more specifically, to a method and system for storing an MDF file of analysis measurement data format.Background technique[0002]MDF file is a file format used to record data. The main feature is that it takes up less space under the same amount of data. The existing storage and analysis solution is to parse the MDF file into a distributed database (for example, MySQL), and then query and analyze it through SQL statements. However, this solution has the following problems:[0003](1) The data structure of the data to be stored needs to be known in advance;[0004](2) Storage expansion and backup are extremely complicated;[0005](3) Too much dependence on the database, unable to read data without connecting to the database;[0006](4) Difficulties in data movement;[0007](5) The storage space occupied by data is larger than that of original data, and the efficiency of analysis and query is limite...

AI Technical Summary

Problems solved by technology

[0004] (2) Storage expansion and backup are extremely complicated;

[0005] (3) Too much dependence on the database, unable to read data without connecting to the database;

[0006] (4) Difficulty in moving data;

[0007] (5) The storage space occupied by the data is larger than that of the original data, and the efficiency of analysis and query is limited by the processing capacity of the database;

[0008] (6) Expansion of database clusters, increased complexity of operation and maintenance, and poor data reliability; and

[0009] (7) In order to ensure the processing capacity of the database, small or large computers can be used, resulting in high system construction costs

[0010] In addition, since relational databases usually use row storage, all data needs to be scanned during query, which is inefficient

Images