Data processing method and system

Abstract

The invention relates to data processing methods and device systems, in particular to a data write-in method and system and a data reading method and system. The data write-in method includes that data of predetermined rows are read so as to generate corresponding file blocks; a header file is established for each file block, the header files comprise index of each file block in a database and index of the files that are stored in each file block, and each file block and the corresponding header file form a data block; the data blocks are written in storage nodes. By means of the methods and the systems, efficient compression of data can be performed, storage costs are lowered, the storage space is saved, and data query and analysis speed is increased.

Description

technical field [0001] Embodiments of the present disclosure generally relate to the field of data storage, for processing data, especially data writing and reading methods and corresponding systems. Background technique [0002] For enterprises, organizations, agencies and other institutions, a large amount of data will be generated in daily life, and these data will increase with the passage of time, and the amount of data will become extremely large. These big data provide valuable raw data for business development, statistical analysis, policy formulation, etc. However, as the amount of collected or collected data continues to increase, the system load capacity for storing these data continues to increase, and higher requirements are placed on the data storage architecture. How to improve the storage capacity and query efficiency of massive data has become a solution One of the hard problems of big data problems. [0003] Traditional data storage is row storage and col...

AI Technical Summary

Problems solved by technology

Specifically, for example, in a database, multiple fields such as "ID", "name", "count", and "year" are included, when it is necessary to use Structured Query Language (SQL) to query the data in the database When, for example, "SELECT name FROM order WHERE year=2014", since the row-based storage database can only be read row by row, when performing a query, each row of data in the database needs to be read out, and then the corresponding query The data of the statement condition is extracted, resulting in slow query speed

In addition, because when the query targets certain columns, unnecessary column reads cannot be skipped; due to the mixture of columns with different data values, row storage is not easy to obtain a very high compression ratio, that is, the space utilization rate is low

[0005] For columnar storage, although the query efficiency of row-based storage is low, different data columns of columnar-based storage data are stored in different storage nodes. Therefore, to obtain a complete data, the tuple data must be re- The cost of the composition is large, resulting in excessive network overhead, which cannot meet the needs of the distributed file system for instant query of big data

For example, if the data of different columns of the same data unit is stored on different storage nodes, in order to obtain the same data unit, it is necessary to repeatedly read multiple nodes to reconstruct related data, resulting in increased network overhead

When reading massive amounts of data, this network overhead can severely impact query speed

Images