Method, system and storage engine device for data storage and query

Abstract

The invention discloses a data storage and query method. In a data block storage structure, at least one compression groove is reserved in a data block head; in an index block storage structure, an index block comprises an index block head and an index block body; the data storage method comprises the steps that received original data files are divided into a plurality of sub-data files according to similar features (such as mobile phone number segments); data in the sub-data files are read in sequence, index IDs corresponding to the data are generated according to the information in the data, index trees are inquired according to the index IDs, data block numbers corresponding to the index IDs are acquired, and the data are stored in the corresponding data blocks one by one. The invention discloses a data storage and query system and a storage engine device. Through the data storage and query method and system and the storage engine device, a large amount of storage space is saved, and the input and output (IO) amount of computer internal storage and external storage is reduced in the data reading process.

Description

technical field [0001] The invention relates to data storage technologies in the fields of communications and computers, in particular to a data storage and query method, system and storage engine device. Background technique [0002] At present, databases, file systems, and the like can be used to store data. In existing data storage methods, data is stored in data blocks divided by databases or file systems. When the data is stored in the data block, it is stored through the relevant storage management algorithm, and the data records are randomly stored by row, and the query is queried through the constructed index. Taking the database as an example, the relevant logical structure of the database is as follows: figure 1 As shown, obviously, in figure 1 , the data is stored by row, and the row ID number (ID, Identity) is stored in the index, wherein the row ID is an ID that uniquely marks the physical location of the record. When performing regular table access, look up ...

AI Technical Summary

Problems solved by technology

For example, a user's general packet radio service (GPRS) bill for a certain month is stored in multiple data blocks, which will cause block-level data redundancy and consume a large amount of storage space; even if the block-level Compression technology for deduplication, also because similar data is stored in different blocks, the data deduplication rate is much lower than the traditional data dictionary deduplication method, and the storage space consumed by massive data is still huge

[0005] 2. The storage structure of the existing database or file system affects the query performance

Images