Multi-column joint storage method based on column storage

Abstract

The invention discloses a multi-column joint storage method based on the column storage, and relates to the technical field of multi-column storage. The method comprises the following steps: S1, creating a joint index for multiple columns, and intervening the sorting distribution of data (according to group sorting storage) during data storage; S2, adopting the column storage by each column in themultiple columns of indexes, storing each column of data in an independent continuous area, and storing each column of index data in blocks, wherein the blocks are minimum units of data storage; S3,establishing a two-stage skip list query structure, and accelerating data retrieval. According to the multi-column joint storage method based on column storage, due to the fact that the data are stored in the form of columns, in the SQL statement execution process, the expenditure of mapping operation in a row database and the characteristics of column-by-column storage and data partitioning of the data are saved. Only useful attributes and data can be read from a magnetic disk according to needs so as to save the IO bandwidth.

Description

technical field [0001] The invention relates to the technical field of multi-column storage, in particular to a multi-column joint storage method based on column storage. Background technique [0002] In recent years, with the substantial increase in data volume in various industries, storage costs and management costs have also increased. Facing the OLAP demand for massive data, row storage databases will consume a lot of time for mapping when statistical data. The operation of massive data makes the database performance limited by computer memory and hard disk. In order to reduce storage costs and the need for machine configuration, the database system uses the method of compressing data to save data. In order to facilitate the statistical needs of OLAP, the method of storing data in columns is adopted. However, traditional column storage is for single-column storage, and there is no association between multiple columns, and the data is not arranged in order, so the effi...

AI Technical Summary

Problems solved by technology

Because it is necessary to read the content of the corresponding row of each column attribute file, a lot of IO time is wasted; the current column store internally sorts multiple columns independently, and consumes too much cpu and io resources when grouping statistics based on multiple columns ; The compression ratio of the existing column store database is not high; the basic unit of the existing column store database execution engine to access data is a single attribute value, and there are a large number of jump access and random access in the access of the column database to the attribute value, which seriously affects the query performance

Images