A columnar compression storage and retrieval method suitable for time series databases

Abstract

The application discloses a columnar compression storage and retrieval method suitable for a time series database and relates to the technical field of time series databases.The application comprises the following steps: (1) constructing a database; (2) dividing the database internally and obtaining a plurality of tables; (3) constructing files in the tables, naming the files according to the types of the tables, and configuring hard disks for the files; (4) storing column data obtained after compression in column blocks, performing alignment operation on the column data in the compression process, performing alignment operation on the column blocks in the compression process, writing all the column blocks in the tables into a file at the same time, using a new file in the table if the current file is full, and attaching a serial number to the file, with the serial number starting from 1; and (5) generating table information from the storage information of the column blocks in the storage and organization process.The application is completely columnar storage, is suitable for fixed-length compression storage of disk blocks, and avoids wasting storage space.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of time series database, and particularly relates to a columnar compression storage and retrieval method suitable for time series database. BACKGROUND

[0002] The time series database is mainly used for processing series data collected in time sequence for an object. At present, the vertical time series database represented by Influx DB has become the mainstream of the time series database market after several generations of development, and according to the characteristics of the time series database, the use of LSM Tree technology is the symbol of the latest generation of time series database.

[0003] In order to achieve efficient retrieval and storage, the time series database currently adopts columnar storage mode, which compresses each column data block first, and then compresses. However, the existing time series database basically has no better planning for the last written disk data block, and cannot best utilize the disk space. Moreover, in the process of reading data, the involved data is often distributed in multiple data blocks, and more disk data blocks need to be read. Therefore, the present application provides a columnar compression storage and retrieval method suitable for time series database, which aims to organize data compression and storage, provide a concise representation for columnar data storage, provide alignment of column data blocks, assist accurate reading of data rows, reduce data reading amount, and improve data block query speed. SUMMARY

[0004] The present application aims to solve the problems raised in the background art, and provides a columnar compression storage and retrieval method suitable for time series database.

[0005] In order to achieve the above-mentioned purpose, the present application specifically adopts the following technical scheme:

[0006] A storage method of time series database, comprising the following steps:

[0007] (1) constructing a database;

[0008] (2) dividing the database internally and obtaining a plurality of tables;

[0009] (3) constructing a file in the table, naming the file according to the type of the table, and configuring a hard disk for the file;

[0010] (4) storing the obtained column data in the column block after compression, and performing alignment operation on the column data in the compression process, writing all column blocks in the table to one or more files at the same time, if the current file is full, using a new file, and attaching a serial number to the file;

[0011] (5) Column block generates table information of its storage information during storage organization, and stores in corresponding table block file after compression.

[0012] Further, the column block is composed of a series of sub-compression blocks, each of which includes the length and data of the sub-compression block.

[0013] Further, the compression mode of column data includes the following steps:

[0014] ① Write the latest data into the empty memory in the column data block buffer;

[0015] ② After the buffer is full, compress the newly written data, reuse the memory occupied by these new data, put the compressed data at the beginning of these new data, and repeat step ①;

[0016] ③ Continue this process until the buffer is full (the last part can not be compressed if the compression rate is not enough, and each compression block has a flag indicating whether it is compressed), then drop the buffer and store the data in the file;

[0017] ④ Then reuse the buffer and repeat step ①.

[0018] Further, when the column block data is dropped, the column block data is aligned.

[0019] Further, the alignment operation is: when the primary key column block is dropped, other columns also compress the current data (even if the buffer is not full) and put them into the corresponding column sub-compression block.

[0020] Further, the saving steps of input data are as follows:

[0021] (1) Input a row of data externally, and the system inserts this row of data into each column;

[0022] (2) The system sets a buffer for each column, compresses and stores it, and drops it to the file;

[0023] (3) After dropping, the column block can be transferred to the historical data linked list as column block historical data, and the expired data can be eliminated according to the LRU algorithm;

[0024] (4) When the primary key column block is changed, the column alignment information needs to be saved;

[0025] (5) The block drop information and column alignment information described above are saved as table block information, and the table block information is also dropped according to the block compression saving mode.

[0026] A retrieval method of a time series database, comprising the following steps:

[0027] (1) User inputs a query statement to start data retrieval;

[0028] (2) Determine the time range of the data, if the data involved is not decompressed, then decompress the data; if the data is not in the memory, then load the historical data;

[0029] (3) When loading the historical data, find the column position according to the context, and load the column block from the disk; find the specific sub-compression block according to the column alignment information, start decompressing the data from the sub-compression block, and save it in the historical data;

[0030] (4) Perform retrieval.

[0031] Further, the above step (3) further comprises the following steps:

[0032] When loading the historical column block data, the sub-compression block data not in the retrieval range is still saved in the memory block data structure in a compressed manner for standby, and the data compression block has a longer cache time relative to the decompressed data.

[0033] Further, the following steps are further included:

[0034] When retrieving, if the current data cache and the compressed cache are not reused in the compressed data module, then the corresponding memory can be submitted to the retrieval module for reuse.

[0035] The beneficial effects of the present application are as follows:

[0036] 1. Complete column storage, block by column, compressed storage with fixed length to adapt to disk blocks, avoiding waste of storage space, and conducive to improving disk I / O efficiency.

[0037] 2. Primary key column alignment, precise positioning of historical data loading, reducing search range, and improving retrieval speed.

[0038] 3. Table block information also uses fixed block compression, which also has the advantages of high storage efficiency and improved disk access.

[0039] 4. The table block uses time primary key to represent table block positioning information, records the starting primary key position of the table block, and the column landing record and primary key alignment information, which can quickly and efficiently find the required data during search.

[0040] 5. In the system memory, when caching time sequence data, support mixed structure of compressed blocks and decompressed blocks, and decompress on demand. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a column block structure diagram of the present application;

[0042] Figure 2It is the memory and disk alignment structure diagram of the present application;

[0043] Figure 3 It is the column block storage mode structure diagram of the present application;

[0044] Figure 4 It is the table block information diagram of the present application;

[0045] Figure 5 It is the column block data structure diagram of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0047] In the prior art, since the time series database periodically and continuously collects time series data of the data source, the insertion performance is required to be high; and when querying, it is often for a specific attribute (column), so there are many advantages in using column storage and processing, and the main advantages are as follows:

[0048] (1) In the column storage mode, only the columns participating in the calculation need to be read, which greatly reduces the I / O cost and speeds up the query;

[0049] (2) The data in the same column belongs to the same type, and the compression effect is significant. Column storage often has a compression ratio as high as ten times or even higher, which saves a large amount of storage space and reduces the storage cost;

[0050] (3) Since the data type of each column is the same, a more suitable encoding method for CPU pipeline can be used to reduce the cache invalidation of CPU.

[0051] Therefore, the present application is aimed at the deficiencies of the current column block storage mode, and the deficiencies are as follows: the data blocks written to the disk at the end are not better planned, the disk space cannot be best utilized, and when reading data, the involved data is often distributed in multiple data blocks, which requires reading more disk data blocks. A time series database storage, saving and retrieval method is proposed, which has the following advantages:

[0052] (1) Complete column storage, block by column, suitable for fixed-length compression storage of disk blocks, avoiding waste of storage space and being beneficial to improving disk I / O efficiency;

[0053] (2) The primary key column is aligned, precise positioning of historical data loading is realized, the search range is reduced, and the retrieval speed is improved;

[0054] (3) The table block information also uses fixed block compression, which also has the advantages of high storage efficiency and improved disk access;

[0055] (4) The table block represents the table block positioning information with time as the primary key, records the starting primary key position of the table block, and records the column disk recording and primary key alignment information (force line). This structure can quickly and efficiently find the required data during search;

[0056] (5) When caching time series data in the system memory, a mixed structure of compressed blocks and decompressed blocks is supported, and decompression is performed on demand.

[0057] To achieve the above advantages and technical means, the specific content of the present application is as follows.

[0058] As Figures 1-5 shown, one embodiment of the present application proposes a storage method of a time series database, comprising the following steps:

[0059] (1) Constructing a database;

[0060] (2) Dividing the database internally and obtaining a plurality of tables;

[0061] (3) Constructing a file in the table, naming the file according to the type of the table, and configuring a hard disk for the file, the size of the file can be configured, and the default is 100MB;

[0062] (4) Compressing and storing the obtained column data in the column block, and performing alignment operation on the column data during compression, wherein the column data can be compressed to 4K, 8K, 16K, etc., but 4K is the best default size, all column blocks in the table are written to a file at the same time, if the current file is full, a new file in the table is used, and a serial number is attached to the file; and there are other feasible schemes for file writing, such as dividing the file by time, taking 2 hours or 1 day as a time period, setting an adjustable time period, and saving the data written in the same file in the time period, so that the file writing mode is relatively flexible;

[0063] (5) In the storage and organization process of the column block, the storage information of the column block is generated into table information, and is stored in the corresponding column block file after compression, wherein the table data can also be compressed to 4K, 8K, 16K, etc., but 4K is the best default size, and the sizes of the table data and the column data correspond.

[0064] First, some names of the present application are defined, which are also used in the following;

[0065] ① Column: corresponding to an attribute, the data collected at multiple time points of this attribute becomes a column of data;

[0066] ② Row: the data collected at a time point, composed of one data of each column at this time point;

[0067] ③ Column blocks: When each column of data is stored as a column, it will be divided into blocks of a fixed size, and each block is a column block;

[0068] ④ Table Block: In the storage organization, the storage method needs to be recorded. This information is stored in the table information, which is also stored in blocks. The data blocks in the table information are called table blocks.

[0069] ⑤ Data block: refers to a contiguous block of data, the size of which is a multiple of the hard disk sector block size;

[0070] ⑥ Write to disk: Data is written to the disk;

[0071] ⑦ LRU is an abbreviation for Least Recently Used, which is a commonly used page replacement algorithm that selects the least recently used page for elimination.

[0072] In the method of this invention, data is organized and stored according to a database. A database can contain multiple tables, and each table contains multiple fields (columns). The data in each column is stored in blocks. The core of the whole scheme is the storage method of table block and column block data, so that the stored data can be read in at any time according to the actual query needs.

[0073] In this invention, such as Figure 1 As shown, each column block consists of a series of sub-compressed blocks. Each sub-compressed block includes its length and data, starting with the length of the current sub-compressed block followed by the data. The length information can also be placed at the beginning or end. It should be noted that column blocks are stored in a simplified manner, or the data length can be omitted, and the offset value of the sub-compressed block within the block can be stored at the beginning or end of the column. When the column block data is stored, corresponding table information is generated. The table information stores the following information: column name (number), column block storage offset on disk, column block alignment information, and the information is stored in chronological order.

[0074] Both column data and table data are compressed and stored in the same column block file with a fixed size. The hard disk size of the column block file is adjustable, with a default of 100MB. Due to the relationship between column blocks and the data index information in the file, the table data and column data are stored in the same file. Currently, data retrieval is indexed by the primary key, which means that the default primary key for querying is the time item of the time series database, which facilitates data retrieval operations.

[0075] In the present application, both column data and table data are compressed to 4K by default, because in the prior art, the prevalence of Intel CPU is affected by the addressing mechanism of the CPU. Currently, the default memory page size of mainstream operating systems on the market (including all operating systems based on Windows and Linux kernels) is 4K. Even the recently popular ARM CPU has a page size of 4K under Linux. The default page used by the mainstream file system in the operating system is almost all 4K. Therefore, the default data block size selected by the present application is also 4K, which can be carefully optimized to achieve complete alignment of data access boundaries at the three levels of database block, memory block and disk block, as shown in Figure 2 , which greatly improves the performance of the database.

[0076] However, the size of the data block of the present application is adjustable, and the default value is 4K. The entire range has 4K, 8K and 16K, etc., but 4K is the best default size.

[0077] To compress column data and table data, taking column data as an example, the compression method of column data includes the following steps:

[0078] ① Write the latest data into the free memory in the column data block buffer buffer;

[0079] ② After the buffer is full, compress the newly written data. The memory occupied by these new data is reused. The compressed data is placed at the beginning of these new data, and step ① is repeated;

[0080] ③ Continue this process until the buffer is full (the last part can not be compressed if the compression rate is not enough. There is a flag in each compressed block indicating whether it is compressed), then drop the buffer and store the data in the file;

[0081] ④ Then reuse the buffer and repeat step ①.

[0082] As shown in Figure 3 , considering that when loading data, it is usually loaded with a time primary key as the range, therefore, in the above compression method, when the column block data is dropped, the column block data is aligned. The alignment operation is: when the primary key column block is dropped, the current uncompressed data of other columns that have not filled the buffer are placed in the sub-compression block. At the same time, the table block will use a record to save the column block alignment information. In this way, when loading historical data in the future, the current sub-compression block of other column blocks can be quickly found according to the primary key block, reducing the loading and decompression time of other column block data.

[0083] And in order to better understand the storage of the above table block information, according to Figure 4As shown, in the table block information, the information of the first and last block of the column block where the table block record starts to be written, and wherein:

[0084] (1) 1004, 1005, etc. are time information (timestamp used as the primary key);

[0085] (2) T1.1, A1.0 are actually the offsets of the column block in the disk, but the first digit thereof represents the sequence number of the column block, and the second digit represents the sequence number of the table block (0 is started in each column);

[0086] (3) Block line describes the storage position of the column block;

[0087] (4) Force line is column alignment information, which is description information based on the alignment of the primary key, and AFx and BFy respectively represent the sub-compression blocks corresponding to the blocks of the index A and the index B, which are actually offset values of file data.

[0088] The advantage of the above table block record information is that, in the block saving mode, the column block is simply stacked, the table block describes the stacking information of the column block, and it is an efficient storage organization mode; the organization of the column block description information records the starting primary key position of the table block, and the column landing record and the primary key alignment information (force line information), which is beneficial to subsequent queries.

[0089] The saving step of the input data is as follows:

[0090] (1) A row of data is externally input, and the system inserts the row of data into each column;

[0091] (2) The system sets a 4K buffer for each column, compressively stores, and lands to the file;

[0092] (3) After landing, the column block is transferred to the historical data linked list as column block historical data, and the expired data is eliminated according to the LRU algorithm;

[0093] (4) When the primary key column is block-switched, the column alignment information needs to be saved;

[0094] (5) The block landing information and the column alignment information are saved as table block information, and the table block information is also saved in the block compression mode.

[0095] A retrieval method of a time series database, comprising the following steps:

[0096] (1) The user inputs a query statement to start data retrieval;

[0097] (2) Determine the time range of the data, if the data involved is not decompressed, then decompress the data; if the data is not in memory, then load the historical data;

[0098] (3) When loading historical data, find the column position according to the context, and load the column block from the disk, find the specific sub-compressed block according to the column alignment information, decompress the data from the sub-compressed block, and save it in the historical data, and when loading the historical column block data, the sub-compressed block data not in the search range is still saved in the memory block data structure in compressed form for standby;

[0099] (4) Perform retrieval.

[0100] A retrieval method of a time series database further includes the following steps: when retrieving, if the current data cache and the compressed cache are not reused in the compressed data module, the corresponding memory can be submitted to the retrieval module for reuse.

[0101] The reuse mode of the compressed block is: when retrieving, organize the data, usually the compressed block and the corresponding decompression are provided, then save the compressed block and the decompressed data in the retrieval data table. When cleaning the data according to the LRU algorithm, the decompressed data will be deleted faster, and the corresponding compressed block will be retained for a longer time, because the compressed data is much smaller, which can better play the role of cache.

[0102] And when retrieving historical data, the database supports the operation of the main SQL syntax, and the speed of data query is the key. When querying data, the data in the memory needs to be fully utilized, and the historical data also needs to be quickly and accurately loaded.

[0103] The data organized for query includes compressed data blocks and decompressed data, and the data is eliminated according to the LRU algorithm.

[0104] Considering that the latest data is often the focus of query, the current inserted time series data will be continuously collected, in addition, the data after falling into the disk, including the compressed column block, the table block, the memory where the data is located can be considered to be put into the organization of historical data for direct reuse.

[0105] According to the accompanying Figure 5 As shown, data falling into the disk is implemented according to blocks, and it is most convenient to organize data internally according to blocks. Each column is represented by a column block linked list. It contains column data and some relationship information between column blocks (Forceline).

[0106] The column block data is data loaded from the disk, which is composed of multiple small compressed blocks, and is decompressed according to needs after loading. The force line information is description information of the alignment time primary key. When loading the historical information, the time column is determined according to the time range, and the compressed data positions of other columns are found according to the force line information, so that accurate positioning is realized.

[0107] According to the column-aligned historical data loading, the specified range of sub-compressed blocks is accurately found.

[0108] The above operation has the advantages of compressed data caching and on-demand decompression: when loading the historical column block data, there are still sub-compressed block data that is not used in each reading of 4k (which can be set) and is saved in the memory block data structure in a compressed manner for standby; this is also a kind of mixed caching of compressed and uncompressed data, which reuses data and reduces unnecessary decompression processing.

[0109] The above description of disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for storing a time-series database, characterized in that, Includes the following steps: (1) Construct the database; (2) Divide the database internally and obtain multiple tables; (3) Construct files in the table, name the files according to the table type, and configure the hard disk for the files; (4) Compress the acquired column data and store it in column blocks. The column data is aligned during the compression process. All column blocks in the table are written to one or more files at the same time. If the current file is full, a new file is used and a serial number is attached to the file. (5) During the storage organization process, the column block generates table information from its storage information, compresses it, and stores it in the corresponding table block file; The column block consists of a series of sub-compression blocks, each of which includes its length and data. The compression method for the column data includes the following steps: ① Write the latest data to the free memory in the column data block cache buffer; ② After the buffer is full, the latest written data is compressed. The memory occupied by this new data is reused. The compressed data is placed at the beginning of this new data, and step ① is repeated. ③ Continue this process until the buffer is full, then write the buffer to disk and store the data in a file; ④ Then reuse this buffer and repeat step ①.

2. The storage method for a time-series database according to claim 1, characterized in that, When writing column block data to disk, the column block data is aligned.

3. The method for storing a time-series database according to claim 2, characterized in that, The alignment operation is as follows: when the primary key column block is written to disk, other columns also compress the current data and put it into the corresponding column sub-compression block.

4. The method for storing a time-series database according to claim 3, characterized in that, The steps for saving the input data are as follows: (1) When a row of data is input from outside, the system inserts this row of data into each column; (2) The system sets up a buffer for each column, compresses and stores it, and writes it to a file on disk; (3) After the data is written to disk, the column block is transferred to the historical data list as the column block historical data. According to the LRU algorithm, expired data is eliminated to form the block writing information. (4) When the primary key column is moved to a different block, the column alignment information needs to be saved; (5) Save the block information and column alignment information into table block information, and save the table block information to disk in a block compression manner.

5. A retrieval method for a time-series database, wherein the time-series database is stored using the storage method described in any one of claims 1-4, characterized in that, Includes the following steps: (1) The user enters a query statement to begin data retrieval; (2) Determine the time range of the data. If the data involved is not decompressed, then decompress the data; if the data is not in memory, then historical data needs to be loaded. (3) When loading historical data, the corresponding column block position is found according to the context, and the column block is loaded from the disk. The specific sub-compressed block is found according to the column alignment information. Data is decompressed from this sub-compressed block and stored in the historical data. (4) Perform the search.

6. The retrieval method for a time-series database according to claim 5, characterized in that, Step (3) above also includes the following steps: When loading historical column block data, sub-compressed block data that is not within the retrieval range will still be stored in the memory block data structure in a compressed manner for later use.

7. The retrieval method for a time-series database according to claim 5, characterized in that, It also includes the following steps: During retrieval, if the current data cache and compressed cache are not reused in the compressed data module, the corresponding memory can be submitted to the retrieval module for reuse.

Images