Methods and systems for time-series data persistence based on data service platforms

Abstract

This invention discloses a method and system for time-series data persistence based on a data service platform, belonging to the field of time-series database technology. The technical problem this invention aims to solve is how to improve the query performance of total time-series data on a data service platform while simultaneously increasing file utilization. The technical solution adopted is as follows: The method involves writing the collected time-series data into memory, dividing it into multiple blocks of fixed size, and filling them with the data. When a block is full, the data is filed into the file system. Specifically, the following steps are taken: defining the file structure, including a data file (.data) and an index file (.idx); converting row-based data to column-based data; compressing by column, configuring the `compress` parameter to enable or disable compression; using different compression algorithms for each column based on its data type; and persisting the compressed data to the file system.

Description

Technical Field

[0001] This invention relates to the field of time-series database technology, specifically a method and system for time-series data persistence based on a data service platform. Background Technology

[0002] Inspur KaiWu Data Platform (KDP) is a data platform product developed by Inspur KaiWu Database Co., Ltd., with a multi-model database at its core, targeting various scenarios such as digital energy. In the digital energy scenario, there are many device terminals, many types of devices, and a large amount of data generated per second. KDP is required to support tens of millions or even hundreds of millions of writes per second and to efficiently and stably store the massive amounts of collected data.

[0003] In the context of widespread big data applications, we actively explore the deep integration of information technologies such as the Internet, big data, and cloud platforms with energy technologies. Leveraging the core advantages of energy + big data, we accelerate the integrated development of new energy industries, unlock the value of big data, and create big data assets. This has led to the development of a cloud-edge-device collaborative distributed new energy microgrid management platform. KDP, as a core component of the digital energy management platform, uses the multi-mode database ZNBase as its core, providing data and algorithm support for both the cloud-based multi-tenant energy management SaaS service and the edge-end energy router microgrid management system.

[0004] Based on the above scenario, stable and efficient storage of time-series data plays a crucial role in query response. Therefore, improving the query performance of total time-series data on the data service platform while simultaneously increasing file utilization is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] The technical objective of this invention is to provide a method and system for persisting time-series data based on a data service platform, in order to address the issues of improving the query performance of total time-series data on the data service platform while simultaneously increasing file utilization.

[0006] The technical objective of this invention is achieved as follows: a method for persistent time-series data based on a data service platform. This method involves writing the collected time-series data into memory, dividing it into multiple blocks of fixed size, and filling each block with the data. Once the blocks are full, the data is filed into the file system. Specifically:

[0007] Define the file structure: including the data file (.data) and the index file (.idx);

[0008] Transform row-based data into column-based data;

[0009] Compression by column: Configure the `compress` parameter to enable or disable compression; at the same time, each column uses a different compression algorithm based on its data type to improve compression efficiency.

[0010] Persist the compressed data to the file system.

[0011] Preferably, the data file .data includes the blocks of each table. The beginning of each block is a block header, which records the index information of the corresponding block so that the index of the current partition in memory can be recovered if the system crashes abnormally and not all of them are written to disk. After the block header is the column information ZColInfo of each column, which records the column id, data type, offset, length, pre-aggregation result and bitmap[]. The position of any column data in the file is located according to the offset and length. The original column data is obtained by using the corresponding decompression algorithm according to the column type. Each bit in bitmap[] indicates whether the column value is empty.

[0012] More preferably, the beginning of the index file .idx records the starting position, length, and number of tables of the table index. After the starting position, length, and number of tables of the table index are the table indexes sorted by Table ID. After the table indexes are the block indexes for each table. The block indexes of the same table are consecutive. The table index records the position of the first block index of the corresponding table in the index file.

[0013] After a block is successfully appended, the block index information in memory is updated, and then the index information in memory is periodically written to the index file.

[0014] If the latest index information in memory is not copied to disk in time when the system crashes, the memory index is rebuilt based on the block header in the data file .data.

[0015] More preferably, the row-based data to column-based data transformation is as follows:

[0016] (1) Obtain the metadata information corresponding to the block; the metadata information includes the number of columns in the table, the type of the columns, and the column ID;

[0017] (2) Retrieve data line by line from the block: If the Nth line is currently being read, traverse and parse each column of the Nth line according to the corresponding metadata information;

[0018] (3) Determine if all rows have been traversed and parsed:

[0019] ① If yes, then the process ends;

[0020] ② If not, proceed to step (4);

[0021] (4) Read each column of the corresponding parsed row in sequence;

[0022] (5) Determine if the initial value of the current column in the parsed row is NULL:

[0023] ① If so, proceed to step (6);

[0024] ② If not, proceed to step (8);

[0025] (6) Is the current column value of the parsed row NULL?

[0026] ① If so, proceed to step (4);

[0027] ② If not, proceed to step (7);

[0028] (7) Based on the column type type and the total number of rows in the block row_count, obtain the space required for the column storage sizeof(type)*count, and allocate temporary space pData;

[0029] (8) Insert the column value into the temporary space pData and set the bitmap; where the column value is read according to the column type;

[0030] (9) Determine whether all columns of the parsed row have been parsed:

[0031] ① If so, proceed to step (2);

[0032] ② If not, proceed to step (4).

[0033] More preferably, column-based compression is as follows:

[0034] Request compressed data space based on the total length of the original data in all non-empty columns;

[0035] Iterate through the objects starting from column 0 (timestamp column) and obtain the pointer to the object in column 0.

[0036] Check if the length of this column is 0:

[0037] If the length of this column is 0, then process the next column;

[0038] If the column length is not 0, then check whether to enable the compression option:

[0039] If compression is not enabled, the original column data will be copied directly to the underlying storage.

[0040] If compression is enabled, the corresponding compression algorithm will be selected based on the data type.

[0041] Determine if secondary compression is enabled:

[0042] If secondary compression is enabled, then secondary general compression is performed, and the offset and length of the data in that column within the block are updated in the column information. This process continues, traversing non-empty columns for compression, until all columns have been processed. Finally, the compressed data is persisted to the file system.

[0043] More preferably, persistence to the file system is as follows:

[0044] The compressed data is persisted to the block queue, and the flush thread persists the data sequentially from the queue.

[0045] After a flush process is completed, the index structure in memory needs to be updated. The block index in memory is updated according to the table_id corresponding to the block (which corresponds to the UUID). The index in memory is periodically written to the index file. Since writing the memory index to the file is an overwrite, in order to avoid writing failure and damaging the original file, the original index file is first renamed to a hidden file with the same name. The memory index is written to the new index file. If the writing is successful, the hidden file is deleted.

[0046] A system for time-series data persistence based on a data service platform, the system comprising:

[0047] The file structure definition unit is used to define the data file (.data) and the index file (.idx);

[0048] Transformation unit, used to convert row-based data to column-based data;

[0049] The compression unit is used to configure the compress parameter and determine whether compression is enabled; at the same time, each column uses a different compression algorithm according to the data type to improve compression efficiency.

[0050] Persistence unit, used to persist compressed data to the file system.

[0051] Preferably, the data file .data includes the blocks of each table. The beginning of each block is a block header, which records the index information of the corresponding block so that the index of the current partition in memory can be recovered if the system crashes abnormally and not all of them are written to disk. After the block header is the column information ZColInfo of each column, which records the column id, data type, offset, length, pre-aggregation result and bitmap[]. The position of any column data in the file is located according to the offset and length. The original column data is obtained by using the corresponding decompression algorithm according to the column type. Each bit in bitmap[] indicates whether the column value is empty.

[0052] The beginning of the index file .idx records the starting position, length, and number of tables of the table index. After the starting position, length, and number of tables of the table index are the table indexes sorted by Table ID. After the table indexes are the block indexes for each table. The block indexes of the same table are consecutive. The table index records the position of the first block index of the corresponding table in the index file.

[0053] After a block is successfully appended, the block index information in memory is updated, and then the index information in memory is periodically written to the index file.

[0054] If the latest index information in memory is not copied to disk in time when the system crashes, the memory index is rebuilt based on the block header in the data file .data.

[0055] The specific working process of the conversion unit is as follows:

[0056] (1) Obtain the metadata information corresponding to the block; the metadata information includes the number of columns in the table, the type of the columns, and the column ID;

[0057] (2) Retrieve data line by line from the block: If the Nth line is currently being read, traverse and parse each column of the Nth line according to the corresponding metadata information;

[0058] (3) Determine if all rows have been traversed and parsed:

[0059] ① If yes, then the process ends;

[0060] ② If not, proceed to step (4);

[0061] (4) Read each column of the corresponding parsed row in sequence;

[0062] (5) Determine if the initial value of the current column in the parsed row is NULL:

[0063] ① If so, proceed to step (6);

[0064] ② If not, proceed to step (8);

[0065] (6) Is the current column value of the parsed row NULL?

[0066] ① If so, proceed to step (4);

[0067] ② If not, proceed to step (7);

[0068] (7) Based on the column type type and the total number of rows in the block row_count, obtain the space required for the column storage sizeof(type)*count, and allocate temporary space pData;

[0069] (8) Insert the column value into the temporary space pData and set the bitmap; where the column value is read according to the column type;

[0070] (9) Determine whether all columns of the parsed row have been parsed:

[0071] ① If so, proceed to step (2);

[0072] ② If not, proceed to step (4);

[0073] The compression unit allocates compressed data space based on the total length of the original data in all non-empty columns; it iterates from column 0 (timestamp column) to obtain the object pointer of column 0;

[0074] Check if the length of this column is 0:

[0075] If the length of this column is 0, then process the next column;

[0076] If the column length is not 0, then check whether to enable the compression option:

[0077] If compression is not enabled, the original column data will be copied directly to the underlying storage.

[0078] If compression is enabled, the corresponding compression algorithm will be selected based on the data type.

[0079] Determine if secondary compression is enabled:

[0080] If secondary compression is enabled, secondary general compression is performed, and the offset and length of the data in the column in the block are updated. The non-empty columns are traversed and compressed until all columns are processed. Finally, the compressed data is persisted to the file system.

[0081] The persistence unit persists the compressed data to the block queue, and the flush thread persists the data sequentially from the queue. After a flush process is completed, the index structure in memory needs to be updated. The block index in memory is updated according to the table_id corresponding to the block (which corresponds to the UUID). The index in memory is periodically written to the index file. Since writing the memory index to the file is an overwrite, in order to avoid writing failure and damaging the original file, the original index file is first renamed to a hidden file with the same name. The memory index is then written to the new index file. If the write is successful, the hidden file is deleted.

[0082] An electronic device includes: a memory and at least one processor;

[0083] The memory contains computer programs;

[0084] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the time-series data persistence method based on the data service platform described above.

[0085] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the method for time-series data persistence based on a data service platform as described above.

[0086] The method and system for time-series data persistence based on a data service platform of the present invention have the following advantages:

[0087] (i) To reduce the disk space required after time-series data is written to disk, block data needs to be compressed before being written to disk. Columnar storage, because the data type of each column is the same, has a better compression ratio than row-based storage, therefore columnar storage occupies less space.

[0088] (ii) The present invention first writes the collected time-series data into memory. During the persistence process, optimization is performed, including converting row-based data into column-based data and compressing it by column. By constructing column-based data storage and compression writing, the data storage method is optimized and the query response is improved.

[0089] (III) According to tests, the present invention improves query performance and file utilization by 10% and 15% respectively compared to direct copying into the database. Attached Figure Description

[0090] The invention will be further described below with reference to the accompanying drawings.

[0091] Appendix Figure 1 This is a schematic diagram illustrating a method for persisting time-series data based on a data service platform.

[0092] Appendix Figure 2 This is a schematic diagram of the structure of the .data file;

[0093] Appendix Figure 3 This is a schematic diagram of the structure of the index file .idx;

[0094] Appendix Figure 4 The flowchart for converting row-based data to column-based data. Detailed Implementation

[0095] The method and system for time-series data persistence based on a data service platform according to the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0096] Example 1:

[0097] As attached Figure 1As shown, this embodiment provides a method for persisting time-series data based on a data service platform. This method involves writing the collected time-series data into memory, dividing it into multiple blocks of fixed size, and filling each block with the data. Once the blocks are full, the data is filed into the file system. Specifically:

[0098] S1. Define the file structure: including the data file .data and the index file .idx;

[0099] S2, row-based data transformed into column-based data;

[0100] S3. Compression by Column: Configure the `compress` parameter to enable or disable compression; each column uses a different compression algorithm based on its data type to improve compression efficiency.

[0101] S4. Persist the compressed data to the file system.

[0102] As attached Figure 2 As shown, the data file .data in step S1 of this embodiment includes blocks of each table. The beginning of each block is a block header, which records the index information of the corresponding block so that the index of the current partition in memory can be recovered if the system crashes abnormally and not all of them are written to disk. After the block header is the column information ZColInfo of each column, which records the column id, data type, offset, length, pre-aggregation result and bitmap[]. The position of any column data in the file is located according to the offset and length. The original column data is obtained by using the corresponding decompression algorithm according to the column type. Each bit in bitmap[] indicates whether the column value is empty.

[0103] As attached Figure 3 As shown, in step S1 of this embodiment, the beginning of the index file .idx records the starting position, length, and number of tables of the table index. After the starting position, length, and number of tables of the table index are the table indexes sorted by Table ID. After the table indexes are the block indexes for each table. The block indexes of the same table are consecutive. The table index records the position of the first block index of the corresponding table in the index file.

[0104] After a block is successfully appended, the block index information in memory is updated, and then the index information in memory is periodically written to the index file.

[0105] If the latest index information in memory is not copied to disk in time when the system crashes, the memory index is rebuilt based on the block header in the data file .data.

[0106] As attached Figure 4As shown, the row-to-column data transformation in step S2 of this embodiment is as follows:

[0107] (1) Obtain the metadata information corresponding to the block; the metadata information includes the number of columns in the table, the type of the columns, and the column ID;

[0108] (2) Retrieve data line by line from the block: If the Nth line is currently being read, traverse and parse each column of the Nth line according to the corresponding metadata information;

[0109] (3) Determine if all rows have been traversed and parsed:

[0110] ① If yes, then the process ends;

[0111] ② If not, proceed to step (4);

[0112] (4) Read each column of the corresponding parsed row in sequence;

[0113] (5) Determine if the initial value of the current column in the parsed row is NULL:

[0114] ① If so, proceed to step (6);

[0115] ② If not, proceed to step (8);

[0116] (6) Is the current column value of the parsed row NULL?

[0117] ① If so, proceed to step (4);

[0118] ② If not, proceed to step (7);

[0119] (7) Based on the column type type and the total number of rows in the block row_count, obtain the space required for the column storage sizeof(type)*count, and allocate temporary space pData;

[0120] (8) Insert the column value into the temporary space pData and set the bitmap; where the column value is read according to the column type;

[0121] (9) Determine whether all columns of the parsed row have been parsed:

[0122] ① If so, proceed to step (2);

[0123] ② If not, proceed to step (4).

[0124] The column-wise compression in step S3 of this embodiment is as follows:

[0125] (1) Request compressed data space based on the total length of the original data of all non-empty columns;

[0126] (2) Start traversing from column 0 (timestamp column) and get the object pointer in column 0;

[0127] (3) Determine if the length of the column is 0:

[0128] ① If the length of this column is 0, then process the next column;

[0129] ② If the length of the column is not 0, then proceed to step (4);

[0130] (4) Determine if the compression option is enabled:

[0131] ① If compression is not enabled, the original column data will be directly copied to the underlying storage;

[0132] ② If compression is enabled, select the corresponding compression algorithm based on the data type;

[0133] (5) Determine if secondary compression is enabled:

[0134] If secondary compression is enabled, then secondary general compression is performed, and the offset and length of the data in that column within the block are updated in the column information. This process continues, traversing non-empty columns for compression, until all columns have been processed. Finally, the compressed data is persisted to the file system.

[0135] The persistence to the file system in this embodiment is as follows:

[0136] (1) Persist the compressed data to the block queue, and the flush thread persists the data from the queue in sequence;

[0137] (2) After a flush disk process is completed, the index structure in memory needs to be updated. The block index in memory is updated according to the table_id corresponding to the block (corresponding to the UUID). The index in memory is written to the index file periodically. Since the memory index is written to the file as an overwrite, in order to avoid writing failure and damaging the original file, the original index file is renamed to a hidden file with the same name. The memory index is written to the new index file. The hidden file is deleted after successful writing.

[0138] Example 2:

[0139] This embodiment provides a system for time-series data persistence based on a data service platform. The system includes:

[0140] The file structure definition unit is used to define the data file (.data) and the index file (.idx);

[0141] Transformation unit, used to convert row-based data to column-based data;

[0142] The compression unit is used to configure the compress parameter and determine whether compression is enabled; at the same time, each column uses a different compression algorithm according to the data type to improve compression efficiency.

[0143] Persistence unit, used to persist compressed data to the file system.

[0144] As attached Figure 2 As shown, the data file .data in this embodiment includes blocks for each table. The beginning of each block is a block header, which records the index information of the corresponding block so that the index of the current partition in memory can be recovered if the system crashes abnormally and not all of them are written to disk. After the block header is the column information ZColInfo for each column, which records the column's id, data type, offset, length, pre-aggregation result and bitmap[]. The position of any column data in the file is located according to the offset and length. The original column data is obtained by using the corresponding decompression algorithm according to the column type. Each bit in the bitmap[] indicates whether the column value is empty.

[0145] As attached Figure 3 As shown, in this embodiment, the beginning of the index file .idx records the starting position, length, and number of tables of the table index. After the starting position, length, and number of tables of the table index are the table indexes sorted by Table ID. After the table indexes are the block indexes for each table. The block indexes of the same table are consecutive. The table index records the position of the first block index of the corresponding table in the index file.

[0146] After a block is successfully appended, the block index information in memory is updated, and then the index information in memory is periodically written to the index file.

[0147] If the latest index information in memory is not copied to disk in time when the system crashes, the memory index is rebuilt based on the block header in the data file .data.

[0148] As attached Figure 4 As shown, the working process of the conversion unit in this embodiment is as follows:

[0149] (1) Obtain the metadata information corresponding to the block; the metadata information includes the number of columns in the table, the type of the columns, and the column ID;

[0150] (2) Retrieve data line by line from the block: If the Nth line is currently being read, traverse and parse each column of the Nth line according to the corresponding metadata information;

[0151] (3) Determine if all rows have been traversed and parsed:

[0152] ① If yes, then the process ends;

[0153] ② If not, proceed to step (4);

[0154] (4) Read each column of the corresponding parsed row in sequence;

[0155] (5) Determine if the initial value of the current column in the parsed row is NULL:

[0156] ① If so, proceed to step (6);

[0157] ② If not, proceed to step (8);

[0158] (6) Is the current column value of the parsed row NULL?

[0159] ① If so, proceed to step (4);

[0160] ② If not, proceed to step (7);

[0161] (7) Based on the column type type and the total number of rows in the block row_count, obtain the space required for the column storage sizeof(type)*count, and allocate temporary space pData;

[0162] (8) Insert the column value into the temporary space pData and set the bitmap; where the column value is read according to the column type;

[0163] (9) Determine whether all columns of the parsed row have been parsed:

[0164] ① If so, proceed to step (2);

[0165] ② If not, proceed to step (4);

[0166] In this embodiment, the compression unit requests compressed data space based on the total length of the original data in all non-empty columns; it traverses from column 0 (timestamp column) to obtain the object pointer of column 0;

[0167] Check if the length of this column is 0:

[0168] If the length of this column is 0, then process the next column;

[0169] If the column length is not 0, then check whether to enable the compression option:

[0170] If compression is not enabled, the original column data will be copied directly to the underlying storage.

[0171] If compression is enabled, the corresponding compression algorithm will be selected based on the data type.

[0172] Determine if secondary compression is enabled:

[0173] If secondary compression is enabled, secondary general compression is performed, and the offset and length of the data in the column in the block are updated. The non-empty columns are traversed and compressed until all columns are processed. Finally, the compressed data is persisted to the file system.

[0174] In this embodiment, the persistence unit persists the compressed data to the block queue, and the flush thread persists the data sequentially from the queue. After a flush process is completed, the index structure in memory needs to be updated. The block index in memory is updated according to the table_id corresponding to the block (which corresponds to the UUID). The index in memory is periodically written to the index file. Since writing the memory index to the file is an overwrite, in order to avoid writing failure and damaging the original file, the original index file is first renamed to a hidden file with the same name. The memory index is then written to the new index file. If the writing is successful, the hidden file is deleted.

[0175] Example 3:

[0176] This invention also provides an electronic device, including: a memory and a processor;

[0177] The memory stores the instructions executed by the computer.

[0178] The processor executes computer execution instructions stored in the memory, causing the processor to perform the time-series data persistence method based on a data service platform according to any embodiment of the present invention.

[0179] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0180] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0181] Example 4:

[0182] This invention also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the method for time-series data persistence based on a data service platform according to any embodiment of this invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0183] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0184] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0185] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0186] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for time-series data persistence based on a data service platform, characterized in that, This method involves writing the collected time-series data into memory, dividing it into multiple blocks of fixed size, and filling them with the data. Once the blocks are full, the data is flushly written to the file system. Specifically: Define the file structure: including the data file (.data) and the index file (.idx); Transform row-based data into column-based data; Compression by column: Configure the `compress` parameter to enable or disable compression; each column uses a different compression algorithm based on its data type. Persist the compressed data to the file system; The data file .data contains blocks for each table. Each block begins with a block header, which records the index information of the corresponding block. After the block header is the column information ZColInfo for each column, which records the column's id, data type, offset, length, pre-aggregation result, and bitmap[]. The offset and length are used to locate the position of any column data in the file. The corresponding decompression algorithm is used to obtain the original column data according to the column type. Each bit in the bitmap[] indicates whether the column value is empty. The beginning of the index file .idx records the starting position, length, and number of tables of the table index. After the starting position, length, and number of tables of the table index are the table indexes sorted by Table ID. After the table indexes are the block indexes for each table. The block indexes of the same table are consecutive. The table index records the position of the first block index of the corresponding table in the index file. After a block is successfully appended, the block index information in memory is updated, and then the index information in memory is periodically written to the index file. If the latest index information in memory is not promptly copied to disk when the system crashes, the memory index is rebuilt based on the blockheader in the data file .data. The specific steps for transforming row-based data into column-based data are as follows: (1) Obtain the metadata information corresponding to the block; the metadata information includes the number of columns in the table, the type of the columns, and the column ID; (2) Retrieve data line by line from the block: If the Nth line is currently being read, traverse and parse each column of the Nth line according to the corresponding metadata information; (3) Determine if all rows have been traversed and parsed: ① If yes, then the process ends; ② If not, proceed to step (4); (4) Read each column of the corresponding parsed row in sequence; (5) Determine if the initial value of the current column in the parsed row is NULL: ① If so, proceed to step (6); ② If not, proceed to step (8); (6) Is the current column value of the parsed row NULL? ① If yes, then proceed to step (4); ② If not, proceed to step (7); (7) Based on the column type type and the total number of rows in the block row_count, obtain the space required for the column storage sizeof(type)*count, and allocate temporary space pData; (8) Insert the column value into the temporary space pData and set the bitmap; where the column value is read according to the column type; (9) Determine whether all columns of the parsed row have been parsed: ① If yes, then proceed to step (2); ② If not, proceed to step (4); Compression by column is as follows: Request compressed data space based on the total length of the original data in all non-empty columns; Iterate from column 0 and obtain the object pointer in column 0; Check if the length of this column is 0: If the length of this column is 0, then process the next column; If the column length is not 0, then check whether to enable the compression option: If compression is not enabled, the original column data will be copied directly to the underlying storage. If compression is enabled, the corresponding compression algorithm will be selected based on the data type. Determine if secondary compression is enabled: If secondary compression is enabled, secondary general compression is performed, and the offset and length of the data in the column in the block are updated. The non-empty columns are traversed and compressed until all columns are processed. Finally, the compressed data is persisted to the file system. The persistence to the file system is as follows: The compressed data is persisted to the block queue, and the flush thread persists the data sequentially from the queue. After a flush process is completed, the index structure in memory needs to be updated. The block index in memory is updated according to the table_id corresponding to the block. The index in memory is periodically written to the index file. Since writing the memory index to the file is an overwrite, in order to avoid writing failure and damaging the original file, the original index file is first renamed to a hidden file with the same name. The memory index is written to the new index file. If the writing is successful, the hidden file is deleted.

2. A system for time-series data persistence based on a data service platform, characterized in that, The system includes, The file structure definition unit is used to define the data file (.data) and the index file (.idx); Transformation unit, used to convert row-based data to column-based data; The compression unit is used to configure the compress parameter and determine whether compression is enabled; at the same time, each column uses a different compression algorithm according to the data type. Persistence unit, used to persist compressed data to the file system; The data file .data contains blocks for each table. Each block begins with a block header, which records the index information of the corresponding block. After the block header is the column information ZColInfo for each column, which records the column's id, data type, offset, length, pre-aggregation result, and bitmap[]. The offset and length are used to locate the position of any column data in the file. The corresponding decompression algorithm is used to obtain the original column data according to the column type. Each bit in the bitmap[] indicates whether the column value is empty. The beginning of the index file .idx records the starting position, length, and number of tables of the table index. After the starting position, length, and number of tables of the table index are the table indexes sorted by Table ID. After the table indexes are the block indexes for each table. The block indexes of the same table are consecutive. The table index records the position of the first block index of the corresponding table in the index file. After a block is successfully appended, the block index information in memory is updated, and then the index information in memory is periodically written to the index file. If the latest index information in memory is not promptly copied to disk when the system crashes, the memory index is rebuilt based on the blockheader in the data file .data. The specific working process of the conversion unit is as follows: (1) Obtain the metadata information corresponding to the block; the metadata information includes the number of columns in the table, the type of the columns, and the column ID; (2) Retrieve data line by line from the block: If the Nth line is currently being read, traverse and parse each column of the Nth line according to the corresponding metadata information; (3) Determine if all rows have been traversed and parsed: ① If yes, then the process ends; ② If not, proceed to step (4); (4) Read each column of the corresponding parsed row in sequence; (5) Determine if the initial value of the current column in the parsed row is NULL: ① If so, proceed to step (6); ② If not, proceed to step (8); (6) Is the current column value of the parsed row NULL? ① If yes, then proceed to step (4); ② If not, proceed to step (7); (7) Based on the column type type and the total number of rows in the block row_count, obtain the space required for the column storage sizeof(type)*count, and allocate temporary space pData; (8) Insert the column value into the temporary space pData and set the bitmap; where the column value is read according to the column type; (9) Determine whether all columns of the parsed row have been parsed: ① If yes, then proceed to step (2); ② If not, proceed to step (4); The compression unit allocates compressed data space based on the total length of the original data in all non-empty columns; it traverses from column 0 and obtains the object pointer of column 0; Check if the length of this column is 0: If the length of this column is 0, then process the next column; If the column length is not 0, then check whether to enable the compression option: If compression is not enabled, the original column data will be copied directly to the underlying storage. If compression is enabled, the corresponding compression algorithm will be selected based on the data type. Determine if secondary compression is enabled: If secondary compression is enabled, secondary general compression is performed, and the offset and length of the data in the column in the block are updated. The non-empty columns are traversed and compressed until all columns are processed. Finally, the compressed data is persisted to the file system. The persistence unit persists the compressed data to the block queue, and the flush thread persists the data sequentially from the queue. After a flush process is completed, the index structure in memory needs to be updated. The block index in memory is updated according to the table_id corresponding to the block. The index in memory is periodically written to the index file. Since writing the memory index to the file is an overwrite, in order to avoid writing failure and damaging the original file, the original index file is first renamed to a hidden file with the same name. The memory index is then written to the new index file. If the write is successful, the hidden file is deleted.

3. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the method for time-series data persistence based on a data service platform as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method for time-series data persistence based on a data service platform as described in claim 1.

Images