Data merging storage method, system and equipment based on time sequence storage and medium

Abstract

The invention discloses a data merging storage method, system and equipment based on time sequence storage and a medium, belongs to the technical field of time sequence database storage optimization, and aims to solve the technical problems of how to realize data merging of time sequence data and optimize the query performance of a time sequence database. According to the technical scheme, the method comprises the following steps: time sequence storage data merging detection: regularly checking whether the number of to-be-merged batch-to-disk data files exceeds a set threshold value or not, and if so, triggering time sequence storage data merging; and merging the time sequence storage data: merging the data files in batches, and deleting the merged data files in batches. The system comprises a time sequence storage data merging detection unit used for regularly checking whether the number of to-be-merged batch-to-disk data files exceeds a set threshold value or not, and if yes, triggering time sequence storage data merging; and the time sequence storage data merging unit is used for merging the data files in batches and deleting the merged data files in batches.

Description

Technical Field

[0001] The present invention relates to the technical field of time series database storage optimization, and specifically to a data merging and storage method, system, device, and medium based on time series storage. Background Art

[0002] Time Series Storage (TSS) is a storage solution designed for time series data. Time series data refers to continuous data points generated in chronological order and with timestamps. It is widely used in the Internet of Things (IoT), industrial monitoring, financial transactions, operations and maintenance monitoring, and other fields.

[0003] With the rapid development of sensor and data acquisition technologies, time series data is increasingly being used in various fields. Time series data typically records the state changes of a system or phenomenon at different points in time, and therefore exhibits strong temporal correlation and dynamics. However, time series data also faces challenges such as large data volumes, high dimensionality, and information redundancy, all of which pose significant challenges to data query processing.

[0004] Given the booming development of the Internet of Things (IoT) industry, the total amount of time series data is growing rapidly, which poses a significant challenge to data writing and querying. To optimize the writing speed, the database writes time series data directly to disk in batches to ensure data writing performance. However, this also brings some problems. The orderliness of the data written to disk in batches is not guaranteed, which affects query performance to a certain extent.

[0005] Therefore, how to realize the data merging of time series data and optimize the query performance of time series databases are technical problems that need to be solved urgently. Summary of the Invention

[0006] The technical task of the present invention is to provide a data merging storage method, system, device and medium based on time series storage to solve the problem of how to realize data merging of time series data and optimize the query performance of the time series database.

[0007] The technical task of the present invention is achieved in the following manner: a data merging and storage method based on time series storage, the method is specifically as follows:

[0008] Time series storage data merge detection: Regularly check whether the number of data files to be merged in batches exceeds the set threshold. If it exceeds, the time series storage data merge is triggered;

[0009] Time series storage data merging: Merge batches of data files and delete the merged batches of data files.

[0010] As a preference, the time series storage data merging detection is specifically as follows:

[0011] The database starts an asynchronous thread and starts execution checks at regular intervals;

[0012] Each startup checks the number of data files flushed to disk in batches. If the number of data files exceeds 3, a merge of the time series storage data is triggered.

[0013] As a preferred method, the time series storage data merging is specifically as follows:

[0014] Obtain the data files to be merged, read all the block data of the data files to be merged, and sort all the block data of the data files to be merged according to the timestamp;

[0015] Create a data file to be written to save the data information after data merging;

[0016] Create a temporary data block in memory;

[0017] Read all block data of the sorted data files to be merged one by one and write them to the temporary database. At the same time, determine whether the number of rows written into the temporary data block reaches the set threshold:

[0018] If the number of rows of data written into the temporary data block reaches the set threshold, the corresponding temporary data block is written to the disk;

[0019] If the final temporary database is less than the set threshold, the corresponding temporary database will be written back;

[0020] After the data merge is completed, the merged batch data files are deleted.

[0021] More preferably, each temporary data block stores a maximum of 4096 rows of data, and the temporary data block is used to temporarily store data to be written into the data file to be written;

[0022] If 4096 rows of data have been written into the temporary data block, the corresponding temporary data block is flushed to disk.

[0023] If the final temporary data block is less than 4096 rows, the corresponding temporary data block is written back.

[0024] Preferably, the corresponding temporary database is written back as follows:

[0025] According to the column storage format, each column of data is compressed one by one, and the compressed data is appended and saved in a temporary string in the memory;

[0026] After all column data is compressed, the temporary string is appended to the data file to be written to complete the disk write operation.

[0027] Preferably, the corresponding temporary database is written back as follows:

[0028] Write the temporary data blocks back to the data files written in batches so that data can be merged the next time.

[0029] A data merging storage system based on time series storage, the system comprising:

[0030] The time series storage data merging detection unit is used to regularly check whether the number of data files to be merged in batches exceeds the set threshold. If so, the time series storage data merging is triggered;

[0031] The time series storage data merging unit is used to merge batches of data files and delete the merged batches of data files.

[0032] Preferably, the time series storage data merging detection unit includes:

[0033] The startup module is used to start asynchronous threads in the database and start execution checks at regular intervals;

[0034] The checking module is used to check the number of data files flushed to disk in batches each time it is started. If the number of data files exceeds three, the time series storage data merge is triggered;

[0035] The time series storage data merging unit includes:

[0036] The acquisition module is used to obtain the data files to be merged, read all the block data of the data files to be merged, and sort all the block data of the data files to be merged according to the timestamp;

[0037] Create module 1, which is used to create the data file to be written, and is used to save the data information after the data is merged;

[0038] Create module 2, which is used to create temporary data blocks in memory; each temporary data block can store a maximum of 4096 rows of data, and the temporary data block is used to temporarily store data to be written into the data file to be written;

[0039] The reading, writing and judging module is used to read all the block data of the sorted data files to be merged one by one, write them into the temporary database, and judge whether the number of rows of data written into the temporary data block reaches the set threshold;

[0040] The disk write module is used to write the corresponding temporary data block to disk when 4096 rows of data in the temporary data block have been written. Specifically, it compresses each column of data one by one according to the column storage format, and appends the compressed data to a temporary string in the memory. When all the column data are compressed, the temporary string is appended to the data file to be written, completing the disk write operation.

[0041] The write-back processing module is used to write back the corresponding temporary data block when the final temporary data block is less than 4096 rows. Specifically, the temporary data block is written back to the data file written in batches so that data can be merged in the next data merge.

[0042] The deletion module is used to delete the merged batch data files after the data merge is completed.

[0043] An electronic device comprising: a memory and at least one processor;

[0044] Wherein, the memory stores a computer program;

[0045] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the data merging and storing method based on time series storage as described above.

[0046] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the data merging and storage method based on time series storage as described above.

[0047] The data merging and storage method, system, device, and medium based on time series storage of the present invention have the following advantages:

[0048] (1) The present invention merges data that has been flushed to disk in batches, merging multiple data files into one data file and sorting the data to optimize query performance.

[0049] (2) The present invention can merge data based on the data already written to disk, ensuring that the data is arranged in timestamp order, and reduces the number of data files, thereby improving query performance;

[0050] (3) The present invention merges data based on the data stored in batches, reduces the number of files, and improves query performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention will be further described below with reference to the accompanying drawings.

[0052] Attachment Figure 1 A schematic diagram of the structure for merging time series storage data;

[0053] Attachment Figure 2 A schematic diagram of block data. DETAILED DESCRIPTION

[0054] The data merging and storing method, system, device and medium based on time series storage of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1:

[0056] This embodiment provides a data merging and storage method based on time series storage, which is specifically as follows:

[0057] S1. Time series storage data merge detection: Regularly check whether the number of data files to be merged in batches exceeds the set threshold. If so, the time series storage data merge is triggered;

[0058] S2. Merge time series storage data: Merge batches of data files and delete the merged batches of data files.

[0059] The time series storage data merging detection in step S1 of this embodiment is specifically as follows:

[0060] S101. The database starts an asynchronous thread and starts execution checks at a scheduled time.

[0061] S102: Each time the execution is started, the number of data files written to the disk in batches is checked. If the number of data files exceeds 3, a time series storage data merge is triggered.

[0062] As attached Figure 1 As shown, the time series storage data merging in step S2 of this embodiment is specifically as follows:

[0063] S201, obtain the data file to be merged, read all the block data of the data file to be merged, as shown in the attached Figure 2 As shown, all block data of the data files to be merged are sorted by timestamp;

[0064] S202: Create a data file to be written, for storing the data information after the data is merged;

[0065] S203, creating a temporary data block in memory;

[0066] S204: Read all block data of the sorted data files to be merged one by one and write them into a temporary database. At the same time, determine whether the number of rows of data written into the temporary data block reaches a set threshold:

[0067] ① If the number of rows of data written into the temporary data block reaches the set threshold, the corresponding temporary data block is written to the disk;

[0068] ② If the final temporary database is less than the set threshold, the corresponding temporary database will be written back;

[0069] S205: After the data merging is completed, the merged batch data files are deleted.

[0070] In this embodiment, each temporary data block stores a maximum of 4096 rows of data, and the temporary data block is used to temporarily store data to be written into the data file to be written;

[0071] If 4096 rows of data have been written into the temporary data block, the corresponding temporary data block is flushed to disk.

[0072] If the final temporary data block is less than 4096 rows, the corresponding temporary data block is written back.

[0073] In this embodiment, the write-back process for the corresponding temporary database is specifically as follows:

[0074] According to the column storage format, each column of data is compressed one by one, and the compressed data is appended and saved in a temporary string in the memory;

[0075] After all column data is compressed, the temporary string is appended to the data file to be written to complete the disk write operation.

[0076] In this embodiment, the write-back process for the corresponding temporary database is specifically as follows:

[0077] Since the amount of data in the data block is less than 4096 rows, there is no need to write it to the data file to be written. The temporary data block is written back to the data file written in batches so that data can be merged during the next data merge.

[0078] Example 2:

[0079] This embodiment provides a data merging and storage system based on time series storage, the system including:

[0080] The time series storage data merging detection unit is used to regularly check whether the number of data files to be merged in batches exceeds the set threshold. If so, the time series storage data merging is triggered;

[0081] The time series storage data merging unit is used to merge batches of data files and delete the merged batches of data files.

[0082] The time series storage data merging detection unit in this embodiment includes:

[0083] The startup module is used to start asynchronous threads in the database and start execution checks at regular intervals;

[0084] The checking module is used to check the number of data files written to disk in batches each time it is started. If the number of data files exceeds 3, the time series storage data merge is triggered.

[0085] The time series storage data merging unit in this embodiment includes:

[0086] The acquisition module is used to obtain the data files to be merged, read all the block data of the data files to be merged, and sort all the block data of the data files to be merged according to the timestamp;

[0087] Create module 1, which is used to create the data file to be written, and is used to save the data information after the data is merged;

[0088] Create module 2, which is used to create temporary data blocks in memory; each temporary data block can store a maximum of 4096 rows of data, and the temporary data block is used to temporarily store data to be written into the data file to be written;

[0089] The reading, writing and judging module is used to read all the block data of the sorted data files to be merged one by one, write them into the temporary database, and judge whether the number of rows of data written into the temporary data block reaches the set threshold;

[0090] The disk write module is used to write the corresponding temporary data block to disk when 4096 rows of data in the temporary data block have been written. Specifically, it compresses each column of data one by one according to the column storage format, and appends the compressed data to a temporary string in the memory. When all the column data are compressed, the temporary string is appended to the data file to be written, completing the disk write operation.

[0091] The write-back processing module is used to write back the corresponding temporary data block when the final temporary data block is less than 4096 rows. Specifically, the temporary data block is written back to the data file written in batches so that data can be merged in the next data merge.

[0092] The deletion module is used to delete the merged batch data files after the data merge is completed.

[0093] Example 3:

[0094] This embodiment also provides an electronic device, including: a memory and a processor;

[0095] wherein the memory stores computer-executable instructions;

[0096] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the data merging and storing method based on time series storage in any embodiment of the present invention.

[0097] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0098] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0099] Example 4:

[0100] This embodiment further provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor and cause the processor to execute the data merging and storage method based on time series storage according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0101] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0102] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0103] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0104] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data merging and storage method based on time series storage, characterized in that: The method is as follows: Time series storage data merge detection: Regularly check whether the number of data files to be merged in batches exceeds the set threshold. If it exceeds, the time series storage data merge is triggered; Time series storage data merging: Merge batches of data files and delete the merged batches of data files.

2. The data merging and storage method based on time series storage according to claim 1 is characterized in that: The details of time series storage data merging detection are as follows: The database starts an asynchronous thread and starts execution checks at regular intervals; Each startup checks the number of data files flushed to disk in batches. If the number of data files exceeds 3, a merge of the time series storage data is triggered.

3. The data merging and storage method based on time series storage according to claim 1 is characterized in that: The details of time series storage data merging are as follows: Obtain the data files to be merged, read all the block data of the data files to be merged, and sort all the block data of the data files to be merged according to the timestamp; Create a data file to be written to save the data information after data merging; Create a temporary data block in memory; Read all block data of the sorted data files to be merged one by one and write them to the temporary database. At the same time, determine whether the number of rows written into the temporary data block reaches the set threshold: If the number of rows of data written into the temporary data block reaches the set threshold, the corresponding temporary data block is written to the disk; If the final temporary database is less than the set threshold, the corresponding temporary database will be written back; After the data merge is completed, the merged batch data files are deleted.

4. The data merging and storage method based on time series storage according to claim 3 is characterized in that: Each temporary data block can store up to 4096 rows of data. The temporary data block is used to temporarily store data to be written into the data file to be written. If 4096 rows of data have been written into the temporary data block, the corresponding temporary data block is flushed to disk. If the final temporary data block is less than 4096 rows, the corresponding temporary data block is written back.

5. The data merging and storing method based on time series storage according to claim 3 is characterized in that: The specific write-back processing for the corresponding temporary database is as follows: According to the column storage format, each column of data is compressed one by one, and the compressed data is appended and saved in a temporary string in the memory; After all column data is compressed, the temporary string is appended to the data file to be written to complete the disk write operation.

6. The data merging and storing method based on time series storage according to claim 3 is characterized in that: The specific write-back processing for the corresponding temporary database is as follows: Write the temporary data blocks back to the data files written in batches so that data can be merged the next time.

7. A data merging storage system based on time series storage, characterized in that: The system includes: The time series storage data merging detection unit is used to regularly check whether the number of data files to be merged in batches exceeds the set threshold. If so, the time series storage data merging is triggered; The time series storage data merging unit is used to merge batches of data files and delete the merged batches of data files.

8. The data merging storage system based on time series storage according to claim 7, characterized in that: The time series storage data merging detection unit includes: The startup module is used to start asynchronous threads in the database and start execution checks at regular intervals; The checking module is used to check the number of data files flushed to disk in batches each time it is started. If the number of data files exceeds three, the time series storage data merge is triggered; The time series storage data merging unit includes: The acquisition module is used to obtain the data files to be merged, read all the block data of the data files to be merged, and sort all the block data of the data files to be merged according to the timestamp; Create module 1, which is used to create the data file to be written, and is used to save the data information after the data is merged; Create module 2, which is used to create temporary data blocks in memory; each temporary data block can store a maximum of 4096 rows of data, and the temporary data block is used to temporarily store data to be written into the data file to be written; The reading, writing and judging module is used to read all the block data of the sorted data files to be merged one by one, write them into the temporary database, and judge whether the number of rows of data written into the temporary data block reaches the set threshold; The disk write module is used to write the corresponding temporary data block to disk when 4096 rows of data in the temporary data block have been written. Specifically, it compresses each column of data one by one according to the column storage format, and appends the compressed data to a temporary string in the memory. When all the column data are compressed, the temporary string is appended to the data file to be written, completing the disk write operation. The write-back processing module is used to write back the corresponding temporary data block when the final temporary data block is less than 4096 rows. Specifically, the temporary data block is written back to the data file written in batches so that data can be merged in the next data merge. The deletion module is used to delete the merged batch data files after the data merge is completed.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the data merging and storing method based on time series storage according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the data merging and storing method based on time series storage according to any one of claims 1 to 6.

Images