A method for storing and querying time series data
By the method of leaving the time -sequential data in the memory and then falling on the memory, combining the management method of the MEMLIST array and the FileList array, solving the problem of poor time -sequential data storage and querying performance, and realized efficient storage and fast inquiries.
Patent Information
- Application Number
- CN202210484342.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-06
- Publication Date
- 2025-01-24
- Estimated Expiration
- 2042-05-06
AI Technical Summary
The prior art is difficult to efficiently store and query massive time series data, resulting in poor storage efficiency and query performance.
Adopting the combination of memory storage and file storage, the newly written timing data first enters the internal stored storage temporary deposit, and then falls into the file storage. efficiency.
It achieves efficient storage and rapid warehousing of massive time series data, and improves the overall performance of query operations. Especially when locating time series data greater than or equal to a certain time point, the time complexity is O(1).
Smart Images

Figure CN114896268B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of databases, and in particular to a time series data storage and query method. Background Art
[0002] Time series data refers to time series data; data columns recorded in chronological order. Each data in the same data column must be of the same caliber and must be comparable. Time series data can be a period number or a time point number. The characteristics of time series data are: (1) The data is time-series and must have a timestamp: networked devices continuously generate data according to a set period; (2) The data is structured; (3) The data volume is large.
[0003] At present, the storage and processing of time series data are often handled by relational databases, but due to the inherent disadvantages of relational databases, they cannot perform efficient storage and data query. Therefore, how to achieve efficient storage and rapid warehousing of massive time series data, while improving the overall performance of query operations, is a technical problem that needs to be solved urgently. Summary of the invention
[0004] The technical task of the present invention is to provide a time series data storage and query method to solve the problem of how to achieve efficient storage and rapid warehousing of massive time series data while improving the overall performance of query operations.
[0005] The technical task of the present invention is achieved in the following way: a time series data storage and query method, the method is as follows:
[0006] Storage: The storage mode is divided into memory storage and file storage. The newly written time series data is first temporarily stored in the memory storage, and then written to the disk for storage in the file storage;
[0007] Query: The query method is divided into memory storage data query and file storage data query.
[0008] As a preference, the storage is as follows:
[0009] Use Memlist array and Filelist array to manage all time series data;
[0010] When new time series data is inserted, it first enters the Memlist array and the insertion operation is completed;
[0011] One or more background threads write the time series data that has been organized in the Memlist array to disk to form files. The time series data after being written to disk is continued to be managed by the Filelist array, and the Memlist array is no longer managed.
[0012] Preferably, the Memlist array is used to manage time series data stored in memory; the space capacity of the Memlist array is defined in the program configuration file, and the default capacity is 8; the Memlist array has been initialized when the program starts, and the length of the Memlist array will not change during the execution of the program;
[0013] The Memlist array stores pointers to MetricVector objects. MetricVector objects store a specific amount of time series data. The quantity threshold is configurable and defaults to 64M. The MetricVector object contains a Blocks array, which stores pointers to Block memory blocks. The Blocks array contains several Blocks, and each Block memory block stores the inserted time series data Metrics in sequence. Each time series data Metric represents a time series data. The structure of Metric time series data is a timestamp and a specific value. Since the timestamp type and the value type are both fixed, the space occupied by each time series data Metric is fixed, even through a certain compression algorithm. Therefore, the time series data Metric can be directly written to the Block memory block by appending, and each appended record can be parsed through the fixed length characteristics of the time series data Metric during the read operation. Among them, the size of the Block memory block is configurable, and the default configuration is 1M.
[0014] The structure of Metric time series data also includes the curOffset variable, which is used to count the position of the latest time series data currently stored in the Block memory block being inserted;
[0015] Whenever the timing data in the Block memory block is full, the program re-applies for a new Block memory block from the memory storage, points the curOffset variable to the first position of the new Block memory block, and inserts the new Block memory block into the array Blocks.
[0016] Preferably, the Filelist array is used to manage the time series data that has been written to the disk; the files are stored in the Filelist array in the order of their generation. The file objects stored in the Filelist array include the absolute path of the file, the file's minTS (minimum timestamp), maxTS (maximum timestamp) and the amount of data (the amount of time series data in each file is the same);
[0017] The time series data in the MetricVector object is written to the file storage in chronological order, and a fixed-length footer is added to the end of the file. The footer contains minTS, maxTS, the size of the space occupied by a metric, the number of metric records, and a checksum.
[0018] As a preferred method, the data query of memory storage is as follows:
[0019] S1. Locate the MetricVector where seekTS is located;
[0020] S2, locate seekTS in the Block memory block of MetricVector;
[0021] S3, locate the position of seekTS in the Block memory block;
[0022] S4. Return position: Returns the position of the indexOpt-th record in the Block memory block.
[0023] Preferably, the MetricVector where seekTS is located is as follows:
[0024] S101. Assume that the length of the Memlist array is L, the minimum timestamp of the time series data stored in the Memlist array is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and query the MetricVector in which seekTS exists in the Memlist array, using the following formula:
[0025]
[0026] S102, after calculating the index, determine whether the index is equal to -1:
[0027] ①. If yes, it means that the data of seekTS is not in the memory;
[0028] ②, if not, execute step S103;
[0029] S103. Determine whether seekTS is within the minTS and maxTS range of the MetricVector object Memlist[index]:
[0030] ①. If yes, jump to step S105;
[0031] ②, if not, execute step S104;
[0032] S104, perform a fine adjustment of index by adding or subtracting one, the formula is as follows:
[0033]
[0034] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the MetricVector object Memlist[index]; indexOpt refers to the fine-tuned index;
[0035] S105, determining the MetricVector object to continue positioning;
[0036] Preferably, the location of seekTS in the Block memory block of MetricVector is as follows:
[0037] S201, assuming that the length of the array Blocks in MetricVector is L, the minimum timestamp of the time series data stored in MetricVector is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is in which Block in Blocks seekTS exists, using the following formula:
[0038]
[0039] S202, after calculating index, determine whether seekTS is within the minTS and maxTS range of Blocks[index]:
[0040] ①. If yes, jump to step S204;
[0041] ②, if not, execute step S203,
[0042] S203, perform a fine adjustment of the index by adding or subtracting one, and the fine adjustment formula is as follows:
[0043]
[0044] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the Block of Blocks[index], and indexOpt refers to the fine-tuned index;
[0045] S204, determining the Block memory block to be positioned further;
[0046] The location of seekTS in the Block memory block is as follows:
[0047] S301, assuming that the amount of time series data stored in the Block memory block is L, the minimum timestamp of the time series data stored in the Block memory block is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query seekTS is the record number in the Block, the formula is as follows:
[0048]
[0049] S302: After calculating the index, determine whether the current position is correct:
[0050] If it is incorrect, execute step S303;
[0051] S303, perform one or more fine adjustments of index plus or minus one, the formula is as follows:
[0052]
[0053] Among them, Block[index] refers to the timestamp of the index-th record stored in the Block memory block; Block[index+1] refers to the timestamp of the index+1-th record; indexOpt refers to the fine-tuned index;
[0054] As a preferred method, the data query of the file storage is as follows:
[0055] S1. Locate the File where seekTS is located; the details are as follows:
[0056] S101. Assume that the length of the Filelist array is L, the minimum timestamp of the time series data stored in the Filelist array is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is performed on the file in which seekTS exists in the Filelist. The formula is as follows:
[0057]
[0058] S102, after calculating the index, determine whether the index is -1:
[0059] ①. If yes, it means that the data of seekTS is not in the file, and the record of minTS is taken as the starting query position for positioning;
[0060] ②, if not, execute step S103;
[0061] S103, determine whether seekTS is within the minTS and maxTS interval of the File file Memlist[index]:
[0062] ①. If yes, jump to step S105;
[0063] ②, if not, execute step S104;
[0064] S104: Perform a fine adjustment of the index by adding or subtracting one, as follows:
[0065]
[0066] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the File object Filelist[index], and indexOpt refers to the fine-tuned index;
[0067] S105, determining the File file to be located;
[0068] S2. Locate seekTS in the data block page of the File file; the details are as follows:
[0069] S201, when disk IO is performed, each read is performed according to the data block. When querying, the data block is located first to improve the overall query efficiency;
[0070] S202, assuming that the total number of data block pages in the File file is L, the minimum timestamp of the time series data stored in the File is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is in which block seekTS is, as follows:
[0071]
[0072] S203, after calculating the index, read the data block page from the storage medium;
[0073] S204, determine whether seekTS is within the minTS and maxTS range of this data block page:
[0074] If yes, execute step S205;
[0075] S205: Perform a fine adjustment of the index by adding or subtracting one, as follows:
[0076]
[0077] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the data block, and indexOpt refers to the fine-tuned index;
[0078] S206: After fine-tuning, the data block page is immediately read out from the storage medium for judgment and analysis.
[0079] S207: Determine the file data block page to be located, which has been read into the memory.
[0080] S3. Locate the position of seekTS in the data block page; the details are as follows:
[0081] S301, assuming that the page length of the data block is L, the minimum timestamp of the stored time series data is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query seekTS is the number of records in the Block, as follows:
[0082]
[0083] S302: After calculating the index, determine whether the current position is correct:
[0084] If it is incorrect, execute step S303;
[0085] S303: Perform one or more fine-tuning of the index by adding or subtracting one. The details are as follows:
[0086]
[0087] Wherein, page[index] refers to the timestamp of the index-th record stored in the data block page, page[index+1] refers to the timestamp of the index+1-th record; indexOpt refers to the index after fine-tuning;
[0088] S4. Return position: Return the position of the indexOpt-th record in the data block page.
[0089] An electronic device comprising: a memory and at least one processor;
[0090] Wherein, the memory stores a computer program;
[0091] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the time series data storage and query method as described above.
[0092] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the time series data storage and query method as described above.
[0093] Among them, Block: memory block, a continuous memory area requested from the operating system.
[0094] Metric: represents a time series data.
[0095] TS: or ts, means timestamp, a specific point in time.
[0096] Page: A data block that represents a storage unit that can be operated at one time on a storage medium such as a disk.
[0097] The time series data storage and query method of the present invention has the following advantages:
[0098] (i) Aiming at the fixed collection frequency of time series data, the present invention provides a positioning algorithm with a time complexity of O(1), which can quickly locate the position of time series data greater than or equal to a certain time point, thereby improving the overall performance of query operations;
[0099] (II) The present invention provides an append-based storage structure for the time series data's time series characteristics and structured characteristics. The write operation only needs to append the new time series data to the memory array and then return immediately, making the write operation very efficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0100] The present invention is further described below in conjunction with the accompanying drawings.
[0101] Attached Figure 1 It is a schematic diagram of the structure of MetricVector;
[0102] Attached Figure 2 This is a schematic diagram of the structure of the file File formed by disk. DETAILED DESCRIPTION
[0103] A time series data storage and query method of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments of the specification.
[0104] Embodiment 1:
[0105] The present invention provides a time series data storage and query method, the method is as follows:
[0106] Storage: The storage mode is divided into memory storage and file storage. The newly written time series data is first temporarily stored in the memory storage, and then written to the disk for storage in the file storage;
[0107] Query: The query method is divided into memory storage data query and file storage data query.
[0108] The storage in this embodiment is as follows:
[0109] (1) Use Memlist array and Filelist array to manage all time series data;
[0110] (2) When new time series data is inserted, it first enters the Memlist array and the insertion operation is completed;
[0111] (3) One or more background threads write the time series data that has been organized in the Memlist array to disk to form files. The time series data after being written to the disk is continued to be managed by the Filelist array, and the Memlist array is no longer managed.
[0112] The Memlist array in this embodiment is used to manage the time series data stored in the memory; the space capacity of the Memlist array is defined in the program configuration file, and the default capacity is 8; the Memlist array has been initialized when the program is started, and the length of the Memlist array will not change during the execution of the program;
[0113] As attached Figure 1 As shown in the figure, the Memlist array stores the pointer of the MetricVector object. The MetricVector object stores a specific amount of time series data. The quantity threshold is configurable and the default value is 64M. The MetricVector object has a Blocks array, which stores the pointer of the Block memory block. The Blocks array contains several Blocks, and each Block memory block sequentially stores the inserted time series data Metric, and each time series data Metric represents a time series data. The structure of the Metric time series data is a timestamp and a specific value. Since the timestamp type and the value type are both fixed, the space occupied by each time series data Metric is fixed, even through a certain compression algorithm, the space occupied by each time series data Metric is also fixed. Therefore, the time series data Metric can be directly written into the Block memory block by appending, and each appended record can be parsed through the fixed length characteristics of the time series data Metric during the read operation. Among them, the size of the Block memory block is configurable, and the default configuration is 1M.
[0114] The structure of Metric time series data also includes the curOffset variable, which is used to count the position of the latest time series data currently stored in the Block memory block being inserted;
[0115] Whenever the timing data in the Block memory block is full, the program re-applies for a new Block memory block from the memory storage, points the curOffset variable to the first position of the new Block memory block, and inserts the new Block memory block into the array Blocks.
[0116] The Filelist array in this embodiment is used to manage the time series data that has been stored on the disk, as shown in the attached Figure 2 As shown in the figure, the files are stored in the Filelist array in the order of their generation time. The file objects stored in the Filelist array include the absolute path of the file, the minTS (minimum timestamp) and maxTS (maximum timestamp) of the file, and the amount of data (the amount of time series data in each file is the same).
[0117] The time series data in the MetricVector object is written to the file storage in chronological order, and a fixed-length footer is added to the end of the file. The footer contains minTS, maxTS, the size of the space occupied by a metric, the number of metric records, and a checksum.
[0118] Time series data is usually queried using a range query, that is, to query whether a certain segment contains all the time series data within the range. Since the time series data in memory and files are arranged sequentially, it is critical to quickly find the starting position of the range query, that is, how to quickly find the first time series data greater than or equal to a certain time point. The following provides an algorithm for locating the time series data of seekTS at a certain time point in memory and files.
[0119] The data query of the memory storage in this embodiment is specifically as follows:
[0120] S1. Locate the MetricVector where seekTS is located; the details are as follows:
[0121] S101. Assume that the length of the Memlist array is L, the minimum timestamp of the time series data stored in the Memlist array is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and query the MetricVector in which seekTS exists in the Memlist array, using the following formula:
[0122]
[0123] S102, after calculating the index, determine whether the index is equal to -1:
[0124] ①. If yes, it means that the data of seekTS is not in the memory;
[0125] ②, if not, execute step S103;
[0126] S103. Determine whether seekTS is within the minTS and maxTS range of the MetricVector object Memlist[index]:
[0127] ①. If yes, jump to step S105;
[0128] ②, if not, execute step S104;
[0129] S104, perform a fine adjustment of index by adding or subtracting one, the formula is as follows:
[0130]
[0131] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the MetricVector object Memlist[index]; indexOpt refers to the fine-tuned index;
[0132] S105, determining the MetricVector object to continue positioning;
[0133] S2. Locate seekTS in the Block memory block of MetricVector; the details are as follows:
[0134] S201, assuming that the length of the array Blocks in MetricVector is L, the minimum timestamp of the time series data stored in MetricVector is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is in which Block in Blocks seekTS exists, using the following formula:
[0135]
[0136] S202, after calculating index, determine whether seekTS is within the minTS and maxTS range of Blocks[index]:
[0137] ①. If yes, jump to step S204;
[0138] ②, if not, execute step S203,
[0139] S203, perform a fine adjustment of the index by adding or subtracting one, and the fine adjustment formula is as follows:
[0140]
[0141] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the Block of Blocks[index], and indexOpt refers to the fine-tuned index;
[0142] S204, determining the Block memory block to be positioned further;
[0143] S3. Locate the position of seekTS in the Block memory block; the details are as follows:
[0144] S301, assuming that the amount of time series data stored in the Block memory block is L, the minimum timestamp of the time series data stored in the Block memory block is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query seekTS is the record number in the Block, the formula is as follows:
[0145]
[0146] S302: After calculating the index, determine whether the current position is correct:
[0147] If it is incorrect, execute step S303;
[0148] S303, perform one or more fine adjustments of index plus or minus one, the formula is as follows:
[0149]
[0150] Among them, Block[index] refers to the timestamp of the index-th record stored in the Block memory block; Block[index+1] refers to the timestamp of the index+1-th record; indexOpt refers to the fine-tuned index;
[0151] S4. Return position: Returns the position of the indexOpt-th record in the Block memory block.
[0152] The data query of the file storage in this embodiment is specifically as follows:
[0153] S1. Locate the File where seekTS is located; the details are as follows:
[0154] S101. Assume that the length of the Filelist array is L, the minimum timestamp of the time series data stored in the Filelist array is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is performed on the file in which seekTS exists in the Filelist. The formula is as follows:
[0155]
[0156] S102, after calculating the index, determine whether the index is -1:
[0157] ①. If yes, it means that the data of seekTS is not in the file, and the record of minTS is taken as the starting query position for positioning;
[0158] ②, if not, execute step S103;
[0159] S103, determine whether seekTS is within the minTS and maxTS interval of the File file Memlist[index]:
[0160] ①. If yes, jump to step S105;
[0161] ②, if not, execute step S104;
[0162] S104: Perform a fine adjustment of the index by adding or subtracting one, as follows:
[0163]
[0164] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the File object Filelist[index], and indexOpt refers to the fine-tuned index;
[0165] S105, determining the File file to be located;
[0166] S2. Locate seekTS in the data block page of the File file; the details are as follows:
[0167] S201, when disk IO is performed, each read is performed according to the data block. When querying, the data block is located first to improve the overall query efficiency;
[0168] S202, assuming that the total number of data block pages in the File file is L, the minimum timestamp of the time series data stored in the File is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is in which block seekTS is, as follows:
[0169]
[0170] S203, after calculating the index, read the data block page from the storage medium;
[0171] S204, determine whether seekTS is within the minTS and maxTS range of this data block page:
[0172] If yes, execute step S205;
[0173] S205: Perform a fine adjustment of the index by adding or subtracting one, as follows:
[0174]
[0175] Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the data block, and indexOpt refers to the fine-tuned index;
[0176] S206: After fine-tuning, the data block page is immediately read out from the storage medium for judgment and analysis.
[0177] S207: Determine the file data block page to be located, which has been read into the memory.
[0178] S3. Locate the position of seekTS in the data block page; the details are as follows:
[0179] S301, assuming that the page length of the data block is L, the minimum timestamp of the stored time series data is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query seekTS is the number of records in the Block, as follows:
[0180]
[0181] S302: After calculating the index, determine whether the current position is correct:
[0182] If it is incorrect, execute step S303;
[0183] S303: Perform one or more fine-tuning of the index by adding or subtracting one. The details are as follows:
[0184]
[0185] Wherein, page[index] refers to the timestamp of the index-th record stored in the data block page, page[index+1] refers to the timestamp of the index+1-th record; indexOpt refers to the index after fine-tuning;
[0186] S4. Return position: Return the position of the indexOpt-th record in the data block page.
[0187] Embodiment 2:
[0188] An embodiment of the present invention further provides an electronic device, comprising: a memory and a processor;
[0189] Wherein, the memory stores computer-executable instructions;
[0190] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the time series data storage and query method in any embodiment of the present invention.
[0191] Embodiment 3:
[0192] The embodiment of the present invention further provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the time series data storage and query method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.
[0193] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
[0194] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.
[0195] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.
[0196] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.
[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for storing and querying time series data, characterized in that: The method is as follows: Storage: The storage mode is divided into memory storage and file storage. The newly written time series data is first temporarily stored in the memory storage, and then written to the disk for storage in the file storage. Query: The query method is divided into memory storage data query and file storage data query; The storage is as follows: Use Memlist array and Filelist array to manage all time series data; When new time series data is inserted, it first enters the Memlist array and the insertion operation is completed; One or more background threads write the time series data that has been organized in the Memlist array to disk to form files. The time series data after being written to the disk is continued to be managed by the Filelist array, and the Memlist array is no longer managed; The Memlist array is used to manage time series data stored in memory. The space capacity of the Memlist array is defined in the program configuration file. The Memlist array is initialized when the program starts. During the execution of the program, the length of the Memlist array does not change. The Memlist array stores pointers to MetricVector objects. MetricVector objects store a specific amount of time series data, and the quantity threshold is configurable. The MetricVector object contains a Blocks array, which stores pointers to Block memory blocks. The Blocks array contains several Blocks, and each Block memory block stores the inserted time series data Metric in sequence. The structure of each time series data Metric is a timestamp and a specific value. The time series data Metric is written to the Block memory block by appending. During the read operation, each appended record is parsed through the fixed length feature of the time series data Metric. The size of the Block memory block is configurable. The structure of Metric time series data also includes the curOffset variable, which is used to count the position of the latest time series data currently stored in the Block memory block being inserted; Whenever the timing data in the Block memory block is full, the program re-applies for a new Block memory block from the memory storage, points the curOffset variable to the first position of the new Block memory block, and inserts the new Block memory block into the array Blocks; The Filelist array is used to manage the time series data that has been written to the disk. The files are stored in the Filelist array in the order of their generation. The file objects stored in the Filelist array include the absolute path of the file, the minTS, maxTS and data volume of the file. The time series data in the MetricVector object is written to the file storage in chronological order, and a fixed-length footer is added to the end of the file. The footer contains minTS, maxTS, the size of the space occupied by a metric, the number of metric records, and a checksum.
2. The method for storing and querying time series data according to claim 1, characterized in that: The data query of memory storage is as follows: S1. Locate the MetricVector where seekTS is located; S2, locate seekTS in the Block memory block of MetricVector; S3, locate the position of seekTS in the Block memory block; S4. Return position: Return the position of the indexOpt-th record in the Block memory block.
3. The time series data storage and query method according to claim 2, characterized in that: The MetricVector for locating seekTS is as follows: S101, assuming that the length of the Memlist array is L, the minimum timestamp of the time series data stored in the Memlist array is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is performed in which MetricVector in the Memlist array seekTS exists, using the following formula: ; S102, after calculating the index, determine whether the index is equal to -1: ①. If yes, it means that the data of seekTS is not in the memory; ②, if not, execute step S103; S103. Determine whether seekTS is within the minTS and maxTS range of the MetricVector object Memlist[index]: ①. If yes, jump to step S105; ②, if not, execute step S104; S104, perform a fine adjustment of index by adding or subtracting one, the formula is as follows: ; Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the MetricVector object Memlist[index]; indexOpt refers to the fine-tuned index; S105: Determine the MetricVector object to continue positioning.
4. The method for storing and querying time series data according to claim 2, characterized in that: The specific location of seekTS in the Block memory block of MetricVector is as follows: S201, assuming that the length of the array Blocks in MetricVector is L, the minimum timestamp of the time series data stored in MetricVector is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is in which Block in Blocks seekTS exists, using the following formula: ; S202, after calculating index, determine whether seekTS is within the minTS and maxTS range of Blocks[index]: ①. If yes, jump to step S204; ②, if not, execute step S203, S203, perform a fine adjustment of the index by adding or subtracting one, and the fine adjustment formula is as follows: ; Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the Block of Blocks[index], and indexOpt refers to the fine-tuned index; S204, determining the Block memory block to continue locating; The location of seekTS in the Block memory block is as follows: S301, assuming that the amount of time series data stored in the Block memory block is L, the minimum timestamp of the time series data stored in the Block memory block is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query seekTS is the record number in the Block, the formula is as follows: ; S302: After calculating the index, determine whether the current position is correct: If it is incorrect, execute step S303; S303, perform one or more fine adjustments of index plus or minus one, the formula is as follows: ; in, Refers to the timestamp of the index-th record stored in the Block memory block; Indicates the timestamp of the index+1th record; indexOpt refers to the fine-tuned index.
5. The method for storing and querying time series data according to claim 1, characterized in that: The data query of file storage is as follows: S1. Locate the File where seekTS is located; the details are as follows: S101, let the length of Filelist array be L, the minimum timestamp of the time series data stored in the Filelist array be minTS, and the maximum timestamp be maxTS; the time point to be queried is seekTS, and the query is in which file in Filelist the seekTS exists. The formula is as follows: ; S102, after calculating the index, determine whether the index is -1: ①. If yes, it means that the data of seekTS is not in the file, and the record of minTS is taken as the starting query position for positioning; ②, if not, execute step S103; S103, determine whether seekTS is within the minTS and maxTS interval of the File file Memlist[index]: ①. If yes, jump to step S105; ②, if not, execute step S104; S104: Perform a fine adjustment of the index by adding or subtracting one, as follows: ; Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the File object Filelist[index], and indexOpt refers to the fine-tuned index; S105, determining the File file to be located; S2. Locate seekTS in the data block page of the File file; the details are as follows: S201, when performing disk IO, each read is performed according to the data block. When querying, the data block is located first to improve the overall query efficiency; S202, assuming that the total number of data block pages in the File file is L, the minimum timestamp of the time series data stored in the File is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query is in which block seekTS is, as follows: ; S203, after calculating the index, read the data block page from the storage medium; S204, determine whether seekTS is within the minTS and maxTS range of this data block page: If yes, execute step S205; S205: Perform a fine adjustment of the index by adding or subtracting one, as follows: ; Among them, minTS and maxTS refer to the minimum and maximum timestamps of the time series data in the data block, and indexOpt refers to the fine-tuned index; S206, after fine-tuning, immediately read the data block page from the storage medium for judgment and analysis; S207, determining the file data block page to be located, the data block page has been read into the memory; S3. Locate the position of seekTS in the data block page; the details are as follows: S301, assuming that the page length of the data block is L, the minimum timestamp of the stored time series data is minTS, and the maximum timestamp is maxTS; the time point to be queried is seekTS, and the query seekTS is the number of records in the Block, as follows: ; S302: After calculating the index, determine whether the current position is correct: If it is incorrect, execute step S303; S303, perform one or more fine adjustments of index addition or subtraction; the details are as follows: ; Among them, page Refers to the timestamp of the index record stored in the data block page. Indicates the timestamp of the index+1th record; indexOpt refers to the fine-tuned index; S4. Return position: Return the position of the indexOpt-th record in the data block page.
6. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the time series data storage and query method according to any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the time series data storage and query method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Time series data storing and querying method and storing and processing platform
CN108399263A
Data storage and query method and device, storage medium and electronic equipment
CN114116795A