Time sequence database query optimization method and device

Abstract

The invention discloses a time sequence database query optimization method and device, and relates to the technical field of data query. Comprising the following steps: step 1: persisting reverse index information to a disk file; step 2: performing multi-dimensional condition filtering query by using a reverse index: filtering in a table block according to a time sequence table name, positioning to a file area where all dimension values corresponding to the table block are located, obtaining a Hash Index of a tag key area corresponding to the time sequence table name, performing hash operation by using the dimension name, and storing the hash operation in a file area where all dimension values corresponding to the table block are located; the method comprises the steps that a dimension name is obtained, a storage area of a dimension value corresponding to the dimension name is positioned, Hash Index of the dimension value area corresponding to the dimension name is obtained, hash operation is carried out through the dimension value, all SeriesIDs corresponding to the dimension value and the offset of Series Key corresponding to the SeriesIDs in a disk file are positioned, the corresponding SeriesKey is loaded according to the SeriesIDs, and a corresponding result set meeting a query condition is searched in the disk file according to the SeriesKey, a fieldKey and a time range.

Description

Technical Field

[0001] The invention discloses a method and a device, and relates to the technical field of data query, in particular to a time series database query optimization method and a device. Background Art

[0002] Time series data is a series of data generated over time, that is, data with timestamps. A time series database (TSDB) is a database used to ingest, process, and store time series data. Time series data may come from various scenarios such as metrics from servers and applications, readings from IoT sensors, user interactions on websites or applications, or trading activities in financial markets. In IoT and big data applications, time series data is often distributed in different modules or devices, and multi-mode queries are required to obtain a complete view of time series data. However, existing database systems often cannot effectively handle multi-mode query situations. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a time series database query optimization method and device, which improves the query performance of multi-mode queries and reduces resource consumption through asynchronous reading.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a time series database query optimization method, comprising:

[0006] Step 1: Persist the inverted index information to a disk file:

[0007] Sort the series ByID and seriesByTagKeyValue data,

[0008] Persisting SeriesKey,

[0009] Persist tagValue,

[0010] Persist all tagKeys to form Tag Block,

[0011] Persist the metadata information of the time series table to form a table block;

[0012] Step 2: Use the inverted index to perform multi-dimensional condition filtering query:

[0013] Filter in table Block according to the time series table name and locate the file area where all dimension values ​​corresponding to the table Block are located.

[0014] Get the Hash Index of the tag key area corresponding to the time series table name, use the dimension name to perform hash operation, and locate the storage area of ​​the dimension value corresponding to the dimension name.

[0015] Get the Hash Index of the dimension value area corresponding to the dimension name, use the dimension value to perform hash operation, locate all SeriesIDs corresponding to the dimension value, and the SeriesID corresponds to the offset of the Series Key in the disk file.

[0016] Load the corresponding SeriesKey according to SeriesID,

[0017] Search the disk file for the corresponding result set that meets the query conditions based on SeriesKey, fieldKey, and time range.

[0018] Furthermore, in the time series database query optimization method, in step 1, a persistence thread is used to determine whether the accumulated inverted index information reaches a threshold or satisfies a condition for performing a persistence operation, and if so, the inverted index information in the memory is written into a disk file.

[0019] Furthermore, when persisting SeriesKey in step 1 of the time series database query optimization method, the Series Block is first constructed, and the corresponding relationship between the SeriesKey and the offset of the Key in the disk file is recorded while constructing. The corresponding relationship is expressed as<SeriesKey,SeriesKeyOffset> .

[0020] Furthermore, when persisting tagValue in step 1 of the time series database query optimization method, all tagValues ​​under the same tagKey are persisted together and a corresponding Hash Index is generated and written into a disk file.

[0021] The present invention also provides a time series database query optimization device, including an index persistence module and a filtering query module.

[0022] The index persistence module persists the inverted index information to disk files:

[0023] Sort the series ByID and seriesByTagKeyValue data,

[0024] Persisting SeriesKey,

[0025] Persist tagValue,

[0026] Persist all tagKeys to form Tag Block,

[0027] Persist the metadata information of the time series table to form a table block;

[0028] The filter query module uses the inverted index to perform multi-dimensional conditional filter queries:

[0029] Filter in table Block according to the time series table name and locate the file area where all dimension values ​​corresponding to the table Block are located.

[0030] Get the Hash Index of the tag key area corresponding to the time series table name, use the dimension name to perform hash operation, and locate the storage area of ​​the dimension value corresponding to the dimension name.

[0031] Get the Hash Index of the dimension value area corresponding to the dimension name, use the dimension value to perform hash operation, locate all SeriesIDs corresponding to the dimension value, and the SeriesID corresponds to the offset of the Series Key in the disk file.

[0032] Load the corresponding SeriesKey according to SeriesID,

[0033] Search the disk file for the corresponding result set that meets the query conditions based on SeriesKey, fieldKey, and time range.

[0034] Furthermore, the index persistence module of the time series database query optimization device uses a persistence thread to determine whether the accumulated inverted index information reaches a threshold or satisfies a condition for executing a persistence operation, and if so, writes the inverted index information in the memory into a disk file.

[0035] Furthermore, when the index persistence module of the time series database query optimization device persists the SeriesKey, the Series Block is first constructed, and the corresponding relationship between the SeriesKey and the offset of the Key in the disk file is recorded while the construction is being carried out. The corresponding relationship is expressed as<SeriesKey,SeriesKeyOffset> .

[0036] Furthermore, when the index persistence module of the time series database query optimization device persists tagValue, all tagValues ​​under the same tagKey are persisted together and a corresponding Hash Index is generated and written into a disk file.

[0037] The benefits of the present invention are:

[0038] The present invention persists the inverted index information to a disk file, and uses the inverted index to perform multi-dimensional conditional filtering queries, thereby improving the query performance of multi-mode queries and reducing resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic flow chart of the method of the present invention.

[0040] Figure 2 It is a schematic diagram of the persistence process of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0042] Example 1

[0043] The present invention provides a time series database query optimization method, comprising:

[0044] Step 1: Persist the inverted index information to the disk file. You can first save the constructed inverted index information in the WAL (Write Ahead Log) pre-write log, and then update it to the memory data structure. When the persistence conditions are met, the inverted index information in the memory is persisted to the disk file.

[0045] The inverted index file on disk mainly consists of 4 parts:

[0046] Index File Trailer, which indicates the tail of the index file: it records the offset and data size of Table Block, Tag Block, and Series Block in the inverted index file);

[0047] Table Block: stores information about tables in the database, including table structure, attributes, and other details.

[0048] Tag Block: used to store the actual data in the file indexed by the tag key-value pair;

[0049] Series Block: stores information about all sequence key values ​​in the database.

[0050] The specific process is:

[0051] Step 11: Use the persistence thread to determine whether the accumulated inverted index information reaches a threshold or meets the conditions for performing a persistence operation. If so, write the inverted index information in the memory to a disk file. The conditions for performing a persistence operation may be stop, exception handling, and other operations.

[0052] Step 12: When performing persistent storage, sort the seriesByID and seriesByTagKeyValue data. seriesByID and seriesByTagKeyValue contain the identifiers of all series in the database and the sequence indexes organized by tag key-value pairs.

[0053] Step 13: Persist SeriesKey. First, build Series Block. While building, record the correspondence between SeriesKey and the offset of the Key in the disk file. The correspondence is expressed as<SeriesKey,SeriesKeyOffset> . SeriesKey usually refers to a keyword or identifier used to uniquely identify time series data in a time series database or similar data structure. Each time series data will have a corresponding SeriesKey, which is similar to the primary key in the database and is used to distinguish different time series data. In a time series database, SeriesKey is usually a unique identifier that can be used to quickly locate and retrieve specific time series data. SeriesKeyOffset is the offset between the SeriesKey and the Key in the file. During the persistence process, storing SeriesKey in the file and recording their offsets in the file can effectively manage and access large amounts of time series data.

[0054] Step 14: Persist tagValue, persist all tagValues ​​under the same tagKey together and generate the corresponding hash index Hash Index and write it to the disk file. This process ensures the validity and fast accessibility of the index data organized by tag key value pairs.

[0055] Step 15: Persist all tagKeys to form a Tag Block.

[0056] TagKey and tagValue are usually used to describe metadata or additional information of time series data. TagKey: In a time series database, tagKey is a keyword or field used to identify or classify time series data, and is used to group or classify time series data. Each time series data point can be accompanied by one or more tagKeys to more accurately filter and retrieve data in data analysis and query.

[0057] Tag value tagValue: Associated with each tagKey, tagValue is the specific value or content of tagKey.

[0058] In a time series database, the purpose of using tagKey and tagValue is to organize, classify, and index large amounts of time series data through these metadata attributes, thereby supporting efficient data management and query operations.

[0059] Step 16: Persist the metadata information of the time series table to form a table block.

[0060] The inverted index has two core data structures in memory: seriesByID map<seriesID,seriesKey> ; seriesByTagKeyValue map <tagKey,<tagvalue,list <seriesid>>>.

[0061] Step 2: Use the inverted index to perform multi-dimensional condition filtering query:

[0062] Filter in table Block according to the time series table name and locate the file area where all dimension values ​​corresponding to the table Block are located.

[0063] Get the Hash Index of the tag key area corresponding to the time series table name, use the dimension name to perform hash operation, and locate the storage area of ​​the dimension value corresponding to the dimension name.

[0064] Get the Hash Index of the dimension value area corresponding to the dimension name, use the dimension value to perform hash operation, locate all SeriesIDs corresponding to the dimension value, and the SeriesID corresponds to the offset of the Series Key in the disk file.

[0065] Load the corresponding SeriesKey according to SeriesID,

[0066] Search the disk file for the corresponding result set that meets the query conditions based on SeriesKey, fieldKey, and time range.

[0067] Example 2

[0068] The present invention also provides a time series database query optimization device, including an index persistence module and a filtering query module.

[0069] The index persistence module persists the inverted index information to disk files:

[0070] Sort the series ByID and seriesByTagKeyValue data,

[0071] Persisting SeriesKey,

[0072] Persist tagValue,

[0073] Persist all tagKeys to form Tag Block,

[0074] Persist the metadata information of the time series table to form a table block;

[0075] The filter query module uses the inverted index to perform multi-dimensional conditional filter queries:

[0076] Filter in table Block according to the time series table name and locate the file area where all dimension values ​​corresponding to the table Block are located.

[0077] Get the Hash Index of the tag key area corresponding to the time series table name, use the dimension name to perform hash operation, and locate the storage area of ​​the dimension value corresponding to the dimension name.

[0078] Get the Hash Index of the dimension value area corresponding to the dimension name, use the dimension value to perform hash operation, locate all SeriesIDs corresponding to the dimension value, and the SeriesID corresponds to the offset of the Series Key in the disk file.

[0079] Load the corresponding SeriesKey according to SeriesID,

[0080] Search the disk file for the corresponding result set that meets the query conditions based on SeriesKey, fieldKey, and time range.

[0081] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0082] Similarly, the device of the present invention persists the inverted index information to a disk file, and uses the inverted index to perform multi-dimensional conditional filtering queries, thereby improving the query performance of multi-mode queries and reducing resource consumption.

[0083] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0084] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.< / seriesid>

Claims

1. A time series database query optimization method, characterized by include: Step 1: Persist the inverted index information to a disk file: Sort the series ByID and seriesByTagKeyValue data, Persisting SeriesKey, Persist tagValue, Persist all tagKeys to form Tag Block, Persist the metadata information of the time series table to form a table block; Step 2: Use the inverted index to perform multi-dimensional condition filtering query: Filter in table Block according to the time series table name and locate the file area where all dimension values ​​corresponding to the table Block are located. Get the Hash Index of the tag key area corresponding to the time series table name, use the dimension name to perform hash operation, and locate the storage area of ​​the dimension value corresponding to the dimension name. Get the HashIndex of the dimension value area corresponding to the dimension name, use the dimension value to perform hash operation, locate all SeriesIDs corresponding to the dimension value, and the SeriesID corresponds to the offset of the Series Key in the disk file. Load the corresponding SeriesKey according to SeriesID, Search the disk file for the corresponding result set that meets the query conditions based on SeriesKey, fieldKey, and time range.

2. According to the time series database query optimization method of claim 1, it is characterized by In step 1, the persistence thread is used to determine whether the accumulated inverted index information reaches a threshold or satisfies the conditions for executing the persistence operation. If yes, the inverted index information in the memory is written to the disk file.

3. According to the method for optimizing time series database query in claim 1, it is characterized in that when persisting SeriesKey in step 1, first construct Series Block, and record the corresponding relationship between SeriesKey and the offset of the Key in the disk file while constructing, and the corresponding relationship is expressed as<SeriesKey,SeriesKeyOffset> .

4. According to the time series database query optimization method described in claim 1, it is characterized in that when persisting the tagValue in step 1, all tagValues ​​under the same tagKey are persisted together and the corresponding Hash Index is generated and written to the disk file.

5. A time series database query optimization device, characterized in that Including index persistence module and filter query module, The index persistence module persists the inverted index information to disk files: Sort the series ByID and seriesByTagKeyValue data, Persisting SeriesKey, Persist tagValue, Persist all tagKeys to form Tag Block, Persist the metadata information of the time series table to form a table block; The filter query module uses the inverted index to perform multi-dimensional conditional filter queries: Filter in table Block according to the time series table name and locate the file area where all dimension values ​​corresponding to the table Block are located. Get the Hash Index of the tag key area corresponding to the time series table name, use the dimension name to perform hash operation, and locate the storage area of ​​the dimension value corresponding to the dimension name. Get the Hash Index of the dimension value area corresponding to the dimension name, use the dimension value to perform hash operation, locate all SeriesIDs corresponding to the dimension value, and the SeriesID corresponds to the offset of the Series Key in the disk file. Load the corresponding SeriesKey according to SeriesID, Search the disk file for the corresponding result set that meets the query conditions based on SeriesKey, fieldKey, and time range.

6. The time series database query optimization device according to claim 5, characterized in that The index persistence module uses the persistence thread to determine whether the accumulated inverted index information reaches a threshold or meets the conditions for executing the persistence operation. If so, the inverted index information in the memory is written to the disk file.

7. The time series database query optimization device according to claim 5, characterized in that When the index persistence module persists SeriesKey, it first builds the Series Block and records the corresponding relationship between the SeriesKey and the offset of the Key in the disk file. The corresponding relationship is expressed as<SeriesKey,SeriesKeyOffset> .

8. The time series database query optimization device according to claim 5, characterized in that When the index persistence module persists tagValue, it persists all tagValues ​​under the same tagKey together and generates the corresponding Hash Index and writes it to the disk file.

Images