Inverted index structure oriented to time series data model and improved InfluxDB

By designing an inverted index structure for timing data models in InfluxDB, the inefficient analysis of InfluxDB when processing time series data and log text is solved, and more powerful text search capabilities and more efficient query performance are achieved.

CN119003686APending Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202411123283.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

InfluxDB lacks effective text search functions when processing time series data and log text mix load, resulting in inefficient analysis. Existing solutions such as storing data separately or using full-text search engines have problems such as high maintenance costs and mismatch in performance.

Method used

A reverse index structure for time series data model is designed, including first-level inverted index and second-level inverted index. The prefix tree word list design is used to filter the timeline according to keywords and maintain the collection of text data entries. Mapping, reduce unnecessary timeline iterator builds, and build inverted indexes within TSM files to reduce update complexity caused by file merging.

Benefits of technology

It improves text search capabilities, reduces the scale of query data, reduces storage and memory overhead, improves query efficiency, realizes O(1) complexity text data filtering, and significantly improves InfluxDB's text search performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003686A_ABST
    Figure CN119003686A_ABST
Patent Text Reader

Abstract

The invention discloses an inverted index structure oriented to a time sequence data model and an improved InfluxDB, relates to the technical field of databases, and solves the technical problem that the analysis efficiency of a time sequence database cannot reach expectation in a complex storage scene containing a large amount of text data. The method comprises a first-level inverted index and a second-level inverted index, the first-level inverted index and an original TSI index in the InfluxDB are located on the same level and used for screening a timeline according to keywords, and the second-level inverted index is located in each TSM file generated by the InfluxDB and used for maintaining mapping from the keywords to a text data entry set; according to the method, timeline screening and time range screening are supported, the data scale accessed in the query process is reduced, and the text search capability of the time sequence database is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to an inverted index structure oriented to a time series data model and an improved InfluxDB. Background Art

[0002] In the era of big data, time series data, as an important form of data, is widely used in various monitoring systems, the Internet of Things, financial data analysis and other fields. InfluxDB excels in time series data processing with its efficient writing and query performance. However, InfluxDB lacks text data search capabilities, so when processing mixed loads of time series data and log text, other solutions have to be adopted to improve analysis efficiency, such as storing time series data and log text separately, or writing time series data into a full-text search engine together. The above solutions face a series of problems in practical applications. Storing time series data and log text in different systems will introduce additional maintenance costs and data redundancy. Storing time series data and log text together in a full-text search engine may lead to reduced operational efficiency due to mismatches in data structure and performance. Specifically:

[0003] 1. As a kind of time series data, log text has gone through the development process from text file storage to traditional relational database storage, and then to time series database storage. Relational databases have efficient storage capabilities for relational data with well-defined schemas and support transactional reading and updating. However, log text often contains unstructured or semi-structured information, and the attributes are dynamic and high-dimensional, and usually do not need to be updated. In terms of storage space, the storage of log text requires a lot of disk space, and traditional relational databases often use a row storage structure, which is not conducive to log compression and results in large storage space overhead. Time series databases have significant advantages in processing large amounts of time series data, such as efficient data writing and reading capabilities. However, faced with complex storage scenarios containing large amounts of text data, the analysis efficiency of time series databases cannot meet expectations.

[0004] 2. The log storage system built based on the full-text search engine, represented by the ELK stack, has become a widely adopted solution. It is based on the inverted index structure and provides powerful full-text search functions, including full-text search, aggregation, fuzzy search, etc. However, the inverted index structure it adopts faces the dimensionality curse problem when facing high-cardinality data such as indicators, values ​​in logs, and timestamps. In the full-text search engine, the inverted index has the ability to quickly search and accurately match text, but when indexing large-scale text data, there are problems such as large space occupation, high memory overhead, and unfriendly to numerical types. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention provides an inverted index structure for a time series data model and an improved InfluxDB, which solves the technical problem that the analysis efficiency of the time series database cannot reach the expected level in the face of complex storage scenarios containing a large amount of text data.

[0006] An inverted index structure for a time series data model includes a primary inverted index and a secondary inverted index. The primary inverted index is located at the same level as the original TSI index in InfluxDB and is used to filter timelines according to keywords. The secondary inverted index is located in each TSM file generated by InfluxDB and is used to maintain a mapping from keywords to a set of text data entries. When a time series data entry is written, the text data is parsed and the index content is updated. When the time series data in the cache is written to the disk, the generated inverted index is written into the disk data file together with other time series data content.

[0007] Furthermore, both the first-level inverted index and the second-level inverted index adopt the design of prefix tree word list. The first-level inverted index is used to maintain the conversion from keywords to timeline sets. This design supplements the original filtering capability of InfluxDB for timelines with the function of filtering by keywords to reduce the construction of unnecessary timeline iterators; the second-level inverted index is used to maintain the conversion from keywords to text data entry sets.

[0008] Furthermore, the first-level inverted index uses the same bitmap-based timeline set representation as TSI, which is conducive to quickly performing intersection operations on the timeline set during the query process to obtain the timeline set that meets all query conditions.

[0009] Furthermore, the secondary inverted index includes a bucket list, which is used to manage a set of text data entries corresponding to keywords. Any data bucket in the bucket list is used to orderly store text data entry identifiers from the same timeline and the same field. The timelines and fields of the text data entry identifiers stored in any two data buckets are different, and the text data entry identifier is a timestamp corresponding to the text data entry.

[0010] Since data in the same timeline in InfluxDB has different timestamps, the timestamp of the text data itself can be used as the identifier of the text data in the data bucket. In the data bucket, it is stored in order by timestamp identifier to improve the efficiency of algorithms such as query and merge.

[0011] Furthermore, the bucket list implementation includes: lock-free jump list, linked list, hash table, and balanced tree.

[0012] Furthermore, the data bucket is identified by a combination of a timeline ID and a field name, wherein the timeline ID is an integer type data of a fixed byte size. Compared with a character string describing a timeline, the timeline ID has a faster comparison speed and requires less storage space.

[0013] An improved InfluxDB uses an inverted index structure for time series data models. In the process of filtering logical blocks in TSM files, the secondary inverted index can be used to obtain the data bucket corresponding to the timeline field with the target keyword. Since the timestamps corresponding to the text data entries in the data bucket are stored in order, the time range of the data in the data bucket can be directly obtained, which is the time range of the target text data in the current TSM file. This time range will provide more accurate judgment conditions for filtering TSM files and filtering logical blocks in files.

[0014] Furthermore, in the process of traversing the text data entries in the logic block to filter out the entries with the target keyword, since the data in the data bucket and the logic block are stored in chronological order, it is only necessary to use a cursor to point to the minimum or maximum timestamp in the data bucket that has not been matched in real time, and compare the timestamp pointed by the cursor with the timestamp corresponding to the text data entry read by the iterator in the logic block. If they are the same, they pass the screening. Therefore, for each piece of data in the logic block, it only takes O(1) time complexity to complete whether it meets the query condition, that is, whether the text data has the target keyword.

[0015] The beneficial effects of the present invention include:

[0016] The present invention designs an inverted index structure TSII (Time-series Inverted Index) for time series data models to support the search of text fields. It consists of two levels of inverted indexes to minimize the size of data required to be accessed during the query process and provide a more powerful text search capability for the time series database. Specifically:

[0017] 1. Support timeline filtering and time range filtering to reduce the size of data accessed during the query process;

[0018] 2. Build an inverted index for the text data in the file based on the TSM file to prevent large-scale updates to the inverted index structure due to TSM file merging;

[0019] 3. Reduce the information required to identify text data to reduce the memory and disk space required to store the inverted index and reduce the I / O overhead required to query the inverted index. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1It is an inverted index structure for time series data model and its positional relationship with other components in the InfluxDB system.

[0021] Figure 2 The inverted index structure in the TSM file of this application is designed.

[0022] Figure 3 For this application, the inverted index in the TSM file is used to assist in the screening of time series data.

[0023] Figure 4 This is a comparison of the writing performance of the present application and the prior art.

[0024] Figure 5 This is a comparison of keyword query performance between this application and the prior art.

[0025] Figure 6 This is a comparison of the numerical query performance of this application and the prior art. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0027] The relationship between an inverted index structure (TSII structure) for time series data models and other components in InfluxDB, such as Figure 1 As shown, it includes a first-level inverted index TSII(1) and a second-level inverted index TSII(2). The first-level inverted index TSII(1) is located at the same level as the original TSI index in InfluxDB, and is used to filter the timelines involved in the InfluxDB query process according to keywords. The second-level inverted index TSII(2) is located in each TSM file and is read or written into the cache of InfluxDB together with the time series data to maintain the mapping from keywords to text data entries. When the time series data entry is written, the system parses the text data and updates the index content. When the time series data in the cache is written to the disk, the generated inverted index is written to the disk data file together with other time series data content.

[0028] Specifically, when new data is written, the following process will be performed:

[0029] 1. If the new data involves the creation of a new timeline, it is necessary to parse the text data, update the first-level inverted index TSII (1), and maintain the mapping relationship between keywords and timeline sets.

[0030] 2. Parse the text data, map the keywords in the new data to their corresponding log entries, and write them to the secondary inverted index TSII (2) in the cache.

[0031] 3. When the amount of data in the cache reaches the threshold, the flush operation is triggered. At this time, the time series data in the cache and the corresponding secondary inverted index TSII (2) are written to the file storage together to form a TSM file.

[0032] In another embodiment, both the first-level inverted index and the second-level inverted index adopt the design of prefix tree word list, and the first-level inverted index is used to maintain the conversion from keywords to timeline sets. This design supplements the function of filtering by keywords on top of the original filtering capability of InfluxDB for timelines to reduce the construction of unnecessary timeline iterators; the second-level inverted index is used to maintain the conversion from keywords to text data entry sets.

[0033] The original TSI (Time Series Index) in InfluxDB is an inverted index with a different purpose from the TSII (1) proposed in this invention. The inverted index provides a mapping from different measurements, different tag keys, and tag values ​​to a timeline set, and is used to assist in the screening of relevant timelines in query requests.

[0034] The text data in different timelines contains different word sets. In existing research on log text compression, researchers have proposed that compared with natural language, log text has obvious "types" because the structure of log text is determined by computer programs, and the points where logs are output in the program are limited. Therefore, the words contained in the log text are also limited. In InfluxDB, different timelines are often responsible for tracking different objects, and the word sets contained in the log text output by different objects may also be different. By taking advantage of this difference, the accuracy of InfluxDB in timeline screening can be further improved.

[0035] The difference between TSII(1) and TSI indexes is that TSI indexes are responsible for maintaining the conversion from measurement and tag information to timeline collections, while TSII(1) is responsible for maintaining the conversion from keywords to timeline collections. This difference affects the design of the index structure. TSI uses a three-level map, while TSII(1) uses a prefix tree vocabulary. This design adds the ability to filter by keyword to the original timeline filtering capability of InfluxDB to reduce the construction of unnecessary timeline iterators.

[0036] In another embodiment, the first-level inverted index TSII(1) uses the same bitmap-based timeline set representation as TSI, which is beneficial for quickly performing intersection operations on the timeline set during the query process to obtain a timeline set that meets all query conditions.

[0037] In another embodiment, the secondary inverted index includes a bucket list, which is used to manage a set of text data entries corresponding to keywords. Any data bucket in the bucket list is used to orderly store text data entry identifiers from the same timeline and the same field. The time lines and fields of the text data entry identifiers stored in any two data buckets are different, and the text data entry identifier is a timestamp corresponding to the text data entry.

[0038] Since data in the same timeline in InfluxDB has different timestamps, the timestamp of the text data itself can be used as the identifier of the text data in the data bucket. In the data bucket, it is stored in order by timestamp identifier to improve the efficiency of algorithms such as query and merge.

[0039] In order to avoid the inverted index from becoming invalid due to the merging of TSM files, it is necessary to build the inverted index inside the TSM file. At the same time, according to the process analysis, each underlying iterator in the InfluxDB query process is in units of timelines, which means that each query for the inverted index will also be in units of timelines. However, from a design perspective, it is not necessary to create a separate inverted index for each timeline in the TSM file. On the one hand, doing so will result in the same words being stored repeatedly in different timeline indexes, causing unnecessary waste of space; on the other hand, building an inverted index for each timeline will cause the index structure to be dispersed, which is not conducive to algorithm implementation and introduces additional complexity in the merging of TSM files. Therefore, an inverted index structure is maintained in the cache, and the inverted index is stored on disk together with the time series data in the cache, so as to achieve a one-to-one correspondence with the TSM file.

[0040] In another embodiment, the bucket list is implemented by: a lock-free skip list, a linked list, a hash table, and a balanced tree. Specifically, in this embodiment, a lock-free skip list is used.

[0041] In another embodiment, the data bucket is identified by a combination of a timeline ID and a field name, wherein the timeline ID is an unsigned integer type with a length of 8 bytes. Compared with a character string describing the timeline, the timeline ID has a faster comparison speed and requires less storage space.

[0042] Considering the identification method of each data bucket in the bucket list, if SeriesKey (a string describing the timeline) + FieldKey (field name) is used directly to identify each data bucket, it may result in a large overhead. This is because SeriesKey itself is composed of measurement (measurement identification name) and tag (a combination of tag name and tag value), and there may be multiple tags, and their length can usually reach tens of bytes. Directly using them as the identification of data buckets will cause certain data redundancy problems. In addition, if SeriesKey is used directly for marking, it will cause certain problems for locating the target data bucket, because in this process it will inevitably be necessary to compare the identifications of different data buckets, and the comparison of long strings is relatively time-consuming.

[0043] In view of the above considerations, the timeline ID is used instead of the SeriesKey. The timeline ID itself is an unsigned integer type with a fixed length of 8 bytes. Compared with the string, it has a faster comparison speed and requires less storage space. The overall design structure of the specific secondary inverted index is as follows: Figure 2 shown.

[0044] In another embodiment, an improved InfluxDB is involved, which applies an inverted index structure for time series data model. In the process of selecting logical blocks in TSM files, the data bucket corresponding to the timeline field with the target keyword can be obtained by using the secondary inverted index. Since the timestamps corresponding to the text data entries in the data bucket are stored in order, the time range of the data in the data bucket can be directly obtained, which is the time range of the target text data in the current TSM file. This time range will provide more accurate judgment conditions for the selection of TSM files and the selection of logical blocks in the file.

[0045] In another embodiment, in the process of traversing the text data entries in the logic block to filter the entries with the target keyword, since the data in the data bucket and the logic block are stored in chronological order, it is only necessary to use a cursor to point to the minimum or maximum timestamp in the data bucket that has not been matched in real time, and compare the timestamp pointed by the cursor with the timestamp corresponding to the text data entry read by the iterator in the logic block. If they are the same, they pass the screening. Therefore, for each piece of data in the logic block, it only takes O(1) time complexity to complete whether it meets the query condition, that is, whether the text data has the target keyword.

[0046] Specific as Figure 3 As shown: After the previous processing, it can be known that the timestamp range of the data entries containing the keyword "network" is ts4 to ts7, and only one data block is filtered out here. Next, it is necessary to traverse the data entries on this data block to check which entries contain the keyword "network". The timestamps corresponding to all data entries containing the keyword have been recorded in sequence in the bucket, here are ts4 and ts7. At the beginning, the cursor in the bucket points to the smallest timestamp ts4, and the iterator starts traversing from the first data entry in the logical block, comparing its timestamp with the timestamp ts4 pointed to by the cursor in the bucket. If they are equal, it means that the keyword "network" is contained, and this data entry is output. When the timestamp corresponding to the data entry is found to be ts4, the cursor moves back to ts7, and the purpose of the iterator in the subsequent traversal process is changed to finding the data entry with the timestamp ts7.

[0047] This approach uses indexes to convert time-consuming string matching operations within logic blocks into fast timestamp comparisons.

[0048] The improved InfluxDB involved in the present invention is experimentally compared with the existing InfluxDB, and the specific contents are as follows:

[0049] Experiment and analysis

[0050] 1. Experimental environment

[0051] The hardware and software environment for system implementation and testing is shown in Table 1.

[0052] Table 1 Experimental environment configuration

[0053]

[0054] 2. Read and write performance test

[0055] The read and write performance of the improved InfluxDB, namely the M-InfluxDB system involved in this application, under different workloads was tested, and the original InfluxDB and the full-text search engine ElasticSearch were compared.

[0056] 2.1 Generation and division of test data

[0057] First, consider the generation principles of time series data and log text in the real world. It can be found that time series data is generally collected by various sensors or monitoring programs at fixed intervals, while log text is generated by specific code logic during program operation. Compared with time series data, the appearance of log text is relatively random and irregular, which means that under the mixed load of time series data and log text, some data entries have missing log text.

[0058] Seven groups of data were selected for testing (as shown in Table 2), where each group of time series data contains 0%, 10%, 20%, 30%, 50%, 70% and 90% of log text respectively. Group 1 is a special group that does not contain log text. The number of data entries in each group of data is 1×10 7 strip.

[0059] Table 2 Test data groups

[0060] Test Data Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Log text ratio 0% 10% 20% 30% 50% 70% 90%

[0061] 2.2 Write Performance Comparison

[0062] In the write performance test, we constructed write methods for InfluxDB and Elasticsearch respectively, and used the same test data set for testing. For the data written to Elasticsearch, each time series data was converted into a document.

[0063] In terms of writing performance, the performance of writing seven sets of test data to InfluxDB, M-InfluxDB, and Elasticsearch is compared. Figure 4 shown.

[0064] When facing a purely numerical time series data write load, M-InfluxDB's write performance is basically consistent with InfluxDB, but as the proportion of text data in the test data increases, M-InfluxDB's write throughput shows a clear downward trend. This is because M-InfluxDB needs to perform word segmentation operations on the written text data, and then pass the word segmentation results to the inverted index for updating. These operations become more frequent as the proportion of text data in the test data set increases. In addition, the write operation of the secondary inverted index in the TSM file also introduces some complexity to the write process of M-InfluxDB. This will cause M-InfluxDB's write performance to decrease slightly when facing text data.

[0065] For write loads where the proportion of text data is not high, such as group 2 and group 3 data, M-InfluxDB can maintain performance similar to InfluxDB. The write throughput of M-InfluxDB is better than Elasticsearch.

[0066] 2.3 Query Performance Comparison

[0067] In terms of query performance measurement, it is divided into queries with and without keyword query conditions, which are used to evaluate InfluxDB, M-InfluxDB and Elasticsearch.

[0068] The test results are as follows Figure 5 As shown in the figure, since group 1 does not contain text data, no test is done here.

[0069] From the test data, we can see that the text search capability of M-InfluxDB is significantly higher than that of InfluxDB. According to the comprehensive analysis of the performance of various log text ratios in the test data, the keyword query performance of M-InfluxDB, that is, the search speed of log text, is 13 to 15.6 times that of the original system InfluxDB. This is because when InfluxDB performs text search, it needs to use regular expressions for matching, and its time complexity is O(n), where n is the length of the text; while M-InfluxDB only needs O(1) time complexity with the assistance of inverted index. Compared with Elasticsearch, M-InfluxDB still has certain advantages in query performance, because Elasticsearch needs to do more processing when returning query results.

[0070] Finally, the query performance of each system on numeric columns is compared, such as Figure 6 As shown in the figure. The test results show that for numeric queries, the query performance of InfluxDB and M-InfluxDB is basically the same. This is because the query process does not involve matching keywords, and therefore does not access the inverted index structure. M-InfluxDB can take advantage of the read and write performance brought by the InfluxDB storage structure to achieve a higher query throughput, which is significantly better than the performance of Elasticsearch.

[0071] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. An inverted index structure for a time series data model, characterized in that: It includes a primary inverted index and a secondary inverted index. The primary inverted index is located at the same level as the original TSI index in InfluxDB and is used to filter timelines based on keywords. The secondary inverted index is located in each TSM file generated by InfluxDB and is used to maintain a mapping from keywords to a set of text data entries.

2. According to claim 1, the inverted index structure for a time series data model is characterized in that: The first-level inverted index and the second-level inverted index both adopt the design of prefix tree word list.

3. According to the inverted index structure for a time series data model as described in claim 1, it is characterized in that: The first-level inverted index uses the same bitmap-based timeline collection representation as TSI.

4. According to the inverted index structure for a time series data model as described in claim 1, it is characterized in that: The secondary inverted index includes a bucket list, which is used to manage a set of text data entries corresponding to keywords. Any data bucket in the bucket list is used to orderly store text data entry identifiers from the same timeline and the same field. The timelines and fields of the text data entry identifiers stored in any two data buckets are different. The text data entry identifier is a timestamp corresponding to the text data entry.

5. According to claim 4, the inverted index structure for a time series data model is characterized in that: The implementation methods of the bucket list include: lock-free jump list, linked list, hash table, and balanced tree.

6. According to claim 4, an inverted index structure for a time series data model is characterized in that: The data bucket is identified by a combination of a timeline ID and a field name, wherein the timeline ID is integer type data of a fixed byte size.

7. An improved InfluxDB, characterized in that: An inverted index structure for a time series data model as described in any one of claims 1 to 6 is applied. In the logical block screening process in the TSM file, the data bucket corresponding to the timeline field with the target keyword can be obtained by using the secondary inverted index. Since the timestamps corresponding to the text data entries in the data bucket are stored in order, the time range of the data in the data bucket can be directly obtained, and this time range is the time range where the target text data in the current TSM file is located.

8. An improved InfluxDB according to claim 7, characterized in that: In the process of traversing the text data entries in the logical block to filter out the entries with the target keywords, since the data in the data bucket and the logical block are stored in chronological order, it is only necessary to use a cursor to point to the minimum or maximum timestamp that has not been matched in the data bucket in real time, and compare the timestamp pointed by the cursor with the timestamp corresponding to the text data entry read by the iterator in the logical block. If they are the same, they pass the screening.

Citation Information

Patent Citations

  • Byte position based reverse index method for network message data

    CN105243150A

  • Time sequence data storage method based on dynamic weight equalization time sequence database cluster

    CN114416737A

  • Timing sequence database read performance optimization method and system based on multi-level index

    CN115374240A

  • OpenGemini-based full-text indexing method and system

    CN117827829A

  • Using an inverted index to produce an answer to a query

    US20130097126A1

Cited By

  • Method and system for integrating inverted indexes in database and medium

    CN119621877A