A method for index construction of a time series data storage engine

Abstract

The application provides an index construction method of a time series data storage engine, and belongs to the technical field of database storage. Specifically, the method comprises the following steps: pre-screening data blocks according to the document frequency of a label key and the occurrence frequency of a label value; extracting features from the pre-screening set through the historical access frequency of the label key, obtaining data features, and performing machine learning for further screening to obtain a target set comprising an index label group of each timeline; extracting target labels from the target set according to a plurality of different indexes in the index label group to obtain a plurality of group label sets; placing the group label sets with the same index label into corresponding timelines to obtain a plurality of timeline sets; assigning a unique group ID to each timeline set, establishing an inverted index of the mapping between the label key-value pair and the group ID, and establishing a prefix index of the mapping between the target label and the inverted index. The application can improve the writing efficiency and index construction efficiency of time series data.

Description

Technical Field

[0001] This invention relates to the field of database storage technology, and in particular to an index construction method for a time-series data storage engine. Background Technology

[0002] With the development of IoT technology, the number of IoT devices and their application scope have increased dramatically. To ensure the high availability and robustness of IoT devices and internet services, the need for more precise and comprehensive monitoring of their real-time operating status has emerged. Time-series databases, as the storage engine for such monitoring data, have received widespread attention from academia and industry in recent years in this context.

[0003] Typical time-series data generally consists of two parts: timeline data and point-in-time data. Point-in-time data typically consists of a 64-bit integer timestamp and a double-precision floating-point (IEEE 754 double) metric value. Timeline data, on the other hand, is more complex in its representation: it generally consists of a monitoring metric string and a series of tag key-value pairs, usually referred to as a timeline (serieskey).

[0004] Current methods for storing and indexing time-series data are based on tag values, timeline identifiers, and identifier sets. These methods construct a two-tiered index structure for retrieving timelines, storing the mapping between tag values ​​and identifier sets, as well as a second mapping between timeline identifiers and timelines. However, this approach cannot adapt to the rapid increase in the number of timelines in a time-series database. As the number of timelines grows, the amount of index construction increases rapidly, impacting both the writing efficiency of time-series data and the efficiency of index construction. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides an index construction method for a time-series data storage engine.

[0006] This invention provides an index construction method for a time-series data storage engine, comprising:

[0007] S1: Based on the document frequency of the tag key and the occurrence frequency of the tag value, pre-filter the data blocks to be stored to obtain a pre-filtered set;

[0008] S2: Extract features from the pre-screened set by using the historical access frequency of the tag keys to obtain data features;

[0009] S3: Perform machine learning on the data features to obtain a filtering function, and use the filtering function to filter the pre-filtered set to obtain a target set, wherein the target set includes at least the indicator label group for each timeline;

[0010] S4: Extract target labels from the target set based on multiple different indicators in the indicator label group to obtain multiple group label sets;

[0011] S5: For a set of group labels with the same indicator labels, insert the timeline corresponding to the indicator label to obtain multiple timeline sets;

[0012] S6: Assign a unique group ID to each timeline set, establish an inverted index mapping tag key-value pairs to group IDs, and establish a front index mapping target tags to the inverted index to complete the index construction of the time-series data storage engine.

[0013] According to the index construction method of a time-series data storage engine provided by the present invention, the data features in step S2 include: document frequency, tag key cardinality, tag key cardinality ranking, tag key cardinality ranking ratio, tag key frequency, tag key frequency ranking, and tag key frequency ranking ratio.

[0014] According to the index construction method of a time-series data storage engine provided by the present invention, the machine learning scheme used for the data features in step S3 is the AdaBoost iterative algorithm.

[0015] According to the index construction method of a time-series data storage engine provided by the present invention, in step S3, each indicator in the indicator tag group uses the indicator name as the tag key and the indicator value as the tag value.

[0016] According to the index construction method of a time-series data storage engine provided by the present invention, each tag key-value pair in the inverted index in step S6 has a corresponding inverted chain, and the inverted chain includes ascending group ID groups.

[0017] According to the index construction method of a time-series data storage engine provided by the present invention, the pre-index in step S6 is implemented by algebraic reconstruction method and dictionary data structure.

[0018] According to the index construction method of a time-series data storage engine provided by the present invention, in the pre-index in step S6, the algebraic reconstruction method is used to map the tag key to the address of the dictionary data structure composed of the corresponding tag value set.

[0019] According to the index construction method of a time-series data storage engine provided by the present invention, in the pre-index in step S6, the dictionary data structure stores the mapping between tag values ​​and corresponding inverted chain offsets.

[0020] This invention provides an index construction method for a time-series data storage engine. Based on heuristic filtering and machine learning, it selects target tags to prevent a rapid increase in index construction volume when facing the current problem of timeline expansion in time-series databases. This ensures efficient writing of time-series data and efficient index construction. Simultaneously, it uses the selected target tags for predictive grouping and establishes a complete index structure system based on this grouping scheme. This optimizes data storage and index construction volume without reducing query performance. Furthermore, this invention utilizes a suitable index structure to optimize for different query conditions and requirements, ensuring good performance even under varying data characteristics and query preferences.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an index construction method for a time-series data storage engine provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the structure of a timeline set constructed using the index construction method of a time-series data storage engine according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0026] In the description of the embodiments of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0027] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0028] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0029] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0030] To better understand this invention, the terms used in this invention will be explained below.

[0031] AdaBoost is an iterative algorithm whose core idea is to train different classifiers (weak classifiers) on the same training set, and then combine these weak classifiers to form a stronger final classifier (strong classifier). The algorithm itself achieves this by changing the data distribution. It determines the weight of each sample based on whether each sample in the training set is correctly classified in each iteration, and the overall classification accuracy of the previous iteration. The new dataset with modified weights is then fed into the next layer of classifiers for training. Finally, the classifiers obtained from each training iteration are merged to form the final decision classifier. Using the AdaBoost classifier can filter out some unnecessary training data features and allocate them to key training data.

[0032] An Adaptive Radix Tree (ART) is an efficient data structure used for storing and retrieving key-value pairs. It is a special type of radix tree designed to address some of the shortcomings of traditional radix trees in terms of memory utilization and performance.

[0033] The characteristics of adaptive radix trees include:

[0034] Efficient memory utilization: Compared to traditional radix trees, ART selects different node types based on the prefix length of the key, thus making better use of memory and reducing node space overhead. High-performance search and insertion operations: ART trees employ multiple node types (4, 16, 48, and 256 leaf nodes), dynamically selecting the node type based on the key length, enabling more efficient search and insertion operations in different scenarios. Suitable for high-concurrency environments: ART trees use optimistic concurrency control, achieving concurrent access to nodes through lock-free or fine-grained locking, making them suitable for high-concurrency read and write scenarios. Suitable for efficient range queries: ART trees support efficient range query operations, quickly locating a set of key-value pairs matching a specific prefix.

[0035] Adaptive radix trees have wide applications in databases, operating systems, network routing tables, and other fields. Their design philosophy is mainly to optimize memory utilization and improve search efficiency, making them an effective data structure for handling large-scale key-value pair storage and retrieval.

[0036] FST is a widely used data structure in natural language processing and speech recognition. In the database field, Lucene has incorporated it into Lucene Apache as an important indexing structure, replacing traditional data structures such as hash tables. This data structure is essentially a special type of Finite State Automaton (FSA), and in application, its structure resembles a weighted Trie that shares not only prefixes but also suffixes. Compared to other data structures, the time complexity of FST when used as a map is... of which, The length of the input sequence is consistent with the time complexity of a hash table, and it represents the minimum time complexity achievable for a single mapping in a data structure with mapping functionality. Therefore, its query speed should be faster than space-optimized hash tables such as maps constructed using red-black trees or HAMT (Hash Array Mapped Trie). Furthermore, compared to Trie, FST has a significantly higher compression ratio due to its shared string suffix feature. However, like dictionary data structures such as Trie, FST stores all information about the string set; that is, the string set it contains can be recovered by traversing the FST, a feature not found in data structures like hash tables. This means that FST can support imprecise matching patterns in the input data, such as regular expression matching, wildcard matching, and fuzzy matching.

[0037] The following is combined Figure 1 Description of embodiments of the present invention.

[0038] This invention provides an index construction method for a time-series data storage engine, comprising:

[0039] First, steps S1 to S3 below are all part of the target label selection stage. Target label selection mainly consists of two stages: a heuristic filtering stage and a machine learning filtering stage. In the heuristic filtering process, labels are initially filtered using predefined rules and corresponding thresholds to generate a pre-filtered set with a much smaller cardinality compared to the original label set (i.e., the set of all labels appearing in a data segment). During this process, data features of the labels contained in the pre-filtered set are extracted. Then, using these data features, historical query frequency is used as the label for machine learning filtering, ultimately generating the target labels.

[0040] S1: Based on the document frequency of the tag key and the occurrence frequency of the tag value, pre-filter the data blocks to be stored to obtain a pre-filtered set;

[0041] In the pre-screening stage, the filtering is mainly based on the document frequency of the tag key and the frequency of the tag value.

[0042] The document frequency of a tag key refers to how many timelines with different metrics a particular tag key exists in. For monitoring different metrics, some tag keys may overlap, and these overlapping tag keys constitute the basic data for the monitoring target. Tags with these overlapping tag keys should be the primary criteria for recall. A data segment typically contains around 102 tag keys, and the filtering threshold is generally set to 30%, meaning that after sorting the tag keys by document frequency, the top 30% of tag keys are selected.

[0043] The selection of tag values ​​primarily relies on their frequency of occurrence. After selecting the aforementioned set of tag keys, each tag key corresponds to approximately 10 to 3000 different tag values. After sorting the frequency of tag values ​​within the data segment, a threshold P is defined (typically 80% or 90%, referred to as the P80 or P90 scheme). Selection proceeds downwards from the highest probability tag values ​​until the sum of their frequencies reaches the threshold P. The selected tag values ​​are then combined with the aforementioned tag keys to form a pre-selected set of tag key-value pairs.

[0044] S2: Extract features from the pre-screened set by using the historical access frequency of the tag keys to obtain data features;

[0045] The tags for each data entry are generated from historical query records. Generally, some tag keys in the historical query records have a much higher access frequency than other tag keys. In this invention, the Boolean tags of these tag keys are set to 1, while the remaining tag keys are set to 0. Specifically, similar to the tag value filtering method in step S1, after sorting the access frequency of different tags in the historical query records, the P80 or P90 scheme is used for filtering, and the selected tag keys are... Set the value.

[0046] The data features in step S2 include: document frequency, tag key cardinality, tag key cardinality ranking, tag key cardinality ranking ratio, tag key frequency, tag key frequency ranking, and tag key frequency ranking ratio.

[0047] This stage uses the data features extracted in the previous stage to train the machine learning model. The extracted features include:

[0048] Tag key cardinality The cardinality of the set of tag values ​​that a tag key possesses;

[0049] Tag key cardinality ranking The above base is the ranking of all tag keys in the timeline set recording the same indicator;

[0050] Tag key cardinality ranking ratio The above ranking represents the percentage of all tag keys in the timeline set for the same metric.

[0051] Tag key frequency The number of times a tag key appears in a timeline set recording the same metric;

[0052] Tag key frequency ranking The above frequency is the ranking of all tag keys in the timeline set recording the same indicator;

[0053] Tag key frequency ranking ratio The above ranking represents the percentage of all tag keys in the timeline set for the same metric.

[0054] Document frequency Document frequency of tag keys.

[0055] S3: Perform machine learning on the data features to obtain a filtering function, and use the filtering function to filter the pre-filtered set to obtain a target set, wherein the target set includes at least the indicator label group for each timeline;

[0056] In step S3, the machine learning scheme used for the data features is the AdaBoost iterative algorithm.

[0057] In step S3, each indicator in the indicator label group uses the indicator name as the label key and the indicator value as the label value.

[0058] Following step S2, a more accurate function than heuristic screening is trained using classic machine learning methods such as AdaBoost. This function yields the final target label. This invention also treats metrics in the timeline data as a set of labels, with the label key being "metric" and the value being the metric value, such as "metric: cpu". This special set of labels is always the target label; therefore, each group's group labels contain one and only one such label, and all timelines within the group have the same metric.

[0059] S4: Extract target labels from the target set based on multiple different indicators in the indicator label group to obtain multiple group label sets;

[0060] Furthermore, based on the target tag filtering results, the target tags contained in each timeline are extracted to form a set, and then placed into a group. Each group has some group tags to form a group tag set. The group tag set is different between different groups and is also the only way to distinguish between different groups.

[0061] S5: For a set of group labels with the same indicator labels, insert the timeline corresponding to the indicator label to obtain multiple timeline sets;

[0062] Furthermore, after obtaining the target label set of a timeline, the timeline is placed into a group of labels that are exactly equivalent to the target label set of the timeline. If no group meets the requirements, a new group that meets the above requirements is created, and the timeline is placed into it.

[0063] This invention divides all timelines within a data segment into different groups, and the distribution structure of the data within each group is as follows: Figure 2 As shown, the timeline group includes indicator 1, indicator 2, indicator 3 and indicator g. Each indicator has a corresponding label 1 to label kg belonging to the indicator itself. Each group has a unique set of group labels, and all timelines in the group also contain all the labels in the above set of group labels. The timelines in the group do not need to store this part of the labels independently, but only need to store the remaining labels they have, as well as their indicator names.

[0064] S6: Assign a unique group ID to each timeline set, establish an inverted index mapping tag key-value pairs to group IDs, and establish a front index mapping target tags to the inverted index to complete the index construction of the time-series data storage engine.

[0065] Furthermore, based on the grouping scheme in step S5 and above, a front index and an inverted index are established for the timeline data. The front index indexes the target label to the inverted chain, and the inverted index indexes the front index result to the location containing the timeline to be indexed.

[0066] In step S6, each tag key-value pair in the inverted index has a corresponding inverted chain, and the inverted chain includes ascending group ID groups.

[0067] In step S6, the pre-index is implemented using algebraic reconstruction and dictionary data structures.

[0068] In step S6, the algebraic reconstruction method is used to map the tag keys to the dictionary data structure address composed of the corresponding tag value set in the pre-index.

[0069] In step S6, the dictionary data structure stores a mapping between label values ​​and corresponding inverted chain offsets in the pre-index.

[0070] Based on the grouping scheme, this invention designs a group-level inverted index. For each group in a data segment, after assigning it a unique group ID, a mapping from tag key-value pairs to group IDs is established. This mapping is the group-level inverted index in this invention. Specifically, each tag key-value pair corresponds to an inverted chain, which contains a series of group IDs in ascending order. The group tag set of the groups corresponding to these IDs contains the tag key-value pair.

[0071] Based on the grouping scheme, this invention designs a pre-index, which maps the target label to the starting position of its corresponding inverted chain. This part of the index will be implemented using a combination of ART algebraic reconstruction method and FST dictionary data structure.

[0072] In this context, ART is used to map the tag key to the address of the FST consisting of its corresponding tag value set. For each different tag key in the target tag set, it corresponds to a set of tag values. The combination of the tag key and any tag value in the set has appeared at least once in the data segment. For this tag value set, the present invention establishes an FST to store it.

[0073] In this invention, an FST stores the mapping from the tag value to its corresponding inverted list offset. The offset is a positive integer based on the base address of the inverted list, and addition operations can be naturally defined between positive integers, so it can be used as the output of the FST. On the other hand, storing the tag value in the form of an FST can well handle queries based on regular expression conditions.

[0074] Unlike the pruning search scheme proposed by Lucene, thanks to the time-order segmentation mechanism, the amount of timeline data contained in each data segment is not large. Furthermore, the grouping scheme further reduces the size of the index built for each data segment, resulting in each FST having only approximately [amount missing]. Orders of magnitude of storage space usage.

[0075] The following describes a method for recalling data stored based on an index construction method for a time-series data engine provided by the present invention, in some embodiments.

[0076] A typical query pattern unit for time-series databases is:

[0077] Query = indicator name + {tag key1: tag value, ..., tag key} : tag value};

[0078] The tag values ​​can be exact strings, regular expressions, or strings with wildcards. The tag key-value pairs are ANDed, meaning the goal is to retrieve timeline data containing all tags from the query tag set. The number of tags... The number typically ranges from 2 to 6.

[0079] When a query arrives, the ART is first queried. For each query condition, if the condition matches a target tag key, it can be mapped to the corresponding FST. Next, the corresponding tag value is queried in the FST. If the tag value is the target tag value, one (or more, depending on regular expression matching, etc.) return value is obtained; this return value is the offset of the inverted index. Next, using the offset obtained above, the inverted index is queried to obtain a series of inverted chains. The intersection of these inverted chains yields all the group numbers required for this query. Finally, the corresponding groups are decompressed sequentially according to their group numbers, and all timeline data within each group is returned, thus completing the processing of this query.

[0080] One possible scenario is that a query condition fails to match any target tag key, requiring filtering for this query. For other query conditions, after obtaining all timeline data for all groups to be recalled, this data needs to be filtered once. The filter condition is the timeline data containing query conditions that did not match the target tag key. This filtered data is then returned. Note that a query always provides one metric name as one of its query conditions. This ensures that no query exists where all its provided query conditions fail to match any target tag. Therefore, this scheme guarantees that a situation will never arise where global filtering at the data segment level is necessary, thus ensuring the lower bound of query latency.

[0081] Another possible scenario is that a query condition hits a target tag key (denoted as ) in the first step. The system successfully obtained the corresponding FST data structure, but no tag value was matched in the FST. This situation also requires filtering, but the amount of data and time overhead for filtering will be much smaller than the previous method. Similarly, after obtaining all timeline data from all groups requiring recall, for each timeline, only one check is needed. The system checks whether the corresponding tag value matches the specified tag value. After filtering, the result returned is the final result of this query.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an index for a time-series data storage engine, characterized in that, include: S1: Based on the document frequency of the tag key and the occurrence frequency of the tag value, pre-filter the data blocks to be stored to obtain a pre-filtered set; S2: Extract features from the pre-screened set by using the historical access frequency of the tag keys to obtain data features; S3: Perform machine learning on the data features to obtain a filtering function, and use the filtering function to filter the pre-filtered set to obtain a target set, wherein the target set includes at least the indicator label group for each timeline; S4: Extract target labels from the target set based on multiple different indicators in the indicator label group to obtain multiple group label sets; S5: For a set of group labels with the same indicator labels, insert the timeline corresponding to the indicator label to obtain multiple timeline sets; S6: Assign a unique group ID to each timeline set, establish an inverted index mapping tag key-value pairs to group IDs, and establish a front index mapping target tags to the inverted index to complete the index construction of the time-series data storage engine.

2. The index construction method for a time-series data storage engine according to claim 1, characterized in that, The data features in step S2 include: document frequency, tag key cardinality, tag key cardinality ranking, tag key cardinality ranking ratio, tag key frequency, tag key frequency ranking, and tag key frequency ranking ratio.

3. The index construction method for a time-series data storage engine according to claim 1, characterized in that, The machine learning scheme used for the data features in step S3 is the AdaBoost iterative algorithm.

4. The index construction method for a time-series data storage engine according to claim 1, characterized in that, In step S3, each indicator in the indicator label group uses the indicator name as the label key and the indicator value as the label value.

5. The index construction method for a time-series data storage engine according to claim 1, characterized in that, In step S6, each tag key-value pair in the inverted index has a corresponding inverted chain, which includes ascending group ID groups.

6. The index construction method for a time-series data storage engine according to claim 1, characterized in that, The preceding index in step S6 is implemented using algebraic reconstruction and dictionary data structures.

7. The index construction method for a time-series data storage engine according to claim 6, characterized in that, In the preceding index in step S6, the algebraic reconstruction method is used to map the tag keys to the dictionary data structure address composed of the corresponding tag value set.

8. The index construction method for a time-series data storage engine according to claim 6, characterized in that, In the preceding index in step S6, the dictionary data structure stores a mapping between label values ​​and corresponding inverted chain offsets.

Images