Index construction method, data query method and related devices

Abstract

The application discloses an index construction method, a data query method and related equipment, and relates to the technical field of computers. The method comprises the following steps: acquiring a timeline composed of labels of time-series data, and assigning a timeline identifier to the timeline, wherein the labels comprise label keys and label values, and the timeline is composed of label values of different label keys; determining a set of timeline identifiers corresponding to the label values, so as to obtain an identifier set of the label values; and constructing an index structure for retrieving the timeline based on the label values, the timeline identifiers and the identifier set, wherein the index structure comprises a first index layer and a second index layer, so as to create a first mapping relationship between the label values and the identifier set through the first index layer, and create a second mapping relationship between the timeline identifiers and the timelines through the second index layer. The application solves the problem in the prior art that an index cannot efficiently support multi-dimensional retrieval of timelines.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an index construction method, a data query method, and related equipment. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. It should not be construed as an admission that the description herein is prior art.

[0003] With the advent of the Internet of Things (IoT) and 5G era, data from time-series scenarios such as IoT, application monitoring, and the Industrial Internet are experiencing explosive growth. This time-series data is typically stored using time-series databases. Generally, a single piece of time-series data is described by elements such as metrics, tags, timestamps, and fields. Metrics indicate where the data is stored, tags indicate who generated the data, timestamps indicate when the data was generated, and fields indicate the content of the data. Tags describe the characteristics of the data source and typically do not change over time. For example, for sensor devices, tags include device ID and the region where the device is located. The database automatically indexes tags, supporting multi-dimensional retrieval queries based on tags. Tags consist of a tag key (TagKey) and a tag value (TagValue), both of which are reference data types (String).

[0004] After time-series data is stored in a time-series database, it is typically necessary to retrieve a set of timelines that meet certain criteria based on different tag values. This allows for calculations such as downsampling and aggregation at the timeline granularity. To support retrieval by tag value, the industry practice is to build an inverted index for the timelines, enabling multi-dimensional retrieval based on tag values. Typically, different tag keys have different numbers of possible tag values. Some tag keys have fewer possible tag values, meaning that almost all timelines of a given time-series data contain that tag value. This indicates that the tag value has low distinguishability, often resulting in a very large inverted list of the corresponding tag in the inverted index. This leads to high I / O overhead (i.e., a percentage of input / output traffic) when reading the inverted list of that tag from disk, thus resulting in low retrieval performance.

[0005] Existing time-series databases in the industry have the following two drawbacks: 1. Indexes cannot efficiently support multi-dimensional retrieval of timelines, and the inverted list directly stores the raw values ​​of the timelines, resulting in high storage overhead. 2. They only provide a simple inverted index implementation without further optimizing the IO overhead of the inverted index retrieval based on the characteristic information of the timelines, making it difficult to improve the query efficiency of multi-dimensional timeline retrieval. Summary of the Invention

[0006] This application provides an index building method, a data query method, and related equipment to at least solve the problem in the prior art that indexes cannot efficiently support multidimensional retrieval of timelines.

[0007] According to one aspect of this application, an index construction method is also provided, comprising:

[0008] Obtain a timeline composed of labels for time series data, and assign a timeline identifier to the timeline, wherein the label includes a label key and a label value, and the timeline is composed of the label values ​​of different label keys;

[0009] Determine the set of timeline identifiers corresponding to the tag value to obtain the identifier set of the tag value;

[0010] Based on the tag value, the timeline identifier, and the identifier set, an index structure for retrieving the timeline is constructed. The index structure includes a first index layer and a second index layer, so as to create a first mapping relationship between the tag value and the identifier set through the first index layer, and to create a second mapping relationship between the timeline identifier and the timeline through the second index layer.

[0011] In some embodiments, the first index layer includes:

[0012] Inverted index;

[0013] The step of creating a first mapping relationship between the tag value and the identifier set through the first index layer includes:

[0014] In the inverted index, a corresponding inverted list is constructed for the tag value, and the timeline identifier is stored in the inverted list. The value of the inverted list is the identifier set.

[0015] In some embodiments, the second index layer includes:

[0016] Forward index;

[0017] The step of creating a second mapping relationship between the timeline identifier and the timeline through the second index layer includes:

[0018] Create a second mapping relationship between the timeline identifier and the timeline in the forward index.

[0019] In some embodiments, before constructing the corresponding inverted list for the tag value in the inverted index, the method further includes:

[0020] Determine whether the inverted index contains the inverted list corresponding to the tag value. If so, store the timeline identifier in the inverted list.

[0021] In some embodiments, before constructing the corresponding inverted list for the tag value in the inverted index, the method further includes:

[0022] Determine whether the inverted index contains the inverted list corresponding to the tag value. If not, construct the corresponding inverted list for the tag value in the inverted index and store the timeline identifier in the inverted list.

[0023] In some embodiments, the step of assigning timeline identifiers to the timeline includes:

[0024] The timeline identifier is assigned to the timeline using integer data.

[0025] The step of storing the timeline identifier in the inverted list includes:

[0026] The timeline identifier is stored in the inverted list using a bitmap array. The bitmap array has multiple storage bits, and the value of each storage bit is either zero or one. When storing the timeline identifier, the storage bit in the bitmap array whose ranking order matches the value of the timeline identifier is set to one, based on the numerical value of the timeline identifier.

[0027] According to another aspect of this application, a data query method is also provided, the method being implemented based on an index structure constructed using the aforementioned index construction method, the method comprising:

[0028] Receive pre-collected tag statistics information, wherein the tag statistics information collects the cardinality of different tag keys of time-series data tags, and the cardinality is the number of tag values ​​corresponding to the tag key;

[0029] Receive a given query condition, which is an array of different tag values ​​used to query a timeline. Determine the query order of the tag values ​​based on the cardinality, where the larger the cardinality of the tag value, the earlier the query order.

[0030] According to the query order, the identifier set is read sequentially from the first index layer of the index structure based on the tag value in the query conditions. During the reading process, before using the next tag value to read the identifier set from the first index layer, it is determined whether the first identifier set read based on the previous tag value exists or is an empty set. If the first identifier set does not exist or is an empty set, it is determined that there is no timeline that satisfies the query conditions, and the query is stopped.

[0031] In some embodiments, before reading the identifier set from the first index layer using the next tag value, after determining whether the first identifier set read based on the previous tag value exists or is an empty set, the method further includes:

[0032] Step 1: If the first set of identifiers exists and is not empty, proceed to the next step;

[0033] Step 2: Determine whether the query condition contains a next tag value. If so, read the identifier set from the first index layer based on the next tag value.

[0034] Step 3: Determine whether the second identifier set read based on the next tag value exists or is empty. If the second identifier set does not exist or is empty, then it is determined that there is no timeline that meets the query conditions, and the query is stopped.

[0035] In some embodiments, after determining whether the second identifier set read based on the next tag value exists or is an empty set, the method further includes:

[0036] If the second set of identifiers exists and is not empty, then the intersection between the first set of identifiers and the second set of identifiers is calculated and obtained;

[0037] Obtain a pre-set threshold for the number of identifiers, and determine whether the number of timeline identifiers in the intersection is less than or equal to the threshold for the number of identifiers. If so, based on the timeline identifiers in the intersection, read the timeline that satisfies the query condition from the second index layer of the index structure.

[0038] In some embodiments, after determining whether the second identifier set read based on the next tag value exists or is an empty set, the method further includes:

[0039] If the second set of identifiers exists and is not empty, then the intersection between the first set of identifiers and the second set of identifiers is calculated and obtained;

[0040] Obtain a pre-set threshold for the number of identifiers, and determine whether the number of timeline identifiers in the intersection is less than or equal to the threshold for the number of identifiers. If not, return to step two.

[0041] According to another aspect of this application, an index building apparatus is also provided, comprising:

[0042] A configuration module is used to obtain a timeline composed of labels of time series data and assign a timeline identifier to the timeline, wherein the label includes a label key and a label value, and the timeline is composed of the label values ​​of different label keys;

[0043] The determining module is used to determine the set of timeline identifiers corresponding to the tag value, so as to obtain the identifier set of the tag value;

[0044] A construction module is used to construct an index structure for retrieving the timeline based on the tag value, the timeline identifier, and the identifier set. The index structure includes a first index layer and a second index layer to create a first mapping relationship between the tag value and the identifier set through the first index layer, and to create a second mapping relationship between the timeline identifier and the timeline through the second index layer.

[0045] According to another aspect of this application, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the above-described method steps when executing the computer program.

[0046] According to another aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0047] According to another aspect of this application, a computer program product is also provided, the computer program product comprising a computer program that, when executed by a processor, implements the above-described method steps.

[0048] In this embodiment, after assigning a timeline identifier to each timeline, the timeline identifier corresponding to each tag value is determined. The identifier set consisting of the timeline identifiers of each tag value is stored in the first index layer of the constructed index structure, thus obtaining a first mapping relationship between tag values ​​and identifier sets. A second mapping relationship between timeline identifiers and timelines is also created using the second index layer. Therefore, the index structure constructed for timelines based on the first and second index layers can efficiently support multidimensional retrieval of timelines, facilitating further optimization of disk I / O overhead in the timeline index query process based on tag statistics, thereby improving the query efficiency of multidimensional timeline retrieval. Attached Figure Description

[0049] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 A flowchart of an index construction method provided in an embodiment of the present invention;

[0051] Figure 2 This is a diagram illustrating a forward index.

[0052] Figure 3 A flowchart of a data query method provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the structure of an index building device provided in an embodiment of the present invention.

[0054] The above figures include the following reference numerals:

[0055] 11. Configuration module; 12. Determine module; 13. Build module. Detailed Implementation

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0057] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0058] Existing technologies, when constructing inverted indexes for time-series data to support multi-dimensional retrieval based on tag values, often suffer from limitations. Different tag keys have varying numbers of possible tag values, with some tag keys having fewer values. This means that almost all timelines within a given time-series data contain the same tag value, resulting in low distinguishability. Consequently, the inverted list of the corresponding tag in the inverted index is typically very large, leading to high I / O overhead and low retrieval performance when reading the inverted list from disk. To address this, this invention proposes a data query method. This method, during actual retrieval, prioritizes reading the inverted list based on I / O overhead, proceeding from lowest to highest overhead. It reads the timelines using the corresponding tag values ​​and calculates their intersections. If the intersection size is less than a certain threshold or is empty (i.e., adjacent tag values ​​do not share a common timeline), the inverted index search process can be terminated early, avoiding the need to read the inverted lists of all tags in the query conditions. This approach effectively reduces disk I / O overhead during inverted index retrieval, thereby improving the efficiency of multi-dimensional time-series retrieval.

[0059] To implement the above data query method and thus efficiently support multidimensional retrieval of timelines, this invention first proposes an index construction method, which can be found in the following embodiments. Figure 1 The method includes the following steps:

[0060] Step S11: Obtain timelines composed of tags from time-series data and assign timeline identifiers to each timeline. Each tag includes a tag key and a tag value, and each timeline is composed of tag values ​​for different tag keys. For example, the tag data for time-series data in a monitoring scenario is shown in Table 1. This scenario's tag data contains three tag keys: cpu, host, and app. Host stands for host machine, cpu for Central Processing Unit, and app for Application. The cpu value has four possible values: 1, 2, 3, and 4, representing the CPU number on the host machine. The host value has three possible values: ip1, ip2, and ip3 (i.e., using IP addresses to distinguish different hosts; IP addresses are short for Internet Protocol Addresses). The app value only has one possible value: tsdb. Each tag combination uniquely identifies a timeline. Therefore, the tag data shown in Table 1 contains eight timelines. Each timeline is assigned a timeline identifier for differentiation.

[0061] Step S12: Determine the set of timeline identifiers corresponding to the tag value to obtain the identifier set of the tag value. As shown in Table 1, the tag value cpu=1 corresponds to three timelines: timeline one, timeline five, and timeline seven. The tag value cpu=1 corresponds to the timeline identifiers of the above three timelines, and the timeline identifiers of each timeline constitute the identifier set corresponding to the tag value.

[0062] Step S13: Based on tag values, timeline identifiers, and identifier sets, construct an index structure for retrieving timelines. This index structure includes a first index layer and a second index layer. The first index layer creates a first mapping relationship between tag values ​​and identifier sets, while the second index layer creates a second mapping relationship between timeline identifiers and timelines. When retrieving timelines based on tag values, the first mapping relationship is used to find the identifier set corresponding to the tag value. The identifier set is then read using the corresponding tag value, and the intersection of timelines is calculated. Specifically, the intersection of the identifier sets read from two adjacent tag values ​​is obtained. This intersection represents the timeline identifiers shared by the two adjacent tag values. During the intersection calculation, if the intersection size is less than a certain threshold or is empty (i.e., the two adjacent tag values ​​have no shared timelines), the search process in the first index layer can be exited early. Instead of reading all the identifier sets of the tags in the query conditions, the second index layer is used to filter and read the timelines corresponding to the timeline identifiers within the intersection based on the second mapping relationship, thus obtaining the timelines that meet the query conditions. This method effectively reduces disk I / O overhead, thereby improving the efficiency of multi-dimensional timeline retrieval.

[0063]

[0064] Table 1 Example of Tag Data

[0065] As can be seen, in this embodiment of the invention, after assigning a timeline identifier to each timeline, the timeline identifier corresponding to each tag value is determined. The identifier set consisting of the timeline identifiers of each tag value is stored in the first index layer of the constructed index structure, thus obtaining a first mapping relationship between tag values ​​and identifier sets. Furthermore, a second mapping relationship between timeline identifiers and timelines is created using the second index layer. Therefore, the index structure constructed for timelines based on the first and second index layers can efficiently support multidimensional retrieval of timelines, facilitating further optimization of disk I / O overhead in the timeline index query process based on tag statistical information, thereby improving the retrieval efficiency of multidimensional timeline retrieval.

[0066] In this embodiment of the invention, the first index layer includes an inverted index. Step S13, which creates a first mapping relationship between tag values ​​and identifier sets through the first index layer, includes: constructing a corresponding inverted list for each tag value in the inverted index, and storing timeline identifiers in the inverted list. The value of the inverted list is the identifier set. Thus, the inverted index supports querying the identifier set corresponding to a tag value based on the tag value, facilitating the calculation of intersections during data querying. Taking example data from an application monitoring scenario, the constructed inverted index is shown in Table 3. Generally, the more elements a list contains, the greater the IO overhead of storing and reading the identifier set within the inverted list. Table 3 shows that the inverted lists for tag values ​​cpu=3 and cpu=4 are the smallest, containing only one timeline identifier, resulting in the lowest IO overhead. The inverted list for tag app=tsdb is the largest, containing a total of 8 timeline identifiers, resulting in the highest IO overhead. Therefore, during subsequent data queries, the IO overhead of data queries can be reduced by avoiding reading the inverted list with the highest IO overhead, efficiently supporting multi-dimensional retrieval of timelines.

[0067] In this embodiment of the invention, the second index layer includes a forward index. Step S13, which involves creating a second mapping relationship between timeline identifiers and timelines using the second index layer, includes: creating a second mapping relationship between timeline identifiers and timelines in the forward index. In this embodiment, the second mapping relationship is a bidirectional mapping relationship between timeline identifiers and timelines, such as... Figure 2 As shown, this means that a forward index consists of two dictionary tables, which can support looking up the corresponding timeline identifier based on the timeline (i.e., Figure 2 The system supports both timeline IDs and timeline lookups based on timeline identifiers. The forward index has lower I / O overhead. By querying based on tag values ​​and finding the intersection of timelines, the system avoids reading the inverted index, which incurs high I / O overhead. After exiting the inverted index early, the data query method provided in this embodiment can use the forward index, which has lower I / O overhead, to read timelines that meet the query conditions. This effectively reduces the heavy I / O overhead of index queries and improves the efficiency of multi-dimensional timeline retrieval.

[0068] Therefore, this embodiment of the invention constructs an index structure for timelines based on inverted and forward indexes, efficiently supporting multi-dimensional timeline retrieval. Based on this index structure, when querying timelines that meet the query conditions, the inverted index search process can be exited early, without needing to read the inverted lists of all tags in the query conditions. Instead, after obtaining the intersection of two adjacent tag sets based on the read tag set, the timelines that meet the query conditions are read from the forward index, which has lower I / O overhead. Using the index structure provided by this embodiment of the invention for data querying can effectively reduce disk I / O overhead during inverted index retrieval, thereby improving the efficiency of multi-dimensional timeline retrieval.

[0069] In this embodiment of the invention, before constructing the corresponding inverted list for the tag value in the inverted index, the index construction method provided by the present invention further includes: determining whether the inverted index has an inverted list corresponding to the tag value; if so, storing the timeline identifier in the inverted list; if not, constructing the corresponding inverted list for the tag value in the inverted index and storing the timeline identifier in the inverted list, thereby completing the creation of the first mapping relationship between the tag value and the identifier set.

[0070] To reduce the storage space and IO overhead of the inverted index during read / write operations, step S11, which assigns timeline identifiers to timelines, includes: assigning timeline identifiers to the timelines using integer data. In this embodiment, the timeline identifier can be a timeline ID configured using integer data (ID is an identification number used to uniquely identify the corresponding timeline), as shown in Table 2. Table 2 shows the timeline IDs configured for each timeline. At this time, the inverted list of the inverted index stores the identifier set of each tag value, as shown in Table 3. For example, when the tag value of the tag key cpu is 1 (i.e., cpu = 1), the identifier set for cpu = 1 is {1, 5, 7}. In this embodiment, the step of storing the timeline identifiers in the inverted list includes: storing the timeline identifiers in the inverted list using a bitmap array. The bitmap array has multiple storage bits, and the value of each storage bit is zero or one. When storing timeline identifiers, according to the value of the timeline identifier, the storage bits in the bitmap array whose ranking order matches the value are set to one.

[0071]

[0072] Table 2. Timeline ID Configuration Diagram

[0073] If the inverted index for CPU=1 stores the identifier set {1, 5, 7}, then its storage state in the bitmap array is [1, 0, 0, 0, 1, 0, 1]. This means the first, fifth, and seventh bits from left to right in the bitmap array have values ​​of 1, and the rest are 0. Similarly, the inverted index for CPU=2 is {2, 6, 8}, so its storage state in the bitmap array is [0, 1, 0, 0, 0, 1, 0, 1]. This means the second, sixth, and eighth bits from left to right in the array have values ​​of 1, and the rest are 0. Since the bitmap array only takes values ​​of 0 and 1, it can be stored bit-wise instead of byte-wise. A byte has 8 bits, so the bitmap array saves 8 times more storage space than a byte array, effectively reducing the storage space of the inverted index and the I / O overhead during read / write operations.

[0074]

[0075] Table 3 shows the inverted list of the inverted index, which stores the set of identifiers for each tag value.

[0076] Therefore, the index structure built on the timeline using forward and inverted indexes in this embodiment of the invention can efficiently support multi-dimensional timeline retrieval, facilitating further optimization of disk I / O overhead during the timeline index query process based on tag statistics, thereby improving the query efficiency of multi-dimensional timeline retrieval. Secondly, using integer data to assign timeline identifiers and using a bitmap array to store the timeline identifiers in the inverted list of the inverted index effectively reduces the storage space of the inverted index and the I / O overhead during read and write operations.

[0077] In the second embodiment of the present invention, when the first index layer of the index structure provided in the first embodiment is an inverted index and the second index layer is a forward index, the construction process of an index structure is specifically described. Assuming a new timeline series = {cpu = 5, host = ip3, app = tsdb} is given, the construction process of the index structure is as follows:

[0078] First, assign a timeline ID (i.e., timeline identifier) ​​to the given timeline series, and create a bidirectional mapping relationship between timelines and timeline IDs in the forward index. As shown in Table 2, the original maximum value of the timeline ID in Table 2 was 8, and the timeline ID was generated in an auto-incrementing manner, so the new timeline ID is configured to be 9.

[0079] Then, the labels in the time series are expanded, and the new time series IDs are added to the inverted lists of each label. For example, in Table 2, the inverted list for cpu=5 does not exist, so an inverted list with the label cpu=5 is added to the inverted index, and the time series identifier stored in the inverted list for cpu=5 is {9}, containing only one time series identifier. After adding the new time series ID 9 to the inverted list for the label host=ip3, its inverted list changes from a set of identifiers with two time series identifiers {7, 8} to a set of identifiers with three elements {7, 8, 9}. After adding the new time series ID to the inverted list for the label app=tsdb, its inverted list changes from a set of identifiers with eight elements {1, 2, 3, 4, 5, 6, 7, 8} to a set of identifiers with nine elements {1, 2, 3, 4, 5, 6, 7, 8, 9}, and so on, completing the index structure used for querying time series data.

[0080] The third embodiment of the present invention also provides a data query method, which is implemented based on the index structure obtained by the index building method provided in the first embodiment of the present invention, such as... Figure 3 As shown, this data query method includes the following steps:

[0081] Step S21: Receive pre-collected tag statistics. The tag statistics include the cardinality of different tag keys for time-series data tags. The cardinality is the number of tag values ​​corresponding to a tag key. The pre-collected tag statistics in this embodiment are shown in Table 4. In Table 4, the cardinality for the tag key "cpu" is at most 4, indicating that with a fixed total number of timelines, the tag values ​​corresponding to the "cpu" tag key are more evenly distributed and have higher distinguishability. The length of the identifier set stored in the inverted list of its corresponding inverted index is smaller than the length of the identifier sets for other tag keys (such as "host" and "app"). This results in lower disk I / O overhead when reading the identifier set of tag values ​​under this tag key from the index structure. Therefore, this provides a basis for querying based on tag values ​​in order of increasing I / O overhead, optimizing the I / O overhead of timeline queries based on the cardinality—a characteristic information of the timeline—and thus improving the query efficiency of multi-dimensional timeline retrieval.

[0082] Step S22: Receive the given query conditions. The query conditions are an array of different tag values ​​used to query the timeline. The query order of the tag values ​​is determined based on the cardinality; the larger the cardinality of the tag value, the earlier it appears in the query order. For example, if the query conditions are {app=tsdb, host=ip3, cpu=1}, based on the tag statistics in Table 4, the query conditions are sorted from largest to smallest according to the cardinality corresponding to the tag key. In the example data, the cardinality of cpu is 4, the cardinality of host is 3, and the cardinality of app is 1. Therefore, the query order of the query conditions is adjusted to {cpu=1, host=ip3, app=tsdb}.

[0083] Step S23: Based on the tag values ​​in the query conditions, read the identifier set from the first index layer of the index structure in the query order. During the reading process, before using the next tag value to read the identifier set from the first index layer, determine whether the first identifier set read based on the previous tag value exists or is empty. If the first identifier set does not exist or is empty, it is determined that there is no timeline that meets the query conditions, and the query stops. Therefore, the process of searching for timeline identifiers in the first index layer is exited early, and there is no need to read the timeline from the second index layer of the index structure, effectively reducing the disk I / O overhead during the timeline query process. Here, the identifier set is the set of timeline identifiers corresponding to the tag values, the timeline identifier is a unique identifier configured to identify the timeline to be queried, and the index structure is constructed by the index construction method provided in the first embodiment of this invention.

[0084]

[0085] Table 4. Label Statistics Information

[0086] In step S23, before using the next tag value to read the identifier set from the first index layer, after determining whether the first identifier set read based on the previous tag value exists or is an empty set, the data query method provided in this embodiment of the invention further includes:

[0087] Step 1: If the first set of identifiers exists and is not empty, proceed to the next step (i.e., proceed to Step 2).

[0088] Step 2: Determine whether the query condition has a next tag value. If so, read the identifier set from the first index layer based on the next tag value.

[0089] Step 3: Determine whether the second identifier set read based on the next tag value exists or is empty. If the second identifier set does not exist or is empty, it is determined that there is no timeline that meets the query conditions, and the query is stopped.

[0090] In step three, after determining whether the second identifier set read based on the next tag value exists or is empty, the data query method provided in this embodiment further includes: if the second identifier set exists and is not empty, calculating and obtaining the intersection between the first identifier set and the second identifier set, obtaining a pre-set identifier count threshold, and determining whether the number of timeline identifiers in the intersection is less than or equal to the identifier count threshold. If so, based on the timeline identifiers in the intersection, reading the timeline that meets the query conditions from the second index layer of the index structure. That is, in the process of obtaining the intersection of timelines for each tag value in the query order from low IO overhead to high IO overhead, this embodiment can exit the process of searching for the identifier set in the first index layer in advance when the number of timeline identifiers in the intersection is less than or equal to the identifier count threshold. This avoids reading the identifier sets corresponding to all tag values ​​in the query conditions, effectively reducing disk IO overhead during the query process in the first index layer, thereby improving the multidimensional retrieval efficiency of timelines. Therefore, this paper proposes a timeline multidimensional retrieval method optimized using cardinality statistics, which reduces disk IO overhead during the index query process by utilizing cardinality statistics, thereby improving the query efficiency of timeline multidimensional retrieval.

[0091] Secondly, after step three determines whether the second identifier set read based on the next tag value exists or is an empty set, the data query method provided in this embodiment of the invention further includes: if the second identifier set exists and is not an empty set, then calculate and obtain the intersection between the first identifier set and the second identifier set, obtain a pre-set identifier number threshold, and determine whether the number of timeline identifiers in the intersection is less than or equal to the identifier number threshold. If not, return to step two, and repeat this process until it is determined whether there is a timeline that meets the query conditions. Then, after reading the timeline that meets the query conditions in the second index layer of the index structure, calculation tasks including aggregation, downsampling, and interpolation can be performed according to the timeline granularity.

[0092] This invention sorts tag values ​​in query conditions based on the cardinality of tag statistics, overcoming the uncertainty caused by the order of the query conditions themselves to the efficiency of timeline queries. Based on the sorted query conditions, the identifier set in the first index layer of the index structure is read. The query process in the first index layer is terminated early by checking if the number of timeline identifiers in the intersection of timelines obtained during the query meets the identifier count threshold. This avoids reading the identifier set with high I / O overhead, and instead uses the second index layer with low I / O overhead for post-filtering to read timelines that meet the query conditions. This effectively reduces the excessive I / O overhead of index queries, thereby significantly improving the efficiency of multi-dimensional timeline retrieval.

[0093] In the four embodiments of the present invention, taking the third embodiment of the present invention, where the first index layer is an inverted index and the second index layer is a forward index, as an example, a query process for time-series data in a monitoring scenario is introduced. In this monitoring scenario, the time-series data has a total of three tag keys: cpu, host, and app. Here, host stands for host, cpu is short for Central Processing Unit, and app is short for Application. The cpu value has four possible values: 1, 2, 3, and 4, representing the CPU number on the host. The host value has three possible values: ip1, ip2, and ip3 (i.e., using IP addresses to distinguish different hosts; IP address is short for Internet Protocol Address). The app value only has one possible value: tsdb.

[0094] First, this embodiment of the invention uses the probability cardinality estimator HyperLogLog (HyperLogLog is an algorithm used for cardinality statistics) to calculate the cardinality of tag values ​​associated with tag keys. Cardinality is the number of tag values ​​under a given tag key. The statistical information includes the following: For example, the tag key cpu has four tag values. The maximum cardinality value corresponding to the tag key cpu is 4, indicating that with a fixed total number of timelines, the tag values ​​corresponding to cpu are more evenly distributed and have higher distinguishability. The length of the inverted list set of its corresponding inverted index is smaller than the inverted index list of other tag keys (such as host, app).

[0095] This embodiment of the invention uses a given timeline of time series = {cpu = 5, host = ip3, app = tsdb} as an example to illustrate the statistical process of tag statistics. The tags in the timeline series are expanded, the hash value of the tag value under each tag is calculated, and the hash value is added to the corresponding probability cardinality estimator. Taking the example data, since only the tag cpu = 5 is a new tag value, after the tag statistics are updated, only the statistical value under the cpu tag key changes, from the original approximate value of 4 to the new approximate value of 5. The statistics for the tags host = ip3 and app = tsdb remain unchanged.

[0096] After obtaining the corresponding tag statistics of time-series data, this embodiment of the invention proposes a multi-dimensional retrieval method for time-series data based on statistical information optimization. The core idea of ​​this method is to use tag statistics combined with an inverted index to support multi-dimensional retrieval of timelines with minimal IO overhead. The main process is to sort the tag values ​​in the query conditions according to the cardinality of the tag keys counted in the tag statistics from largest to smallest, and then iterate through the query elements in the query conditions—tag values—reading the inverted list of the query conditions from the inverted index of the index structure and finding the intersection. The entire query process is as follows:

[0097] First, assuming the given query conditions are arrays TagFilter[] filters, the inverted index post-filter threshold is N (i.e., the threshold for the number of tags is N), and the post-filter flag isPostFilter (the post-filter flag is used to determine whether to read the timeline that meets the query conditions from the forward index), and the result of the index query is an inverted list result, then the query process includes the following steps:

[0098] 11. Assign an empty set to the inverted list of query results (result), and set isPostFilter to false to indicate whether post-filtering is needed.

[0099] 12. Sort the query order of each tag value in the query conditions according to the cardinality of the tag key in the tag statistics information from largest to smallest.

[0100] 13. Retrieve the first condition of the query (i.e., the tag value that is first in the query order), tag Key1 = tagValue1. Then, read the inverted list corresponding to this tag value from the inverted index. If the inverted list does not exist or is an empty set, proceed to step 17. Otherwise, assign the contents of the inverted list (the tag set) obtained in this query to result, and proceed to step 14.

[0101] 14. Determine if there is another element (tag value) in the query condition. If not, go to step 17. Otherwise, retrieve the query condition tagKeyi = tagValuei (the i-th tag value in the query order), read the inverted list postings from the inverted index. If postings does not exist or is empty, go to step 17. Otherwise, find the intersection of result and postings, assign the intersection result to result, and go to step 15.

[0102] 15. Check if result is empty. If it is empty, go to step 17. Otherwise, check if the size of the result collection is less than or equal to N. If not, go to step 14. Otherwise, set the post-filter flag isPostFilter to true and go to step 16.

[0103] 16. Determine if the post-filter flag isPostFilter is true. If not, proceed to step 17; otherwise, create a new inverted list collection newResult. Sequentially retrieve the timeline IDs of the results, and read the corresponding timeline series from the forward index based on the timeline ID. Determine if the timeline series contains all the tag values ​​in the query conditions. If so, add the timeline to newResult; otherwise, continue retrieving the timeline IDs from the results and filtering them in the forward index until all elements in the results are retrieved. Finally, assign newResult to the results and proceed to step 17.

[0104] 17. Return the timeline search result (result) to complete the data query.

[0105] As can be seen from the above, this embodiment of the invention uses inverted and forward indexes to construct an index for the timeline, and uses a probability cardinality estimator to approximately and efficiently estimate the cardinality of tag values ​​for tag keys under massive timelines. During the retrieval process, the cardinality calculated using tag statistics is used to reorder the tag values ​​in the query conditions, overcoming the uncertainty caused by the order of the tag values ​​themselves in the query conditions on the timeline query efficiency. Therefore, based on the sorted query conditions, the inverted list of tag values ​​in the inverted index is read, and the query process of the inverted index is terminated early based on whether the post-filtering conditions are met. This avoids the high IO overhead of reading the inverted list, instead using the low IO overhead of the forward index for post-filtering, effectively reducing the excessive IO overhead of index queries, thereby effectively improving the multi-dimensional retrieval efficiency of the timeline.

[0106] Furthermore, the index structure built on the timeline using forward and inverted indexes in this embodiment of the invention can efficiently support multi-dimensional timeline retrieval. Secondly, using integer data to assign IDs to the timeline and using a bitmap array to store the values ​​of the inverted list in the inverted index effectively reduces the storage space of the inverted index and the IO overhead during read / write operations.

[0107] This embodiment of the invention further illustrates the above query process using labeled data of time-series data in a monitoring scenario as an example. Assuming the given query conditions are {app=tsdb, host=ip3, cpu=1}, and the filtering threshold after the inverted index is 1 (the threshold for the number of identifiers is 1), the query process is as follows:

[0108] 21. Sort the query conditions in descending order of the cardinality corresponding to the tag keys. As shown in Table 4, the cardinality of cpu is 4, the cardinality of host is 3, and the cardinality of app is 1. Therefore, the query order of the conditions is adjusted to {cpu=1, host=ip3, app=tsdb}.

[0109] 22. Read the inverted list of cpu=1 from the inverted index of the index structure, and assign the contents of the inverted list of cpu=1 (identifier set) to result, i.e. result={1,5,7}.

[0110] 23. Read the inverted list of host=ip3, posts={7,8} from the inverted index, then find the intersection with result, and assign the intersection result to result={7}.

[0111] 24. Since the current query result result = {7} is less than or equal to the inverted index post-filter threshold of 1, there is no need to continue reading the inverted list of app = tsdb in the inverted index. Instead, proceed to step 25 and directly perform post-filtering on the search results based on the forward index.

[0112] 25. Read the timeline series with timeline ID 7 from the forward index, i.e.

[0113] The time series = {cpu = 1, host = ip3, app = tsdb} contains the query conditions, therefore the query conditions are met.

[0114] 26. Finally, the inverted index retrieval result is returned as result = {7}.

[0115] In this embodiment of the invention, cardinality statistics of tag values ​​are constructed for the tag keys of the timeline. During the retrieval process, the cardinality information is used to sort the query order of tag values ​​in the query conditions. At the same time, combined with the post-filtering threshold, it is determined whether it is necessary to exit the reading of the inverted list of the inverted index in advance. After exiting the reading of the inverted list in advance, the intermediate results of the inverted index retrieval results are filtered using the forward index with low IO overhead. This effectively avoids the IO overhead when reading extremely large inverted lists in most scenarios and improves the query efficiency of multi-dimensional timeline retrieval.

[0116] The fifth embodiment of the present invention also provides an index building apparatus, such as... Figure 4As shown, the index building device includes a configuration module 11, a determination module 12, and a building module 13. The configuration module 11 is used to acquire timelines composed of tags from time-series data and assign timeline identifiers to the timelines. Each tag includes a tag key and a tag value, and each timeline is composed of tag values ​​for different tag keys. The determination module 12 is used to determine the set of timeline identifiers corresponding to each tag value, thereby obtaining a set of identifiers for each tag value. The building module 13 is used to construct an index structure for retrieving timelines based on the tag values, timeline identifiers, and the identifier set. The index structure includes a first index layer and a second index layer, to create a first mapping relationship between tag values ​​and the identifier set through the first index layer, and to create a second mapping relationship between timeline identifiers and timelines through the second index layer.

[0117] The index building apparatus provided in this embodiment of the invention assigns a timeline identifier to each timeline through the configuration module 11, and then the determination module 12 determines the timeline identifier corresponding to each tag value. The identifier set consisting of the timeline identifiers of each tag value is stored in the first index layer of the index structure built by the building module 13, thus obtaining a first mapping relationship between tag values ​​and identifier sets. A second mapping relationship between timeline identifiers and timelines is also created using the second index layer. Therefore, the index structure built for timelines based on the first and second index layers can efficiently support multidimensional retrieval of timelines, and facilitates further optimization of disk I / O overhead in the timeline index query process based on tag statistics, thereby improving the query efficiency of multidimensional timeline retrieval.

[0118] In this embodiment of the invention, the first index layer of the index structure constructed by the construction module 13 includes an inverted index. The step of the construction module 13 creating a first mapping relationship between tag values ​​and identifier sets through the first index layer includes: constructing an inverted list corresponding to the tag value in the inverted index, and storing the timeline identifier in the inverted list; the value of the inverted list is the identifier set. This supports retrieving the identifier set corresponding to the tag value based on the tag value, facilitating the calculation of intersections during data querying. Taking example data from an application monitoring scenario, the constructed inverted index is shown in Table 3. Generally, the more elements the inverted list contains, the greater the IO overhead of storing and reading the identifier set within the corresponding inverted list. As can be seen from Table 3, the inverted lists for tag values ​​cpu=3 and cpu=4 are the smallest, containing only one timeline identifier, and thus have the lowest IO overhead. The inverted list for tag app=tsdb is the largest, containing a total of 8 timeline identifiers, and thus has the highest IO overhead. Therefore, during subsequent data queries, the IO overhead of data queries can be reduced by avoiding reading the inverted list with the highest IO overhead, efficiently supporting multi-dimensional retrieval of timelines.

[0119] In this embodiment of the invention, the second index layer of the index structure constructed by the construction module 13 includes a forward index. The step of the construction module 13 creating a second mapping relationship between timeline identifiers and timelines through the second index layer includes: creating a second mapping relationship between timeline identifiers and timelines in the forward index, the content of which is as follows: Figure 2 As shown. In this embodiment of the invention, the second mapping relationship is a bidirectional mapping relationship between timeline identifiers and timelines. That is, the forward index consists of two dictionary tables, which can support finding the corresponding timeline identifier based on the timeline (i.e.,...). Figure 2 The system supports both timeline IDs and timeline lookups based on timeline identifiers. The forward index has lower I / O overhead. By querying based on tag values ​​and finding the intersection of timelines, the system avoids reading the inverted index, which incurs high I / O overhead. After exiting the inverted index early, the data query method provided in this embodiment can use the forward index, which has lower I / O overhead, to read timelines that meet the query conditions. This effectively reduces the heavy I / O overhead of index queries and improves the efficiency of multi-dimensional timeline retrieval.

[0120] Therefore, the index building apparatus provided in this embodiment of the invention can construct an index structure for timelines based on inverted and forward indexes, which can efficiently support multidimensional retrieval of timelines. Based on this index structure, when querying timelines that meet the query conditions, the inverted index search process can be exited early, without reading the inverted list of all tag values ​​in the query conditions. Instead, after obtaining the intersection of two adjacent tag sets based on the read tag set, the timelines that meet the query conditions are read from the forward index, which has lower IO overhead, based on the timeline tags within the intersection. Using the index structure provided in this embodiment of the invention for data querying can effectively reduce disk IO overhead during inverted index retrieval, thereby improving the efficiency of multidimensional retrieval of timelines.

[0121] In this embodiment of the invention, before constructing the corresponding inverted list for the tag value in the inverted index, the construction module 13 determines whether the inverted index has an inverted list corresponding to the tag value. If so, the timeline identifier is stored in the inverted list. If not, the corresponding inverted list for the tag value is constructed in the inverted index, and the timeline identifier is stored in the inverted list, thus completing the creation of the first mapping relationship between the tag value and the identifier set.

[0122] To reduce the storage space and I / O overhead of the inverted index during read and write operations, the configuration module 11 assigns timeline identifiers to timelines using integer data. The timeline identifier can be a timeline ID (ID stands for Identity Document, used to uniquely identify the corresponding timeline) configured using integer data, as shown in Table 2. Table 2 shows the timeline IDs configured for each timeline using integer data. At this time, the inverted list of the inverted index stores the identifier set for each tag value, as shown in Table 3. For example, when the tag key `cpu` has a tag value of 1 (i.e., `cpu = 1`), the identifier set for `cpu = 1` is {1, 5, 7}. The construction module 13 stores the timeline identifiers in the inverted list using a bitmap array. The bitmap array has multiple storage bits, with values ​​of zero or one. When storing timeline identifiers, the storage bits in the bitmap array whose ranking order matches the value are set to one, based on the numerical value of the timeline identifier. Since bitmap arrays only take the values ​​0 and 1, they can be stored bit-by-bit rather than byte-by-byte. A byte has 8 bits, so bitmap arrays save 8 times the storage space compared to byte arrays, effectively reducing the storage space required for inverted indexes and the I / O overhead during read / write operations.

[0123] The sixth embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The feature is that when the processor executes the computer program, it implements the steps of a data query method. For details of the data query method, please refer to the content provided in the third embodiment of the present invention. The embodiments of the present invention will not be described again here.

[0124] The seventh embodiment of the present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of a data query method. For details of the data query method, please refer to the content provided in the third embodiment of the present invention. The embodiments of the present invention will not be described again here.

[0125] The eighth embodiment of the present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of a data query method. For details of the data query method, please refer to the content provided in the third embodiment of the present invention. The embodiments of the present invention will not be described again here.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0127] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An index construction method, characterized in that, include: Obtain a timeline composed of labels for time series data, and assign a timeline identifier to the timeline, wherein the label includes a label key and a label value, and the timeline is composed of the label values ​​of different label keys; Determine the set of timeline identifiers corresponding to the tag value to obtain the identifier set of the tag value; Based on the tag value, the timeline identifier, and the identifier set, an index structure for retrieving the timeline is constructed. The index structure includes a first index layer and a second index layer, so as to create a first mapping relationship between the tag value and the identifier set through the first index layer, and to create a second mapping relationship between the timeline identifier and the timeline through the second index layer. The first index layer includes an inverted index, the inverted list of which stores a set of identifiers for each tag value; a bitmap array is used to store the timeline identifiers in the inverted list, the bitmap array has multiple storage bits, the value of each storage bit is zero or one, when storing the timeline identifier, according to the value of the timeline identifier, the value of the storage bit in the bitmap array that is in the same order as the value is set to one.

2. The method according to claim 1, characterized in that, The steps of creating a first mapping relationship between the tag values ​​and the identifier set through the first index layer include: In the inverted index, a corresponding inverted list is constructed for the tag value, and the timeline identifier is stored in the inverted list. The value of the inverted list is the identifier set.

3. The method according to claim 1, characterized in that, The second index layer includes: Forward index; The step of creating a second mapping relationship between the timeline identifier and the timeline through the second index layer includes: Create a second mapping relationship between the timeline identifier and the timeline in the forward index.

4. The method according to claim 2, characterized in that, Before constructing the corresponding inverted list for the tag value in the inverted index, the method further includes: Determine whether the inverted index contains the inverted list corresponding to the tag value. If so, store the timeline identifier in the inverted list.

5. The method according to claim 2, characterized in that, Before constructing the corresponding inverted list for the tag value in the inverted index, the method further includes: Determine whether the inverted index contains the inverted list corresponding to the tag value. If not, construct the corresponding inverted list for the tag value in the inverted index and store the timeline identifier in the inverted list.

6. The method according to claim 2, characterized in that, The steps for configuring a timeline identifier to uniquely identify the timeline include: The timeline identifier is configured using integer data to uniquely identify the timeline.

7. A data query method, characterized in that, The method is implemented based on the index structure obtained by the index construction method according to any one of claims 1 to 6, and the method includes: Receive pre-collected tag statistics information, wherein the tag statistics information collects the cardinality of different tag keys of time-series data tags, and the cardinality is the number of tag values ​​corresponding to the tag key; Receive a given query condition, which is an array of different tag values ​​used to query a timeline. Determine the query order of the tag values ​​based on the cardinality, where the larger the cardinality of the tag value, the earlier the query order. According to the query order, the identifier set is read sequentially from the first index layer of the index structure based on the tag value in the query conditions. During the reading process, before using the next tag value to read the identifier set from the first index layer, it is determined whether the first identifier set read based on the previous tag value exists or is an empty set. If the first identifier set does not exist or is an empty set, it is determined that there is no timeline that satisfies the query conditions, and the query is stopped.

8. The method according to claim 7, characterized in that, Before using the next tag value to read the identifier set in the first index layer, after determining whether the first identifier set read based on the previous tag value exists or is an empty set, the method further includes: Step 1: If the first set of identifiers exists and is not empty, proceed to the next step; Step 2: Determine whether the query condition contains a next tag value. If so, read the identifier set from the first index layer based on the next tag value. Step 3: Determine whether the second identifier set read based on the next tag value exists or is empty. If the second identifier set does not exist or is empty, then it is determined that there is no timeline that meets the query conditions, and the query is stopped.

9. The method according to claim 8, characterized in that, After determining whether the second identifier set read based on the next tag value exists or is an empty set, the method further includes: If the second set of identifiers exists and is not empty, then the intersection between the first set of identifiers and the second set of identifiers is calculated and obtained; Obtain a pre-set threshold for the number of identifiers, and determine whether the number of timeline identifiers in the intersection is less than or equal to the threshold for the number of identifiers. If so, based on the timeline identifiers in the intersection, read the timeline that satisfies the query condition from the second index layer of the index structure.

10. The method according to claim 8, characterized in that, After determining whether the second identifier set read based on the next tag value exists or is an empty set, the method further includes: If the second set of identifiers exists and is not empty, then the intersection between the first set of identifiers and the second set of identifiers is calculated and obtained; Obtain a pre-set threshold for the number of identifiers, and determine whether the number of timeline identifiers in the intersection is less than or equal to the threshold for the number of identifiers. If not, return to step two.

11. An index construction apparatus, characterized in that, include: A configuration module is used to obtain a timeline composed of labels of time series data and assign a timeline identifier to the timeline, wherein the label includes a label key and a label value, and the timeline is composed of the label values ​​of different label keys; The determining module is used to determine the set of timeline identifiers corresponding to the tag value, so as to obtain the identifier set of the tag value; A construction module is used to construct an index structure for retrieving the timeline based on the tag values, the timeline identifiers, and the identifier set. The index structure includes a first index layer and a second index layer. The first index layer creates a first mapping relationship between the tag values ​​and the identifier set, and the second index layer creates a second mapping relationship between the timeline identifiers and the timeline. The first index layer includes an inverted index, and the inverted list of the inverted index stores the identifier set for each tag value. A bitmap array is used to store the timeline identifiers in the inverted list. The bitmap array has multiple storage bits, and the value of each storage bit is either zero or one. When storing the timeline identifier, the storage bit in the bitmap array whose ranking order matches the value of the timeline identifier is set to one, based on the numerical value of the timeline identifier.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 7 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 7 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 7 to 10.

Images