Multi-model learning index construction method and system for time series database

Abstract

The application discloses a multi-model learning index construction method and system for a time series database, and belongs to the technical field of time series databases. In order to solve the problem that existing index schemes are not specifically optimized and developed for time series databases, and therefore the index ability for time series databases is limited, the application adopts an LSM tree storage structure to store data of the time series database; in the process of constructing a learning index, a constant model is adopted to establish the learning index for RLE encoded timestamp data, a segmented model is adopted to establish the learning index for integer type data whose ZigZag algorithm encoded value does not exceed (1 << 60)-1, Packed encoded timestamp data and a floating point number, a linear model is adopted to establish the learning index for integer type data whose ZigZag algorithm encoded value is greater than (1 << 60)-1, RAW encoded timestamp and a string.

Description

Technical Field

[0001] This invention belongs to the field of time series database technology, specifically relating to a method and system for constructing a multi-model learning index for time series databases. Background Technology

[0002] Currently, time series databases are still a relatively new field, and there is no research on learning indexes for time series databases. Existing research on learning indexes is based on traditional relational databases.

[0003] Chaohong Ma et al. proposed the FILM learned index, a fully learned index for larger-than-memory databases. FILM is a tree-like structure that uses a simple approximation model to index data across different storage devices. Compared to existing techniques for larger-than-memory databases (such as decaching), FILM allows for more efficient query processing with significantly reduced main memory overhead. FILM also aims to effectively address one of the bottlenecks in existing methods for indexing larger-than-memory databases: data exchange between memory and disk. More specifically, the LRU (Least Recently Used) structure used in existing cold data identification methods (which determines data to be evicted to disk when available memory is exhausted) often leads to significant latency in query processing. FILM takes a radical approach, proposing an adaptive LRU structure and incorporating its updates into query processing with minimal overhead.

[0004] Liu Li et al. proposed a highly scalable data-aware learning indexing scheme called EWALI. It builds its index model based on a lightweight data-aware data partitioning algorithm. When data distribution changes, EWALI can automatically split relevant leaf nodes and retrain the corresponding model to adapt to different workloads. Furthermore, EWALI designs an alternative double buffer to handle new data and employs a delayed update mechanism to merge data, significantly reducing write locking and improving write performance.

[0005] Zhang Zhou et al. proposed PLIN, which contributes in four aspects: an NVM-aware data placement strategy, locally unordered and globally ordered leaf nodes, a model replication mechanism, and a hierarchical insertion strategy. Furthermore, PLIN is proposed for NVM-only architectures, supporting immediate recovery. The paper also proposes optimistic concurrency control and fine-grained locking mechanisms, enabling PLIN to scale to concurrent requests.

[0006] Although time series databases have many characteristics that make learning indexes more reasonable and applicable, existing indexing schemes have not been specifically optimized for time series databases, thus limiting their indexing capabilities. Summary of the Invention

[0007] The purpose of this invention is to address the problem that existing indexing schemes are not specifically optimized for time series databases, thus limiting their indexing capabilities for time series databases.

[0008] A method for constructing a multi-model learning index for a time series database is provided. The method includes a process of encoding the data and a process of constructing a learning index. The data is from a time series database, and the data storage structure of the time series database is an LSM tree.

[0009] The process of encoding the data includes the following steps:

[0010] For integer data, the ZigZag algorithm is used for encoding;

[0011] For timestamp data, first perform differential encoding on the timestamp data to obtain a difference array. If all values ​​in the difference array are the same, then RLE encoding is used for all values ​​in the difference array. Otherwise, continue to check if the maximum value in the difference array is greater than (1<<60)-1. If it is greater, then RAW encoding is used for all values ​​in the difference array; if it is not greater, then Packed encoding is used for all values ​​in the difference array.

[0012] The process of constructing the learning index includes the following steps:

[0013] A constant model is used to build a learning index for RLE-encoded timestamp data;

[0014] For integer data whose values ​​after ZigZag algorithm encoding do not exceed (1<<60)-1, a segmented model is used to build a learning index for timestamp data and floating-point numbers encoded with Packed encoding.

[0015] For integer data whose values ​​are greater than (1<<60)-1 after ZigZag algorithm encoding, as well as timestamps and strings encoded in RAW, a linear model is used to build a learning index.

[0016] Furthermore, for integer data, if the value encoded by the ZigZag algorithm does not exceed (1<<60)-1, the encoded value is compressed using the simple8b algorithm; if it exceeds this value, no compression is performed.

[0017] The Gorilla compression algorithm is used to compress floating-point numbers;

[0018] For string data, the Snappy compression algorithm is used for compression.

[0019] Furthermore, for integer data that is not compressed, additional storage space is still needed as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

[0020] Furthermore, the timestamp data encoded in RAW is uncompressed and still requires additional storage space as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

[0021] Furthermore, when storing the data corresponding to the learning index built using the segmented model, the interval data of each segment is saved; when searching for data, binary search is used to determine the interval in which the data falls and to return the value stored in the corresponding interval. That is, binary search is performed on the segment points of the segmented model for integer data, floating-point data, or packed timestamp data. Each segment stores one value. When the interval in which the data falls is found, the value stored in the corresponding interval is returned.

[0022] A multi-model learning indexing system for time series databases is provided, comprising a data encoding unit and a learning indexing unit; the data is from a time series database, and the data storage structure of the time series database is an LSM tree.

[0023] The encoding unit encodes the data, and the encoding process includes the following steps:

[0024] For integer data, the ZigZag algorithm is used for encoding;

[0025] For timestamp data, first perform differential encoding on the timestamp data to obtain a difference array. If all values ​​in the difference array are the same, then RLE encoding is used for all values ​​in the difference array. Otherwise, continue to check if the maximum value in the difference array is greater than (1<<60)-1. If it is greater, then RAW encoding is used for all values ​​in the difference array; if it is not greater, then Packed encoding is used for all values ​​in the difference array.

[0026] The learning index unit is used to provide an index for the queried data. The index is a learning index, and the process of building the learning index includes the following steps:

[0027] A constant model is used to build a learning index for RLE-encoded timestamp data;

[0028] For integer data whose values ​​after ZigZag algorithm encoding do not exceed (1<<60)-1, a segmented model is used to build a learning index for timestamp data and floating-point numbers encoded with Packed encoding.

[0029] For integer data whose values ​​are greater than (1<<60)-1 after ZigZag algorithm encoding, as well as timestamps and strings encoded in RAW, a linear model is used to build a learning index.

[0030] Furthermore, for integer data, if the value encoded by the ZigZag algorithm does not exceed (1<<60)-1, the encoded value is compressed using the simple8b algorithm; if it exceeds this value, no compression is performed.

[0031] The Gorilla compression algorithm is used to compress floating-point numbers;

[0032] For string data, the Snappy compression algorithm is used for compression.

[0033] Furthermore, for integer data that is not compressed, additional storage space is still needed as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

[0034] Furthermore, the timestamp data encoded in RAW is uncompressed and still requires additional storage space as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

[0035] Furthermore, when storing the data corresponding to the learning index built using the segmented model, the interval data of each segment is saved; when searching for data, binary search is used to determine the interval in which the data falls and to return the value stored in the corresponding interval. That is, binary search is performed on the segment points of the segmented model for integer data, floating-point data, or packed timestamp data. Each segment stores one value. When the interval in which the data falls is found, the value stored in the corresponding interval is returned.

[0036] Beneficial effects:

[0037] This invention creates a learning index based on the LSM tree storage structure of time-series databases, fully leveraging the characteristics of learning indexes. Furthermore, after research, this invention creates a learning index based on the management methods of time-series databases, enabling comprehensive and effective management of time-series databases and significantly improving their indexing capabilities. In addition, this invention creates three different models specifically for learning data distributions based on the different data distribution rules of time-series databases. This combines the characteristics of time-series database data storage with model learning, which will greatly improve the accuracy of the models. Attached Figure Description

[0038] Figure 1 Flowchart for compressing timestamp type data.

[0039] Figure 2 This is the offset distribution curve for segmented data.

[0040] Figure 3 This is a general data offset distribution curve. Detailed Implementation Specific implementation method one:

[0042] This implementation method is a multi-model learning index construction method for time series databases.

[0043] Based on a thorough study and analysis of the characteristics of time series databases, the proposed invention has the following four features:

[0044] 1. Regarding data storage structure, time-series databases essentially use an LSM tree structure. Therefore, the location of the data on disk is often constant, only changing during compaction. Newly inserted data does not affect the location of existing data without compaction, which is highly beneficial for learning index models. Traditional databases, on the other hand, use a B+Tree storage structure. Each data insertion can cause the leaf nodes of the B+Tree to split. This means that each insertion leads to a change in data location, resulting in more frequent retraining of the learning index model. Therefore, model retraining in the context of time-series databases is more lightweight and less frequent, while also contributing to the stability of the learned index.

[0045] 2. Regarding data characteristics, a key feature of time-series data is that it does not modify historical data, thus avoiding the need to retrain the corresponding learning index model. This is precisely a problem that traditional database learning indexes cannot avoid. In the context of time-series databases, the workload consists only of appending and querying, which also reduces the pressure of model retraining. Therefore, this characteristic also affects the performance of the learning index. This invention considers that time-series databases do not require modification of historical data, so there will be no learning index failure caused by modification of historical data, which also benefits the stability of the learning index.

[0046] 3. In terms of data management, data on each timeline is managed by a series, each with a unique series key. Data from different series are not related. For storage, data from different series is stored in different blocks on disk, with each block storing only one series. Therefore, a learning index can be created for the data corresponding to each series. This ensures that changes in data from other series will not affect the current series, and consequently, the learning index. It also means that the size of the learning index model can be controlled to no more than one block of data, ensuring that all data corresponding to the model has the same block ID. Furthermore, data within the same series will better conform to the patterns of the recorded events, making the model easier to fit.

[0047] 4. Regarding data distribution, time-series databases organize data according to time, dividing it into different shards for management. This ensures that the data on a single shard is not too large, allowing us to use a relatively smaller and more precise model to implement a learning index. Traditional databases, on the other hand, manage data uniformly, and while manual partitioning based on different attributes is possible, it lacks flexibility and standardization, and is not as well supported as time-series databases. This characteristic belongs to a higher dimension of characteristic 3, manifested in the final creation of a learning index for a single series on a single shard.

[0048] Then, based on this, a multi-model learning index is implemented using InfluxDB. InfluxDB uses different compression algorithms and stores data based on different data types. Therefore, this invention will use three different models to implement a multi-model learning index for time series databases, based on different data distribution rules for different data types.

[0049] First, let's explain the data distribution rules of InfluxDB:

[0050] A. For integer (int) data, the ZigZag algorithm is first used for encoding. If the encoded value does not exceed (1<<60)-1, the simple8b algorithm is used to compress the encoded data. If it exceeds this value, no compression is performed, but additional storage space is still needed as a flag to indicate which compression algorithm was used, even if no compression was performed. Here, 1<<60 is a formula for computer data calculation, which is 1152921504606846975 in decimal.

[0051] The ZigZag algorithm is relatively simple in principle and works well for relatively small numbers. Its main idea is to move the first sign bit of the integer data to the end of the binary number and remove any leading zeros, thus achieving compression.

[0052] The simple8b algorithm is used to pack multiple data points into a single 64-bit integer. It stores multiple integers within a single 64-bit integer. The simple8b algorithm's selector occupies 4 bits, which are mapped to a value Bits according to index rules. This value is then represented by each of the next 60 bits, with each Bits representing a numerical value.

[0053] B. Timestamp data exhibits certain patterns, therefore InfluxDB treats it as an independent data type with different storage rules. First, the timestamp data is differentially encoded to obtain a difference array. Then, different algorithms are applied based on the different results of the difference array. The flowchart is as follows: Figure 1 As shown, if all values ​​in the difference array are the same, then RLE encoding is applied to all values ​​in the difference array; otherwise, it continues to check if the maximum value in the difference array is greater than (1 << 60) - 1. If it is greater, then RAW encoding is applied to all values ​​in the difference array; if it is not greater, then Packed encoding is applied to all values ​​in the difference array. RLE encoding only stores a portion of the constants, Packed encoding uses the simple8b compression algorithm, and RAW encoding does not perform compression storage, but it also requires additional storage space as a flag to indicate which compression algorithm was used (even if no compression was performed).

[0054] C. The floating-point number compression method directly adopts the Gorilla compression algorithm proposed by Facebook. The Gorilla compression algorithm mainly uses the XOR concept. Therefore, this invention uses a segmented model for learning floating-point numbers.

[0055] D. String data directly uses the Snappy compression algorithm released by Google.

[0056] A constant model, as the name suggests, uses a single, identical value to store and replace all other values. In time series scenarios, monitoring is a common application. We cannot monitor the status of the required metrics in real time, as this would consume enormous resources. Therefore, a point-based approach is used, retrieving the corresponding monitoring metrics at a specified frequency. In this scenario, the generated timestamp data will be a strict arithmetic sequence (a type of timestamp). According to timestamp data compression and storage rules, it will use RLE encoding. The core idea of ​​this algorithm is to record the number of repetitions of the data. Since all values ​​in the difference array are the same, it is necessary to save the first value of the original array (the first timestamp) and record the differences in the difference array. Therefore, the distribution of offsets on the storage medium for this data will be constant. In this case, a constant model is chosen for the data, which stores only one constant value. Queries on the data offsets of all data within this model return this constant value.

[0057] The segmented model is essentially a piecewise function. In time series scenarios, timestamp data often becomes more regular due to human factors, but the metrics monitored by edge devices are real-world factors and often do not have strict arithmetic progression characteristics, so a segmented model is needed (timestamp data uses a constant model in the above context, and a segmented model in other contexts). For general integer and floating-point data, there are not many patterns to be found, so a segmented model is also necessary. Therefore, the simple8b compression algorithm is more commonly used for integer data, while the Gorilla compression algorithm is more commonly used for floating-point data. Since the principle of the simple8b compression algorithm is to pack multiple integers into a 64-bit integer, to query the data within this 64-bit integer, one must start from the Selector at the beginning of the 64-bit integer to determine how many bits represent one integer. Furthermore, the Gorilla compression algorithm for floating-point numbers essentially uses the XOR concept. It first stores the first floating-point number, and then the subsequent binary bits are related to the specific XOR value. Therefore, to query any data, the first floating-point number must be known. Therefore, for integer data using the simple8b compression algorithm, packed-encoded timestamp data, and floating-point data using the Gorilla compression algorithm, their offset distribution on the storage medium is as follows: Figure 2As shown, in this case, the present invention selects a segmented model for the data, which is actually a piecewise function. For data retrieval, we store the interval data of each segment. During the query, binary search is used to determine the interval in which the data falls and returns the value stored in the corresponding interval. That is, binary search is performed on the segmentation points of the segmented model for integer data, floating-point data, or packed timestamp data. Each segment stores one value, so the value is found and returned.

[0058] Linear models cannot accurately complete regression tasks; they can only provide a rough approximation, resembling a linear function. After establishing models for integer and floating-point data, the remaining challenges are integer data (too large for compression), timestamp data (RAW-encoded timestamps) too large for compression, and string data. This invention uses linear models to learn from these. Because string lengths are uncertain, it's difficult to discern patterns in the offset distribution of compressed string data on the storage medium. However, since data is appended to the storage medium, the overall offset tends to increase. Therefore, we use a linear regression model to fit the true data distribution, with the following results: Figure 3 As shown. In this case, the present invention selects a linear model for the data, which is a linear regression model obtained by learning and training on a set of data. For data lookup, the index is passed into the model, the model is used to make predictions, and the estimated value of the model is obtained and returned.

[0059] That is: a learning index is built using a constant model for RLE-encoded timestamp data;

[0060] For integer data that is compressed by the simple8b algorithm after encoding, i.e. integer data that does not exceed (1<<60)-1, a segmented model is used to build a learning index for timestamp data and floating-point numbers that are packed and encoded.

[0061] For integer data greater than (1 << 60) - 1, and for timestamps and strings encoded in RAW, a linear model is used to build a learning index.

[0062] Due to the characteristics of LSM trees, the location of data on disk often remains unchanged, only changing during compaction. Newly inserted data does not affect the location of existing data without compaction, which is highly beneficial for learning index models. Therefore, this invention creates a learning index based on the LSM tree storage structure of time-series databases to fully leverage the characteristics of learning indexes. Furthermore, research has revealed that time-series databases often store data collected from a single data source centrally. This makes it easier to identify patterns in data collected from a single source, facilitating the subsequent use of multiple models to learn the data distribution. Therefore, this invention creates a learning index based on the management methods of time-series databases, enabling comprehensive and effective management of these databases. In addition, this invention creates three different models specifically for learning data distribution based on the different data distribution rules of time-series databases. This approach, combined with the characteristics of time-series database data storage, significantly improves model accuracy. Specific Implementation Method Two:

[0064] This embodiment describes a multi-model learning index system for time series databases. After establishing the learning index using the multi-model learning index construction method for time series databases described in Specific Embodiment 1, the established learning index can form a multi-model learning index system for time series databases.

[0065] The multi-model learning indexing system for time series databases described in this embodiment includes a data encoding unit and a learning indexing unit; the data is data from a time series database, and the data storage structure of the time series database is an LSM tree;

[0066] The encoding unit encodes the data, and the encoding process includes the following steps:

[0067] For integer data, the ZigZag algorithm is used for encoding;

[0068] For timestamp data, first perform differential encoding on the timestamp data to obtain a difference array. If all values ​​in the difference array are the same, then RLE encoding is used for all values ​​in the difference array. Otherwise, continue to check if the maximum value in the difference array is greater than (1<<60)-1. If it is greater, then RAW encoding is used for all values ​​in the difference array; if it is not greater, then Packed encoding is used for all values ​​in the difference array.

[0069] The learning index unit is used to provide an index for the queried data. The index is a learning index, and the process of building the learning index includes the following steps:

[0070] A constant model is used to build a learning index for RLE-encoded timestamp data;

[0071] For integer data whose values ​​after ZigZag algorithm encoding do not exceed (1<<60)-1, a segmented model is used to build a learning index for timestamp data and floating-point numbers encoded with Packed encoding.

[0072] For integer data whose values ​​are greater than (1<<60)-1 after ZigZag algorithm encoding, as well as timestamps and strings encoded in RAW, a linear model is used to build a learning index.

[0073] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for constructing a multi-model learning index for time series databases, characterized in that, The method includes a process of encoding data and a process of building a learning index; the data is data from a time series database, and the data storage structure of the time series database is an LSM tree; The process of encoding the data includes the following steps: For integer data, the ZigZag algorithm is used for encoding; For timestamp data, first perform differential encoding on the timestamp data to obtain a difference array. If all values ​​in the difference array are the same, then RLE encoding is used for all values ​​in the difference array. Otherwise, continue to check if the maximum value in the difference array is greater than (1<<60)-1. If it is greater, then RAW encoding is used for all values ​​in the difference array; if it is not greater, then Packed encoding is used for all values ​​in the difference array. The process of constructing the learning index includes the following steps: A constant model is used to build a learning index for RLE-encoded timestamp data; For integer data whose values ​​after ZigZag algorithm encoding do not exceed (1<<60)-1, a segmented model is used to build a learning index for timestamp data and floating-point numbers encoded with Packed encoding. For integer data whose values ​​are greater than (1<<60)-1 after ZigZag algorithm encoding, as well as timestamps and strings encoded in RAW, a linear model is used to build a learning index.

2. The method for constructing a multi-model learning index for time series databases according to claim 1, characterized in that, For integer data, if the value encoded by the ZigZag algorithm does not exceed (1<<60)-1, the encoded value is compressed using the simple8b algorithm; if it exceeds this value, no compression is performed. The Gorilla compression algorithm is used to compress floating-point numbers; For string data, the Snappy compression algorithm is used for compression.

3. The method for constructing a multi-model learning index for time series databases according to claim 2, characterized in that, For integer data that is not compressed, additional storage space is still needed as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

4. The method for constructing a multi-model learning index for a time series database according to claim 3, characterized in that, Timestamp data encoded in RAW is uncompressed and still requires additional storage space as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

5. The method for constructing a multi-model learning index for a time series database according to any one of claims 1 to 4, characterized in that, When storing data corresponding to the learning index built using a segmented model, the interval data of each segment is saved. When searching for data, binary search is used to determine the interval in which the data falls and to return the value stored in the corresponding interval. That is, binary search is performed on the segment points of the segmented model for integer data, floating-point data, or packed timestamp data. Each segment stores one value. When the interval in which the data falls is found, the value stored in the corresponding interval is returned.

6. A multi-model learning indexing system for time series databases, characterized in that, The system includes a data encoding unit and a learning index unit; the data is from a time series database, and the data storage structure of the time series database is an LSM tree; The encoding unit encodes the data, and the encoding process includes the following steps: For integer data, the ZigZag algorithm is used for encoding; For timestamp data, first perform differential encoding on the timestamp data to obtain a difference array. If all values ​​in the difference array are the same, then RLE encoding is used for all values ​​in the difference array. Otherwise, continue to check if the maximum value in the difference array is greater than (1<<60)-1. If it is greater, then RAW encoding is used for all values ​​in the difference array; if it is not greater, then Packed encoding is used for all values ​​in the difference array. The learning index unit is used to provide an index for the queried data. The index is a learning index, and the process of building the learning index includes the following steps: A constant model is used to build a learning index for RLE-encoded timestamp data; For integer data whose values ​​after ZigZag algorithm encoding do not exceed (1<<60)-1, a segmented model is used to build a learning index for timestamp data and floating-point numbers encoded with Packed encoding. For integer data whose values ​​are greater than (1<<60)-1 after ZigZag algorithm encoding, as well as timestamps and strings encoded in RAW, a linear model is used to build a learning index.

7. The multi-model learning indexing system for time series databases according to claim 6, characterized in that, For integer data, if the value encoded by the ZigZag algorithm does not exceed (1<<60)-1, the encoded value is compressed using the simple8b algorithm; if it exceeds this value, no compression is performed. The Gorilla compression algorithm is used to compress floating-point numbers; For string data, the Snappy compression algorithm is used for compression.

8. The multi-model learning indexing system for time series databases according to claim 7, characterized in that, For integer data that is not compressed, additional storage space is still needed as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

9. The multi-model learning indexing system for time series databases according to claim 8, characterized in that, Timestamp data encoded in RAW is uncompressed and still requires additional storage space as a flag to indicate the compression algorithm used; however, the compression algorithm indicates that no compression has been performed.

10. The multi-model learning indexing system for time series databases according to any one of claims 6 to 9, characterized in that, When storing data corresponding to the learning index built using a segmented model, the interval data of each segment is saved. When searching for data, binary search is used to determine the interval in which the data falls and to return the value stored in the corresponding interval. That is, binary search is performed on the segment points of the segmented model for integer data, floating-point data, or packed timestamp data. Each segment stores one value. When the interval in which the data falls is found, the value stored in the corresponding interval is returned.

Images