A method for efficient storage and query of time series data

Abstract

The application relates to a time series data efficient storage and query method, comprising the following steps: S1, respectively calculating the access heat of each time series data; S2, dividing each time series data into different levels according to the access heat of the time series data, and storing the time series data in corresponding storage media; S3, adaptively compressing the time series data according to the numerical variance, numerical repetition rate and base number of each time series data; S4, indexing a query request of a user according to the query frequency, time span, label filtering quantity and aggregation query proportion of the query request, obtaining an index result; S5, adaptively optimizing the index result according to the predicate selection rate and scanning cost of the query request; and S6, evaluating the adaptively compressed time series data, and performing a maintenance operation on the time series data reaching a maintenance condition. The time series data efficient storage and query method has the advantages that the access performance and storage cost of the time series data are finely balanced.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a method for efficient storage and retrieval of time-series data. Background Technology

[0002] With the rapid development of IoT technology and the continuous expansion of industrial monitoring applications, sensor devices continuously generate massive amounts of time-series data at fixed intervals, placing stringent demands on the write throughput and query response latency of data storage systems. Time-series data possesses distinct characteristics, including the inherent presence of timestamps, relatively fixed data structures, and exponential growth in data volume over time. These characteristics cause traditional relational databases to face severe performance bottlenecks when processing time-series data.

[0003] Currently, time-series data storage is divided into two levels: memory storage and file storage. Newly written time-series data is first temporarily stored in memory and then persistently saved to file storage via a background thread. This addresses the issue of fast data writing and avoids performance degradation caused by random writes through append-only writing. During queries, separate query mechanisms are used for memory-stored and file-stored data. A timestamp-based linear interpolation formula achieves data location with constant time complexity, and an efficient location algorithm is designed leveraging the fixed acquisition frequency of time-series data. However, this storage method cannot dynamically adjust the storage levels according to the actual access characteristics of the data, resulting in inefficient use of storage resources and an overall decline in access performance. It also struggles to achieve an effective balance between high compression ratios and decompression performance. Furthermore, this query method lacks multi-dimensional index support, leading to significant inefficiencies in query efficiency for complex query scenarios involving tag filtering and aggregation calculations, and exhibiting high response latency when facing complex analytical queries. Therefore, there is an urgent need for a query method that can efficiently utilize storage resources and access performance, balance high compression ratio and decompression performance, and achieve higher query efficiency and lower response latency. Summary of the Invention

[0004] Based on this, the purpose of this invention is to improve the storage and retrieval efficiency of time-series data and to provide a method for efficient storage and retrieval of time-series data.

[0005] An efficient method for storing and retrieving time-series data includes:

[0006] S1: Calculate the access frequency of each time series data;

[0007] S2: The time-series data are divided into different levels according to their access frequency and stored in corresponding storage media; wherein, when the access frequency is greater than or equal to a preset hot data threshold, the corresponding time-series data is determined as hot data and stored in the memory storage layer; when the access frequency is less than a preset cold data threshold, the time-series data is determined as cold data and stored in the object storage layer; when the access frequency is less than the hot data threshold but greater than or equal to the cold data threshold, the time-series data is determined as warm data and stored in the solid-state storage layer.

[0008] S3: Perform adaptive compression based on the numerical variance, numerical repetition rate, and cardinality of each time series data;

[0009] S4: Index the query request based on the query frequency, time span, number of tag filters, and aggregation query ratio to obtain the index result;

[0010] S5: Adaptively optimize the index results based on the predicate selectivity and scan cost of the query request;

[0011] S6: Evaluate the adaptively compressed time series data, and perform maintenance operations on the time series data that meet the maintenance conditions.

[0012] Compared to existing technologies, this invention employs a three-tiered hierarchical storage architecture that classifies and stores time-series data in different storage layers based on access frequency. This architecture enables rapid access to frequently accessed data and low-cost archiving of historical data, achieving a fine balance between access performance and storage costs. While ensuring rapid access to frequently accessed data, it significantly reduces the storage overhead of massive amounts of historical data, effectively improving the storage and query efficiency of time-series data. Furthermore, adaptive compression is performed based on the specific content of the time-series data, ensuring both compression speed and effectiveness. In addition, through multi-dimensional indexing and optimization, the query efficiency and accuracy of time-series data are improved for specific query requests.

[0013] Furthermore, the formula for calculating the access popularity is:

[0014] ,

[0015] in, For the number to be calculated One time series data, For time series data Popularity rating; For time series data Access frequency within the statistical period, For time series data The time interval since the most recent access. The time decay factor, For frequency weighting coefficients, It is a logarithmic function. It is an exponentially decaying function.

[0016] Furthermore, step S3 specifically includes:

[0017] S31: Each of the time-series data is divided into a compression unit according to a fixed length;

[0018] S32: Extract the numerical variance, numerical repetition rate, and cardinality of each compression unit;

[0019] S33: Calculate the differential coding weight, run-length coding weight, and dictionary coding weight based on the numerical variance, numerical repetition rate, and cardinality of each compression unit;

[0020] S34: Select a compression strategy for each compression unit according to the differential coding weight, run-length coding weight and dictionary coding weight, and perform compression to obtain a compressed block;

[0021] S35: Write the encoding strategy identifier and decompression parameters at the beginning of the compressed block.

[0022] Further, in step S34, the compression strategy includes a single encoding method and a concatenated combination method. When the difference between one of the encoding weights and the other encoding weights is greater than a preset weight threshold, the encoding method with the highest encoding weight is adopted. The concatenated combination method is adopted when the difference between the encoding weights is less than the weight threshold. First, differential encoding is applied to eliminate temporal correlation, then run-length encoding is applied to compress continuous repeated values, and finally dictionary encoding is applied to reduce the storage overhead of unique values.

[0023] Furthermore, step S3 also includes:

[0024] S36: Calculate the compression ratio of each compression block;

[0025] S37: When the compression ratio is higher than the compression threshold, the thermal data threshold is appropriately increased; when the compression ratio is lower than the compression threshold, the thermal data threshold is decreased.

[0026] The formula for calculating the compression ratio is:

[0027] ,

[0028] in, To show the compression ratio, Indicates the size of the compressed data. This indicates the original data size. The compression ratio is calculated to measure the compression effect and fed back to the data hierarchy classification. When the compression ratio is lower than expected, hot data can be increased to reduce memory storage pressure; when the compression ratio is higher than expected, the hot data threshold is lowered to retain more data in the high-performance storage layer, ensuring the efficiency and performance of time-series data storage and retrieval.

[0029] Furthermore, step S4 specifically includes:

[0030] S41: Collect and analyze the query frequency, time span, number of tag filters, and proportion of aggregated queries for user query requests;

[0031] S42: Calculate the priority score based on the query frequency, time span, number of tag filters, and aggregate query ratio;

[0032] S43: Select the corresponding indexing method based on the priority score to index the query request and obtain the indexing result.

[0033] Furthermore, the expression for the priority scoring is:

[0034] ,

[0035] in, Build priority scores for indexes. For query frequency weighting, Weighted by time span Weighted by the number of tags. For aggregation ratio weight, For query frequency, For the time span, For the number of tags to filter, This represents the aggregation query ratio. This represents the maximum query frequency within the statistical period. For time span mapping function, This is a function for mapping the number of tags.

[0036] Furthermore, the expression for the differential coding weights is:

[0037] ,

[0038] in, For the selection of weights in differential encoding, The variance of the current data block. The reference variance threshold is used for the normalized variance index;

[0039] The expression for the run-length encoding weight is:

[0040] ,

[0041] in, Selection weights for run-length encoding; This represents the repetition rate sensitivity index, which controls the strength of the influence of repetition rate on weights.

[0042] The expression for the dictionary encoding weight is:

[0043] ,

[0044] in, The dimensionless numerical value representing the selection weight of the dictionary encoding ranges from [0, 1]. Indicates the cardinality of the data block. This indicates the total number of data points in the data block.

[0045] Furthermore, step S5 specifically includes:

[0046] S51: Parse the query request and extract its time range predicate, tag matching predicate, and numerical filtering predicate;

[0047] S52: Calculate the predicate selectivity and scan cost of the query request;

[0048] S53: When the predicate selection rate is less than the pushdown threshold, the query request is processed by predicate pushdown to obtain the query result; otherwise, step S54 is executed.

[0049] S54: When the scanning cost is less than the cost threshold, the query request is scanned in parallel; otherwise, step S55 is executed.

[0050] S55: Perform pre-computed materialized view processing on the query request.

[0051] Furthermore, the formula for calculating the predicate selectivity rate is as follows:

[0052] ,

[0053] in, This represents the overall selectivity of the combined predicates; the smaller the value, the better the filtering effect. Indicates the number of predicate conditions; Indicates the first The independent selection rate of each predicate condition is estimated based on column statistics.

[0054] The formula for estimating the scanning cost is:

[0055] ,

[0056] in, This is the estimated time for parallel scanning, in milliseconds (ms). This refers to the number of data partitions involved. This represents the average data size of a single partition, in bytes. Indicates the predicate selectivity; This represents the parallel read bandwidth, expressed in bytes per millisecond. This indicates the number of parallel threads.

[0057] Based on the same inventive concept, the present invention also includes an electronic device comprising a processor; a memory for storing a computer program executed by the processor; wherein, when the processor executes the computer program, it implements the time-series data efficient storage and retrieval method described above.

[0058] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0059] Figure 1 This is a flowchart of the method for efficient storage and retrieval of time-series data according to the present invention;

[0060] Figure 2 Here is a detailed flowchart of step S3;

[0061] Figure 3 Here is a detailed flowchart of step S4;

[0062] Figure 4 The detailed flowchart for step S5 is shown below. Detailed Implementation

[0063] This invention addresses a series of problems existing in the storage and querying methods of time-series data, such as low storage resource utilization and high query response latency. It designs an efficient storage and querying method for time-series data, which includes a three-layer hierarchical storage architecture, a feature-aware adaptive compression combination strategy, multi-dimensional index collaboration, and index query optimization. This method enables fast access to hot data and low-cost archiving of historical data, as well as higher compression performance, shorter complex query response time, and higher aggregation query performance.

[0064] Specifically, please refer to Figure 1 , Figure 1 This is a flowchart of the time-series data efficient storage and retrieval method of the present invention. The present invention provides a time-series data efficient storage and retrieval method, specifically including the following steps:

[0065] S1: Calculate the access popularity of each time series data point. The access popularity is calculated using a comprehensive evaluation model that integrates access frequency and time decay factor. For any time series data point, the formula for calculating its access popularity is:

[0066] ,

[0067] in, For the number to be calculated One time series data, For time series data The popularity score has a value of [0, 100]. For time series data The frequency of accesses within the statistical period, expressed as times per hour; Number of time series data The time interval since the most recent access, in hours; The time decay factor is a positive real number between 0 and 1, with units of / h. It controls the rate at which the data heat decreases over time; the larger the value, the faster the heat decays. This is the frequency weighting coefficient, with a value range of [0,1], used to balance the contribution ratio of access frequency and time decay to the popularity score; It is a logarithmic function, which smooths the access frequency to avoid excessive inflation of the popularity score of extremely high-frequency access data. The time decay function simulates the natural decline in data access popularity over time, thus conforming to the temporal locality characteristic of time-series data access. In this embodiment, the time decay factor is set to 0.05, and the frequency weighting coefficient is set to 0.6.

[0068] S2: The time-series data are categorized into different levels based on their access frequency and stored in corresponding storage media. By comparing the access frequency score and a hot data threshold with a cold data threshold, the time-series data are categorized into hot data, warm data, and cold data. Specifically, when the access frequency score is greater than or equal to the hot data threshold... At that time, that is When the time series data is determined to be hot data, the heat score is determined to be less than the cold data threshold. At that time, that is When the time series data is less than the hot data threshold, it is classified as cold data; when the heat score is less than the hot data threshold, it is classified as cold data. And greater than or equal to the cold data threshold. At that time, that is At that time, the time-series data is determined to be warm data. Subsequently, the hot data is stored in the memory storage layer, the warm data is stored in the solid-state storage layer, and the cold data is stored in the object storage layer; wherein, the memory storage layer uses a circular buffer structure to organize the data, the solid-state storage layer uses a partitioned time-series file format, and the object storage layer uses a columnar storage format. In this embodiment, the hot data threshold... The cold data threshold is 70. With a value of 35, approximately 20% of high-frequency access data is retained in the memory layer to ensure millisecond-level response, 30% of mid-frequency access data is stored in the solid-state storage layer to balance performance and cost, and 50% of low-frequency historical data is moved to the object storage layer to minimize storage overhead.

[0069] S3: Adaptive compression is performed based on the numerical variance, numerical repetition rate, and cardinality of each time series data point. The numerical variance reflects the dispersion of values ​​within the time series data, and is the expected value of the squared difference between a data point and the mean. The numerical repetition rate represents the proportion of adjacent data points with the same value, and is the number of adjacent identical data point pairs divided by the total number of data point pairs. The cardinality represents the number of different value types within the time series data. The adaptive compression intelligently selects the optimal encoding combination strategy based on the numerical distribution of the time series data to achieve a dual optimization of high compression ratio and fast decompression performance. The adaptive compression strategy library includes differential coding, run-length encoding, and dictionary coding. Differential coding stores the difference between adjacent data points instead of the original values. For time-series data with gradual changes, the absolute value of the difference is usually much smaller than the original value, thus requiring fewer bits for storage. Run-length encoding compresses consecutively repeating numerical sequences into tuples of value and repetition count, achieving extremely high compression efficiency for time-series data with a large number of consecutive identical values. Dictionary coding establishes a mapping dictionary from values ​​to codes, using fixed-length codes instead of variable-length original values. Significant compression gains can be achieved when the cardinality is much smaller than the total number of data points. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a detailed flowchart of step S3. Step S3 specifically includes:

[0070] S31: The timing data is divided into compression units according to a fixed length. The fixed length is 64KB, and each compression unit is a data block.

[0071] S32: Extract the numerical variance, numerical repetition rate, and cardinality of each compression unit.

[0072] S33: Calculate the differential coding weight, run-length coding weight, and dictionary coding weight based on the numerical variance, numerical repetition rate, and cardinality of each compression unit.

[0073] The expression for the differential coding weights is:

[0074] ,

[0075] in, For the selection of weights in differential encoding, The variance of the current data block. A reference variance threshold is used to normalize the variance index. In this embodiment, the reference variance threshold is set to 100. The differential encoding weight is a dimensionless value based on the numerical variance. The smaller the numerical variance, the smoother the data variation, the better the compression effect of differential encoding, and the higher the differential encoding weight.

[0076] The expression for the run-length encoding weight is:

[0077] ,

[0078] in, Selection weights for run-length encoding; The repetition rate sensitivity index controls the strength of the impact of repetition rate on weights. The run-length encoding weights amplify the weight differences in high-repetition-rate scenarios using a power function, meaning that the higher the repetition rate of the data block, the more significant the compression benefit of run-length encoding. In this embodiment, the repetition rate sensitivity index is set to 1.5.

[0079] The expression for the dictionary encoding weight is:

[0080] ,

[0081] in, The dimensionless numerical value representing the selection weight of the dictionary encoding ranges from [0, 1]. Indicates the cardinality of the data block. This represents the total number of data points in the data block. In the dictionary encoding weight expression, a lower cardinality of the data block means fewer different numerical types, resulting in a more significant space saving effect for dictionary encoding. Consistency across data scales is achieved through logarithmic ratio normalization. In this embodiment, a single data block contains 65536 data points.

[0082] S34: Based on the differential coding weights, run-length encoding weights, and dictionary coding weights, select a compression strategy for each compression unit and perform compression to obtain a compressed block. The compression strategy includes a single coding method and a concatenated combination method. The single coding method uses the coding method corresponding to the highest weight when the difference between a certain coding weight and other weights is greater than a preset weight threshold. The concatenated combination method is used when the difference between each coding weight is less than the weight threshold. It first applies differential coding to eliminate temporal correlation, then applies run-length encoding to compress consecutive repeated values, and finally applies dictionary coding to reduce the storage overhead of unique values. In this embodiment, the weight threshold is 0.15. For example, when the difference between the differential coding weight and both the run-length encoding weight and the dictionary coding weight is greater than 0.15, differential coding is used for compression.

[0083] S35: Write the encoding strategy identifier and decompression parameters at the beginning of the compressed block.

[0084] S36: Calculate the compression ratio of each compression block. The formula for calculating the compression ratio is:

[0085] ,

[0086] in, The compression ratio is a dimensionless value ranging from (0,1). The smaller the value, the better the compression effect. This indicates the size of the compressed data, in bytes. This indicates the size of the original data, in bytes.

[0087] S37: Dynamically adjust the hot data threshold and cold data threshold based on the compression ratio. When the compression ratio is higher than the compression threshold, appropriately increase the hot data threshold; when the compression ratio is lower than the compression threshold, decrease the hot data threshold. By increasing the hot data threshold, more data is retained in the high-performance storage layer when the compression ratio is better than expected; by decreasing the hot data threshold, memory storage pressure is reduced when the compression ratio is worse than expected.

[0088] S4: Perform a multi-dimensional adaptive index on the query request based on the query frequency, time span, number of tag filters, and aggregation query ratio to obtain the index result. Please refer to [link / reference]. Figure 3 , Figure 3 This is a detailed flowchart of step S4 of the present invention. Step S4 specifically includes:

[0089] S41: Collect and statistically analyze the query frequency, time span, number of tag filters, and aggregation query ratio of the query requests.

[0090] S42: Calculate the priority score based on the query frequency, time span, number of filtered tags, and proportion of aggregated queries. The priority score adopts a multi-factor weighted model, and its expression is:

[0091] ,

[0092] in, Build priority scores for indexes. For query frequency weighting, Weighted by time span Weighted by the number of tags. For aggregation ratio weight, For query frequency, For the time span, For the number of tags to filter, This represents the aggregation query ratio. This represents the maximum query frequency within the statistical period. This is a time span mapping function that maps a time span to a score value in the interval [0, 1]. This is a tag number mapping function that maps the tag number to a score value in the interval [0, 1].

[0093] The query frequency is the number of query requests per unit time, expressed as times per hour (times / h), and is a non-negative real number greater than or equal to 0; the time span is the size of the time window covered by the query request, expressed as seconds (s), and is a positive real number greater than 0; the number of tag filters is the number of tag filtering conditions included in the query conditions, and is a non-negative integer greater than or equal to 0; the aggregation query ratio is the proportion of queries containing aggregation operations to the total number of queries, and its value range is [0, 1].

[0094] In this embodiment, the query frequency weight is 0.3, the time span weight is 0.2, the tag quantity weight is 0.25, and the aggregation ratio weight is 0.25. The time span mapping function adopts a piecewise linear function, mapping 0.2 for less than 1 hour, 0.5 for 1 hour to 24 hours, 0.8 for 24 hours to 7 days, and 1.0 for more than 7 days. The tag quantity mapping function adopts a logarithmic mapping, and its expression is:

[0095] .

[0096] S43: Select the corresponding indexing method based on the priority score to index the query request and obtain the indexing result. In this embodiment, the indexing method library for selecting the indexing method based on the priority score includes time range index, tag inverted index, and downsampling aggregate index.

[0097] S44: Calculate the index hit rate and optimize the compression strategy based on the index hit rate. For data blocks with a high index hit rate, a more aggressive compression strategy is adopted to save storage space; for data blocks with a low index hit rate, decompression performance is prioritized to cope with possible full scan scenarios. In this embodiment, an aggressive compression strategy is adopted when the index hit rate reaches 80%, and decompression performance is prioritized when the index hit rate is below 50%.

[0098] S5: Adaptively optimize the index results based on the predicate selectivity and scan cost of the query request. The adaptive optimization employs predicate pushdown, parallel scanning, and pre-computed materialized views. Figure 3The three optimization strategies employ a gradient selection relationship from low to high cost: when the predicate selectivity is low, predicate pushdown achieves efficient filtering with minimal computational cost and is prioritized; when the predicate selectivity is high and the scan cost is controllable, parallel scanning leverages multi-threaded concurrency to accelerate data reading; when the scan cost exceeds a threshold, pre-computed materialized views are used to directly return cached results to avoid redundant calculations. These three optimization strategies progressively ensure the selection of the optimal execution path for different query characteristics. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a detailed flowchart of step S5 of the present invention. Step S5 specifically includes:

[0099] S51: Parse the query request and extract its time range predicate, tag matching predicate, and numerical filtering predicate. The time range predicate corresponds to the timestamp comparison expression in the query conditions, the tag matching predicate corresponds to the tag key-value matching expression in the query conditions, and the numerical filtering predicate corresponds to the data value comparison expression in the query conditions.

[0100] S52: Calculate the predicate selectivity and scan cost of the query request. The formula for calculating the predicate selectivity is:

[0101] ,

[0102] in, This represents the overall selectivity of the combined predicates; the smaller the value, the better the filtering effect. Indicates the number of predicate conditions; Indicates the first The independent selection rate of each predicate condition is estimated based on column statistics.

[0103] The formula for estimating the scanning cost is:

[0104] ,

[0105] in, This is the estimated time for parallel scanning, in milliseconds (ms). This refers to the number of data partitions involved. This represents the average data size of a single partition, in bytes. Indicates the predicate selectivity; This represents the parallel read bandwidth, expressed in bytes per millisecond. This indicates the number of parallel threads.

[0106] S53: When the predicate selectivity is less than the pushdown threshold, predicate pushdown processing is performed on the query request to obtain the query result; otherwise, step S54 is executed. The predicate pushdown processing directly pushes the time range predicate down to the time range index layer for execution, pushes the tag matching predicate down to the tag inverted index layer for execution, and pushes the numerical filtering predicate down to the data scanning layer for execution. This advances the filtering conditions in the user query request to the storage layer, avoiding loading a large amount of irrelevant data into the computation layer before filtering. In this embodiment, the pushdown threshold is 0.1. When the predicate selectivity... At that time, the benefits of predicate pushdown are significant.

[0107] S54: When the scanning cost is less than the cost threshold, the query request is scanned in parallel; otherwise, step S55 is executed. The parallel scan splits the query request into multiple subtasks, creates a thread pool to concurrently execute the data reading and filtering operations of each subtask, and collects the execution results of each subtask and merges them into the query result in chronological order; each subtask scans the corresponding index result. In this embodiment, the cost threshold is set to 500ms, that is, when the estimated time of the parallel scan exceeds 500ms, the scanning cost is determined to be too high, and a pre-calculated materialized view strategy is adopted instead.

[0108] S55: Perform pre-computation materialized view processing on the query request. This pre-computation materialized view processing pre-calculates and caches query results for high-frequency aggregation query scenarios. When a query request hits a materialized view, the pre-computation result is returned directly, avoiding redundant aggregation calculation overhead. Specifically, it includes two modes: synchronous update and asynchronous update. The synchronous update mode updates the relevant materialized view immediately when the underlying data is written, ensuring strong consistency between the view and the original data. The asynchronous update mode uses a delayed refresh mechanism to batch update materialized views when the system load is low, suitable for analysis scenarios where consistency requirements are not strict. The two modes are selected based on query frequency, computational cost, storage cost, and update cost, prioritizing the materialization of aggregation results corresponding to high-frequency queries and aggregation operations with high computational costs, ensuring that the view size does not exceed the configured storage quota, and that the maintenance overhead of the view is within an acceptable range when the underlying data changes. In this embodiment, the pre-computation materialized view processing uses an automated materialized view selection algorithm, which determines the set of materialized views that need to be maintained based on query mode statistics and storage cost constraints.

[0109] S56: Dynamically adjust the index building priority and index structure parameters according to the method of processing the query request.

[0110] S6: Evaluate the adaptively compressed time-series data and perform maintenance operations on the time-series data that meet the maintenance conditions. The maintenance operations include downsampling, cross-layer migration, expired deletion, and alarms. The maintenance conditions include the upper limit of the survival of time-series data, the lower limit of its value, and storage constraints. The cross-layer migration automatically adjusts the storage location of the data based on the dynamic changes in the data popularity score; when the popularity score of hot data drops below the warm data threshold, the data block is migrated from the memory storage layer to the solid-state storage layer; when the popularity score of warm data further drops below the cold data threshold, the data block is migrated from the solid-state storage layer to the object storage layer; and when the popularity score of time-series data rises, reverse migration is performed. The downsampling process adopts a segmented downsampling strategy, setting different downsampling ratios according to the data age, and retaining the maximum, minimum, average, and last values ​​within each sampling window. The alarm is triggered when the usage rate of any storage layer exceeds the threshold, and the downsampling process or cross-layer migration operation is accelerated simultaneously to speed up data sinking.

[0111] In this embodiment, data older than 7 days is kept at its original resolution without downsampling, ensuring that recent data retains its original accuracy to support detailed analysis; intermediate data (7 to 30 days old) is downsampled at a ratio of 10:1, retaining a 1-minute granularity to meet daily monitoring needs; historical data (30 to 90 days old) is downsampled at a ratio of 60:1, retaining a 10-minute granularity to support trend analysis; archived data older than 90 days is downsampled at a ratio of 360:1, retaining a 1-hour granularity for long-term review. The downsampling ratio represents a dimensionless positive real number greater than 1, representing the ratio of the number of data points before and after downsampling. Simultaneously, the capacity alarm threshold for the memory storage layer is set to 80%, the capacity alarm threshold for the solid-state storage layer is set to 85%, and the capacity alarm threshold for the object storage layer is set to 90%.

[0112] Furthermore, when maintaining the time-series data, the capacity allocation ratio of each storage layer and the data migration trigger threshold in step S1 are adjusted according to the storage utilization.

[0113] Compared to existing technologies, this invention employs a three-tiered hierarchical storage architecture that classifies and stores time-series data in different storage layers based on access frequency. This architecture enables rapid access to frequently accessed data and low-cost archiving of historical data, achieving a fine balance between access performance and storage costs. While ensuring rapid access to frequently accessed data, it significantly reduces the storage overhead of massive amounts of historical data, effectively improving the storage and query efficiency of time-series data. Furthermore, adaptive compression is performed based on the specific content of the time-series data, ensuring both compression speed and effectiveness. In addition, through multi-dimensional indexing and optimization, the query efficiency and accuracy of time-series data are improved for specific query requests.

[0114] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments and claims of this application are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that, unless otherwise stated, “a plurality” means two or more; the terms “first,” “second,” “third,” etc., are used only to distinguish and not to describe a particular order or sequence, nor should they be construed as indicating or implying relative importance. The term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items. When the above description relates to drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of this application, those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0115] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A method for efficient storage and retrieval of time-series data, characterized in that, include: S1: Calculate the access frequency of each time series data; S2: The time-series data are divided into different levels according to their access frequency and stored in corresponding storage media; wherein, when the access frequency is greater than or equal to a preset hot data threshold, the corresponding time-series data is determined as hot data and stored in the memory storage layer; when the access frequency is less than a preset cold data threshold, the time-series data is determined as cold data and stored in the object storage layer; when the access frequency is less than the hot data threshold but greater than or equal to the cold data threshold, the time-series data is determined as warm data and stored in the solid-state storage layer. S3: Adaptive compression is performed based on the numerical variance, numerical repetition rate, and cardinality of each time series data; specifically including: S31: Each of the time-series data is divided into a compression unit according to a fixed length; S32: Extract the numerical variance, numerical repetition rate, and cardinality of each compression unit; S33: Calculate the differential coding weight, run-length coding weight, and dictionary coding weight based on the numerical variance, numerical repetition rate, and cardinality of each compression unit; S34: Select a compression strategy for each compression unit according to the differential coding weight, run-length coding weight, and dictionary coding weight, and compress the compressed block to obtain a compressed block; wherein, the compression strategy includes a single coding method and a concatenated combination method. When the difference between one of the coding weights and the other coding weights is greater than a preset weight threshold, the coding method with the highest coding weight is adopted; the concatenated combination method is adopted when the difference between the coding weights is less than the weight threshold. First, differential coding is applied to eliminate temporal correlation, then run-length coding is applied to compress continuous repeated values, and finally dictionary coding is applied to reduce the storage overhead of unique values. S35: Write the encoding strategy identifier and decompression parameters at the beginning of the compressed block; S4: Index the query request based on its query frequency, time span, number of filtered tags, and aggregation query ratio to obtain the index results; specifically including: S41: Collect and statistically analyze the query frequency, time span, number of tag filters, and aggregate query ratio of the query requests; S42: Calculate the priority score based on the query frequency, time span, number of tag filters, and aggregate query ratio; the priority score adopts a multi-factor weighted model; S43: Select the corresponding indexing method based on the priority score to index the query request and obtain the indexing result; S44: Calculate the index hit rate and optimize the compression strategy based on the index hit rate; S5: Adaptively optimize the index results based on the predicate selectivity and scan cost of the query request; specifically including: S51: Parse the query request and extract its time range predicate, tag matching predicate, and numerical filtering predicate; S52: Calculate the predicate selectivity and scan cost of the query request; S53: When the predicate selection rate is less than the pushdown threshold, the query request is processed by predicate pushdown to obtain the query result; otherwise, step S54 is executed. S54: When the scanning cost is less than the cost threshold, the query request is scanned in parallel; otherwise, step S55 is executed. S55: Perform pre-calculated materialized view processing on the query request; S56: Dynamically adjust the index building priority and index structure parameters according to the method for processing the query request; S6: Evaluate the adaptively compressed time series data, and perform maintenance operations on the time series data that meet the maintenance conditions.

2. The method for efficient storage and retrieval of time-series data according to claim 1, characterized in that, The formula for calculating the access popularity is: ， in, For the number to be calculated One time series data, For time series data Popularity rating; For time series data Access frequency within the statistical period, For time series data The time interval since the most recent access. The time decay factor, For frequency weighting coefficients, It is a logarithmic function. It is an exponentially decaying function.

3. The method for efficient storage and retrieval of time-series data according to claim 1, characterized in that, Step S3 also includes: S36: Calculate the compression ratio of each compression block; S37: When the compression ratio is higher than the compression threshold, increase the thermal data threshold; when the compression ratio is lower than the compression threshold, decrease the thermal data threshold. The formula for calculating the compression ratio is: ， in, For compression ratio, Indicates the size of the compressed data. Indicates the size of the original data.

4. The method for efficient storage and retrieval of time-series data according to claim 1, characterized in that, The expression for the priority score is: ， in, Build priority scores for indexes. For query frequency weighting, Weighted by time span Weighted by the number of tags. For aggregation ratio weight, For query frequency, For the time span, For the number of tags to filter, This represents the aggregation query ratio. This represents the maximum query frequency within the statistical period. For time span mapping function, This is a function for mapping the number of tags.

5. The method for efficient storage and retrieval of time-series data according to claim 1, characterized in that, The expression for the differential coding weights is: ， in, For the selection of weights in differential encoding, The variance of the current data block. The reference variance threshold is used for the normalized variance index; The expression for the run-length encoding weight is: ， in, Selection weights for run-length encoding; This represents the repetition rate sensitivity index, which controls the strength of the influence of repetition rate on weights. The expression for the dictionary encoding weight is: ， in, The dimensionless numerical value representing the selection weight of the dictionary encoding ranges from [0, 1]. Indicates the cardinality of the data block. This indicates the total number of data points in the data block.

6. The method for efficient storage and retrieval of time-series data according to claim 1, characterized in that, The formula for calculating the predicate selectivity rate is as follows: ， in, This represents the overall selectivity of the combined predicates; the smaller the value, the better the filtering effect. Indicates the number of predicate conditions; Indicates the first The independent selection rate of each predicate condition is estimated based on column statistics. The formula for estimating the scanning cost is: ， in, This is the estimated time for parallel scanning, in milliseconds. This refers to the number of data partitions involved. This represents the average data size of a single partition, in bytes. Indicates the predicate selectivity; This represents the parallel read bandwidth, expressed in bytes per millisecond. This indicates the number of parallel threads.

7. An electronic device, characterized in that, processor; Memory for storing computer programs executed by the processor; Wherein, when the processor executes the computer program, it implements the time-series data efficient storage and retrieval method according to any one of claims 1-6.

Images