An Incremental Update Method and System for Time-Series Data in an Adaptive Hierarchical Index

Abstract

This invention provides a method and system for incremental time-series data updates using adaptive hierarchical indexes, belonging to the field of database indexing technology. The method includes: constructing a three-level hierarchical index structure comprising a hot layer, a warm layer, and a cold layer; the hot layer using an in-memory skip list, the warm layer using an LSM tree, and the cold layer using a segmented B+ tree; configuring an incremental merge scheduler to dynamically trigger inter-layer data sinking and index merging based on data volume, query frequency, and write rate; employing a lock-free concurrency strategy to avoid merging blocking foreground read / write operations; embedding time-series feature summaries in index nodes to implement query pruning; and introducing a pre-write log and checkpoint mechanism to ensure atomicity and crash recovery. This invention can maintain stable query latency under continuous high-throughput write conditions.

Description

Technical Field

[0001] This invention relates to the field of database indexing technology, and in particular to a time-series data incremental update method and system for adaptive hierarchical indexes. Background Technology

[0002] Time-series data refers to data sequences recorded in chronological order, and is widely used in scenarios such as IoT sensor monitoring, financial transaction records, industrial equipment status collection, and internet service logs. With the deepening of digital transformation, the generation rate of time-series data is growing exponentially. A single large IoT platform can generate tens of terabytes of time-series data every day, which poses a severe challenge to the storage and indexing mechanisms of time-series databases.

[0003] Chinese patent CN118093962A discloses a data retrieval method, apparatus, system, electronic device, and readable storage medium. This technical solution constructs a multi-layered index structure with at least two layers of different index precision based on the data distribution characteristics of a vector dataset. The index data and index parameters corresponding to the index structure are dynamically updated based on changes in the data within the index and user query feedback analysis. This solution obtains an adaptive retrieval model pre-trained on a retrieval sample dataset, and fuses entity features from a knowledge graph with vector data representations from a vector database to obtain semantically enhanced vectors, thereby improving the retrieval efficiency and accuracy based on the vector database.

[0004] However, the aforementioned existing technical solutions have the following technical drawbacks: First, the hierarchical index structure of this solution is based on index precision, failing to fully consider the access frequency characteristics of time-series data. This results in frequently accessed hot data being mixed with cold data, making it difficult to optimize query performance. Second, the index updates in this solution rely on user query feedback and time-series analysis, which is a passive update mechanism. It cannot proactively sense data write pressure and dynamically adjust the index merging strategy, easily causing write blocking in high-throughput write scenarios. Third, this solution does not involve a concurrency control mechanism during the index merging process. When index reconstruction or merging operations are performed, foreground read and write requests may be blocked, leading to significant system performance fluctuations. Fourth, the index nodes in this solution only store pointers to data or detailed information, without including data statistical characteristics. Queries require accessing a large number of index nodes to locate the target data, resulting in query efficiency that needs improvement. Fifth, this solution does not provide a robust crash recovery mechanism. After an abnormal power outage or crash, incrementally updated index data may be lost or in an inconsistent state.

[0005] In summary, existing technologies cannot effectively solve the problem of system performance fluctuations caused by frequent rebuilding of traditional indexes in scenarios involving continuous writing of massive time-series data. There is an urgent need for a hierarchical incremental index update scheme that can adaptively adjust. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive hierarchical index for incremental updating of time-series data and its system, so as to solve the technical problems in the prior art that frequent rebuilding of time-series data indexes leads to system performance fluctuations, low query efficiency, and lack of crash recovery capabilities.

[0007] To achieve the above objectives, the first aspect of the present invention provides a method for incremental updating time-series data using an adaptive hierarchical index, comprising the following steps:

[0008] Step S1: Construct a three-level hierarchical index structure including hot layer index, warm layer index and cold layer index. The hot layer index uses an in-memory skip list to receive real-time writes and maintain fine-grained timestamp indexes. The warm layer index uses an LSM tree structure to store compressed index blocks of recent data. The cold layer index uses a segmented B+ tree to manage historical archived data.

[0009] Step S2: Configure the incremental merge scheduler to calculate the merge trigger conditions based on the data volume, query popularity, and write rate of each layer. When the merge trigger conditions are met, dynamically trigger the inter-layer data sinking and index merging.

[0010] Step S3: During the index merging process, a lock-free concurrency strategy is adopted. The merging thread and the foreground read / write thread are executed concurrently through the version number mechanism and atomic operations to avoid the merging operation blocking the foreground read / write.

[0011] Step S4: Embed a time-series feature summary into each index node of the thermal layer index, the warm layer index, and the cold layer index. The time-series feature summary includes the extreme value information, mean information, and trend information of the data within the corresponding time period of the index node.

[0012] Step S5: Upon receiving a query request, match the query conditions with the temporal feature summaries of each index node, and perform query pruning to skip index segments that are irrelevant to the query conditions.

[0013] Step S6 introduces a pre-write log and checkpoint mechanism, where incremental update operations are first written to the pre-write log and checkpoints are generated periodically to ensure the atomicity of incremental updates and the data recovery capability after system crashes.

[0014] A second aspect of the present invention provides an adaptive hierarchical index-based incremental update system for time-series data, comprising:

[0015] The hierarchical index building module is used to build a three-level hierarchical index structure containing hot layer index, warm layer index and cold layer index. The hot layer index uses a memory skip list, the warm layer index uses an LSM tree structure and the cold layer index uses a segmented B+ tree.

[0016] The incremental merge scheduling module is used to calculate the merge trigger conditions based on the data volume, query frequency, and write rate of each layer. When the merge trigger conditions are met, the inter-layer data sinking and index merging are dynamically triggered.

[0017] The concurrency control module is used to employ a lock-free concurrency strategy during the index merging process, and to achieve concurrent execution of the merging thread and the foreground read / write thread through version number mechanism and atomic operations;

[0018] The feature summary module is used to embed time-series feature summaries into each index node of the thermal layer index, the temperature layer index, and the cold layer index. The time-series feature summaries include extreme value information, mean information, and trend information.

[0019] The query optimization module is used to match and judge the query conditions with the time-series feature summaries of each index node, and perform query pruning to skip irrelevant index segments;

[0020] The persistence guarantee module is used to ensure the atomicity of incremental updates and crash recovery capabilities through pre-write logging and checkpointing mechanisms.

[0021] The present invention has the following advantages over the prior art:

[0022] First, this invention constructs a three-level hierarchical index structure based on data popularity (hot layer, warm layer, and cold layer). This structure keeps frequently accessed hot data in an in-memory skip list for fast read and write operations, stores recent data in an LSM tree to balance write performance and space efficiency, and stores historical archive data in a B+ tree to ensure stable query performance. The three-layer structure works together to form a non-linear performance gain. Compared with existing hierarchical schemes based on index precision, query latency is reduced by more than 40%, and write throughput is increased by more than 35%.

[0023] Second, this invention achieves joint decision-making based on three factors: data volume, query popularity, and write rate through an incremental merging scheduler. It proactively senses system load and dynamically adjusts the merging strategy, avoiding the write blocking problem caused by the passive update mechanism of existing technologies, and maintaining a stable 99th percentile query latency under continuous high-throughput write conditions.

[0024] Third, this invention enables the index merging operation and the foreground read and write operation to be executed in parallel through a lock-free concurrency strategy, eliminating the performance jitter problem during index reconstruction in existing technologies and reducing the system throughput fluctuation to within 5%.

[0025] Fourth, by embedding time-series feature summaries in index nodes and optimizing query pruning, this invention can quickly skip index segments that are irrelevant to the query conditions during the query, reduce invalid index traversal, and reduce the index access volume of range queries by more than 60%.

[0026] Fifth, this invention ensures the atomicity of incremental updates and crash recovery capabilities through a pre-write log and checkpoint mechanism, solving the problem of the lack of a sound crash recovery mechanism in the prior art. After a system crash, it can be restored to a consistent state within seconds. Attached Figure Description

[0027] Figure 1 This is a flowchart of the time-series data incremental update method for adaptive hierarchical index provided in an embodiment of the present invention.

[0028] Figure 2 This is an architecture diagram of the time-series data incremental update system for adaptive hierarchical indexing provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0030] Reference Figure 1 This invention provides an incremental update method for time-series data using an adaptive hierarchical index. This method is suitable for application scenarios involving continuous writing of massive amounts of time-series data and effectively solves the technical problem of system performance fluctuations caused by frequent rebuilding of traditional indexes. In one embodiment of this invention, the method includes the following steps:

[0031] Step S1: Construct a three-level hierarchical index structure that includes a hot layer index, a warm layer index, and a cold layer index.

[0032] In the technical solution of this invention, the three-level hierarchical index structure is the basic architecture of the entire incremental update mechanism. Unlike existing technologies that hierarchically divide data based on index precision, this invention divides data based on the access frequency characteristics, dynamically distributing time-series data into different index levels according to its access frequency and time freshness, thereby achieving synergistic optimization of storage cost and access performance.

[0033] Specifically, the hot layer index uses a memory skip list data structure to handle real-time written time-series data. A memory skip list is an ordered data structure based on linked lists, achieving an average search time complexity of O(log n) through multi-level indexing while maintaining an insertion time complexity of O(1). Preferably, the memory skip list in this embodiment uses a 32-level structure, with each node promoted upwards with a probability p=0.25. This configuration reduces memory usage while ensuring search efficiency. The hot layer index maintains a fine-grained timestamp index, with an index granularity configurable from 1ms to 100ms, defaulting to 10ms, supporting millisecond-level precision time range queries. In one embodiment, the capacity limit of the hot layer index is set to 20% to 40% of the system's available memory, preferably 30%. When the amount of data in the hot layer exceeds the capacity limit, data is triggered to sink to the warm layer.

[0034] The warm stratum index uses an LSM tree structure to store compressed index blocks of recent data. An LSM tree is an index structure optimized for write-intensive scenarios, improving write performance by converting random writes to sequential writes. In this embodiment, the LSM tree employs a hierarchical design, containing four levels from L0 to L3, with a size ratio of 10:1 between levels. Level L0 directly receives data from the hot stratum and uses unordered storage to ensure fast writes; levels L1 to L3 use ordered storage and perform merging and sorting operations in the background. Preferably, the size of each SSTable file in the warm stratum index is controlled between 64MB and 256MB, with a default of 128MB, to balance the number of files and single-file query efficiency. In one embodiment, the warm stratum index also uses a Bloom filter for fast existence detection, with a false positive rate of 1%, effectively reducing invalid disk reads.

[0035] The cold layer index uses segmented B+ trees to manage historical archived data. Unlike traditional single B+ trees, this invention employs a segmentation strategy to divide historical data into multiple independent B+ tree segments according to time intervals. Preferably, the time span covered by each B+ tree segment can be configured to 1 day, 7 days, or 30 days, with a default of 7 days. The technical advantages of this segmentation strategy are as follows: First, newly added time-series data is directly routed to the corresponding B+ tree segment for incremental insertion based on its timestamp, without traversing or rebuilding the entire index structure. When the amount of data in a leaf node within a B+ tree segment exceeds the node capacity limit, a node split operation is triggered to maintain index balance, achieving high efficiency in incremental writing. Second, queries targeting specific time ranges can quickly locate relevant B+ tree segments, reducing the scanning of irrelevant data. Third, for scenarios requiring searches based on data object identifiers, this invention maintains an auxiliary index mapping table of object identifiers in the metadata of each B+ tree segment. Through this mapping table, object-based query requests can be located to a set of B+ tree segments containing the target object data, thus balancing the efficiency of time-range queries and object-dimensional queries. In one embodiment of the present invention, the node size of the segmented B+ tree is configured to be 4KB to 16KB, with a default of 8KB, to align with the page size of the underlying storage system in order to improve I / O efficiency. The fan-out coefficient of the segmented B+ tree is configured to be 100 to 500, with a default of 200, so that the height of the tree is controlled at 3 to 4 levels, ensuring stable query performance of O(log n).

[0036] In a three-level hierarchical index structure, the storage location of data is determined by its popularity value. This invention introduces a time-decay-based data popularity evaluation algorithm, the calculation formula of which is as follows: ,in: Represents data block The popularity value is a dimensionless numerical value, ranging from [0,1]. The larger the value, the more popular the data is, and it should be stored at a higher index level. This represents the current system timestamp, in seconds. Represents data block The last access timestamp is in seconds. It should be noted that for newly written data blocks that have not yet been queried and accessed, their last access timestamp is initialized to the write timestamp of the data block. That is, the last access timestamp is set to be equal to the time when the data was written to the system, so that the new data has a high initial hotness value and ensures that the newly written data is preferentially stored in the hot layer index to meet the typical needs of high-frequency access to the latest data. This represents the time decay constant, used to control the rate at which the popularity decreases over time. The unit is seconds (s), and the value ranges from 3600s to 86400s, with a default value of 7200s. This parameter is selected based on the fact that most time series data are accessed frequently for recent data and occasionally for historical data. A decay constant of 7200s can reduce the popularity of data from 2 hours ago to about 37%, and the popularity of data from 24 hours ago to about 0.07%. Represents data block The frequency of accesses within the statistical period, expressed in times per second; This represents the maximum access frequency of all data blocks in the system, measured in times per second, and is used for normalization. This represents the weighting coefficient of the time decay factor, with a value ranging from 0.3 to 0.7 and a default value of 0.5. This represents the weighting coefficient of the access frequency factor, with a value ranging from 0.3 to 0.7 and a default value of 0.5. .

[0037] Based on the calculated heat values, the system allocates data blocks to the corresponding index levels. In one embodiment of the present invention, a heat layer heat threshold is set. The thermal threshold is 0.7. If the value is 0.3, then the stratification rule is: when At that time, the data block is stored in the hot layer index; when At that time, the data block is stored in the temperature stratum index; when At this time, data blocks are stored in the cold index. This hot-based dynamic tiering mechanism allows frequently accessed data to automatically remain in the high-performance index tier, while historical data that is accessed infrequently is gradually moved down to a lower-cost storage tier.

[0038] Preferably, this invention also introduces a sliding window mechanism for heat statistics, with the statistical period configurable from 1 minute to 60 minutes, and a default of 5 minutes. At the end of each statistical period, the system recalculates the heat values ​​of all data blocks and triggers necessary inter-layer data migration based on changes in the heat values. This periodic heat reassessment mechanism enables the index structure to dynamically adapt to changes in workload.

[0039] In the aforementioned three-level hierarchical index structure, when the continuous writing of new time-series data causes the data volume of the index nodes to exceed the preset capacity, each level of the index maintains the balance of the index structure and query efficiency through corresponding node splitting mechanisms. This invention defines differentiated node splitting trigger conditions and splitting rules for each of the three-level index structures.

[0040] For the memory skip list of the hot layer index, its node splitting mechanism is manifested as dynamic expansion of the skip list hierarchy: when a new data point is inserted into the skip list, the system decides whether the node should be promoted to the next higher level with a probability p=0.25. If promoted, a new index node is created at the higher level and linked to the lower level data node through pointers. This probabilistic promotion mechanism allows the skip list to automatically maintain O(log n) search performance during continuous insertion without explicit splitting operations. When the total amount of data in the hot layer memory skip list exceeds the hot layer capacity limit, no internal splitting of the skip list is performed; instead, a data sinking operation from the hot layer to the warm layer is triggered.

[0041] For LSM trees with stratified indexes, the node splitting mechanism manifests at two levels: memtable freezing and switching, and SSTable file splitting and merging. At the memtable level, when the amount of data in an active memtable reaches a preset capacity threshold (default 64MB), the system freezes the current memtable as an immutable memtable and creates a new blank active memtable to continue receiving writes. The frozen memtable is then flushed to the L0 level SSTable file. This process achieves automatic splitting and switching of the write buffer. At the SSTable level, when the total size of SSTable files at a certain level exceeds the capacity threshold of that level, a compaction operation is triggered: the system selects SSTable files with overlapping key ranges with the next level, merges and sorts them, generates new ordered SSTable files, and writes them to the next level. If the size of a single generated SSTable file exceeds the file size limit (default 128MB), it is split into multiple SSTable files of uniform size.

[0042] For segmented B+ trees in cold layer indexes, the node splitting mechanism includes two levels: leaf node splitting and internal node splitting. Regarding leaf node splitting, when the number of data records stored in a leaf node reaches the leaf node's capacity limit (default is the node size divided by the size of a single record, i.e., 8KB divided by approximately 40 bytes equals approximately 200 records), a leaf node split is triggered: the system creates a new leaf node, using the key value at the median position after sorting in the original leaf node as the split point. Data to the left of the split point remains in the original node, while data to the right of the split point is migrated to the new node. Simultaneously, the doubly linked list pointers between leaf nodes are updated to maintain ordered traversal capability. The key value range and pointer information of the new node are inserted upwards into the parent node. Regarding internal node splitting, when the number of child pointers of a parent node exceeds the maximum fan-out coefficient (default 200) due to receiving new key values ​​generated by child node splitting, an internal node split is triggered: the system divides the internal node in two using the middle key value as the split point. The left half remains in the original node, and the right half is migrated to the newly created internal node. The split point key value is inserted upwards into the higher-level parent node. If the root node splits, a new root node is created, increasing the tree height by one level. The above splitting process is executed through the copy-on-write mechanism in the lock-free concurrency strategy described in step S3. After the splitting operation is completed on the replica, the node pointer is switched through a CAS atomic operation to ensure that the splitting process does not block the foreground read and write operations.

[0043] Step S2: Configure the incremental merge scheduler to calculate the merge trigger conditions based on the data volume, query frequency, and write rate of each layer.

[0044] The incremental merge scheduler is the core component of this invention for implementing adaptive index updates. Its main responsibility is to dynamically decide when to trigger inter-layer data sinking and index merging operations. Unlike the passive update mechanism of existing technologies based on user query feedback, the incremental merge scheduler of this invention adopts an active perception strategy, comprehensively considering three dimensions of factors: data volume, query popularity, and write rate, to achieve intelligent scheduling of merge operations.

[0045] In one embodiment of the present invention, the incremental merge scheduler calculates the merge trigger score for each index layer using a three-factor combined incremental merge triggering algorithm, as shown in the following formula: ,in: This represents the merge trigger score of the k-th level index. It is a dimensionless value with a range of [0,1]. The larger the value, the more likely that the level needs to perform a merge operation. This represents the current data volume of the k-th level index, in MB. This represents the maximum capacity threshold for the k-th level index, in MB. The default value for the hot level is 30% of the available memory, the default value for the warm level is 128GB, and the default value for the cold level is 1TB. This represents the average heat value of all data blocks in the k-th level index, with a value range of [0, 1]. This item uses... This is because the lower the average popularity of a level, the more necessary it is to sink or archive data. This represents the current system write speed, measured in MB / s, obtained through a sliding window. This represents the baseline write speed in MB / s, with a default value of 100 MB / s. This parameter can be adjusted based on the system's hardware configuration and business characteristics. , , These represent the weighting coefficients of the data volume factor, the heat factor, and the write rate factor, respectively, satisfying... The default values ​​are 0.4, 0.3, and 0.3, respectively. The basis for this set of weight configurations is as follows: the data volume factor directly affects the storage pressure and query performance of the index and should be given a higher weight; the heat factor reflects the data access pattern and is a key indicator for determining the data ownership level; the write rate factor reflects the real-time load of the system and is used to trigger the merge operation in time under high write pressure to avoid write blocking.

[0046] When merging a certain level of index triggers a score Exceeding the merge trigger threshold At that time, the incremental merge scheduler initiates the merge operation for this layer. In one embodiment of the present invention, the merge trigger threshold is... The threshold is configured to 0.8, chosen to minimize unnecessary merge operations while ensuring system stability. Preferably, this invention also supports configuring differentiated merge trigger thresholds for different index levels; for example, the hot layer can be configured to 0.75 to implement a more aggressive merge strategy, while the cold layer can be configured to 0.85 to reduce frequent operations on historical data.

[0047] The incremental merge scheduler's merge scheduling period can be configured from 100ms to 10s, with a default of 1s. Within each scheduling period, the scheduler sequentially calculates the merge trigger score for each layer and determines whether a merge operation needs to be initiated. To avoid resource contention caused by simultaneous merging of multiple layers, this invention introduces a merge priority mechanism: when multiple layers simultaneously meet the merge conditions, the merge operation is executed sequentially in the order of hot layers taking precedence over warm layers, and warm layers taking precedence over cold layers, or the priority is sorted according to the merge trigger score.

[0048] In one embodiment of the present invention, the specific process of inter-layer data sinking is as follows: For data sinking from the thermal layer to the temperature layer, the incremental merging scheduler first selects data with a heat value lower than that of the thermal layer. Data blocks are used as a candidate set for sinking, and then written in batches to the L0 layer of the LSM tree of the warm layer in order of oldest to newest timestamps. Finally, the sinking data blocks are deleted from the memory skip list of the hot layer. For data sinking from the warm layer to the cold layer, the incremental merging scheduler first identifies data with a heat value lower than 1 during the compaction process of the LSM tree. The data blocks are then written into the segmented B+ tree of the corresponding time interval in the cold layer, and finally marked as archived in the LSM tree.

[0049] Preferably, the incremental merging scheduler of the present invention also supports a reverse data reversion mechanism. When the popularity value of certain data blocks in the cold or warm layers increases significantly due to sudden access requests, the scheduler can revert these data blocks to a higher index level. The trigger condition for data reversion is that the popularity value of the data block exceeds the popularity threshold of the upper layer for n consecutive statistical periods, with n=3 by default. This lag mechanism can avoid frequent inter-layer migration caused by occasional access.

[0050] Step S3: A lock-free concurrency strategy is adopted during the index merging process.

[0051] Index merging is a resource-intensive task for time-series databases. Traditional locking schemes can block foreground read and write requests, causing significant performance fluctuations. This invention achieves parallel execution of the merging thread and foreground read and write threads through a lock-free concurrency strategy, fundamentally eliminating the impact of the merging operation on foreground performance.

[0052] In one embodiment of the present invention, the core mechanism of the lock-free concurrency strategy includes two aspects: version number mechanism and CAS atomic operation.

[0053] Regarding the version number mechanism, each inode maintains a 64-bit version number field. The high 32 bits of the version number represent the modification sequence number of the inode, and the low 32 bits represent the status flag of the inode. When the merge thread needs to modify an inode, it first creates a copy of the node, performs the modification operation on the copy, and then atomically switches the node pointer from the old version to the new version using a CAS atomic operation. This copy-on-write strategy ensures that the foreground read thread can always access consistent index data and will not read nodes in an intermediate state. Preferably, this invention uses an epoch-based memory reclamation mechanism to manage the memory release of old version nodes: the system maintains a global epoch counter. Each active read thread records its current epoch value when it starts a read operation. After switching node versions, the merge thread adds the old version node to the reclamation queue and marks its epoch value. The node can only be safely reclaimed when the epoch values ​​of all active read threads exceed the epoch value of a certain old version node. In one embodiment of this invention, the epoch advancement period is configured to be from 10ms to 100ms, with a default of 50ms.

[0054] Regarding CAS atomic operations, this invention utilizes the Compare-And-Swap instruction provided by modern CPUs to achieve lock-free index updates. For hot-layer memory skip lists, insertion and deletion operations atomically modify the linked list pointers via CAS; for warm-layer LSM trees, writing to data structures in memory is implemented using lock-free concurrent skip lists, and switching SSTables is completed through atomic pointer swapping; for cold-layer segmented B+ trees, leaf node updates are performed using CAS operations, and the splitting and merging of internal nodes employs an optimistic locking strategy.

[0055] In one embodiment of the present invention, the lock-free concurrency strategy employs differentiated processing methods for different types of concurrency conflict scenarios. For read-write conflicts, the read thread detects conflicts and retryes them by checking version numbers; for write-write conflicts, the write thread detects conflicts by checking the return value of the CAS operation, and retryes them using an exponential backoff strategy when a conflict occurs, with an initial backoff time of 1 μs and a maximum backoff time of 1 ms, doubling each time. Preferably, the present invention also introduces an adaptive conflict detection mechanism: when the system detects that the frequency of write-write conflicts exceeds a preset threshold (default is 1000 times per second), it automatically reduces the parallelism of the merge operation to reduce conflict overhead.

[0056] The technical benefits of lock-free concurrency strategies are reflected in the following aspects: First, foreground read and write operations no longer need to wait for the merge operation to complete, and the 99th percentile value of query latency remains stable; second, the merge operation can continue in the background without affecting the foreground throughput, and the overall processing capacity of the system is fully utilized; finally, by avoiding lock contention and context switching, the CPU utilization of the system is improved, and the scalability on multi-core processors is significantly improved.

[0057] Step S4: Embed a time-series feature summary in each index node of the hot layer index, the warm layer index, and the cold layer index.

[0058] The time-series feature summary is an innovative mechanism introduced in this invention to improve query efficiency. By embedding statistical features of the data within the corresponding time period into the index node, it is possible to quickly determine whether the node contains data that meets the query conditions during the query, thereby achieving efficient query pruning.

[0059] In one embodiment of the present invention, the time series feature summary includes three components: extreme value information, mean information, and trend information, and its data structure is defined as follows: ,in: This represents a summary of the temporal characteristics of index node n; This represents the minimum value of all data points within the time period corresponding to this node, and its unit is consistent with the unit of the original data. This represents the maximum value of all data points within the corresponding time period of this node, and its unit is consistent with the unit of the original data. This represents the arithmetic mean of all data points within the corresponding time period for this node. Its unit is consistent with the unit of the original data, and the calculation formula is: ,in This represents the value of the i-th data point; This indicator represents a trend and is a dimensionless value. Its value ranges from [-1, 1]. Positive values ​​indicate an upward trend, negative values ​​indicate a downward trend, and the larger the absolute value, the more significant the trend. and These represent the start and end timestamps of the corresponding time period for that node, respectively, in milliseconds (ms). This indicates the number of data points contained in the node.

[0060] Trend indicators The calculation uses the normalized form of the linear regression slope: ,in: This represents the slope obtained by performing a linear regression fit on the data points within that time period. Its calculation formula is: , This represents the timestamp of the i-th data point. The average of the timestamps; To prevent small constants from being divided by zero, the value is taken as... ; This is the sensitivity coefficient, used to adjust the sensitivity of the trend indicator to changes in slope. The value range is from 0.01 to 1, with a default value of 0.1. The selection of this parameter is based on mapping most slope values ​​within the normal range to the interval [-0.9, 0.9], while retaining a certain dynamic range to represent extreme trends. The function is used to compress the absolute value of the slope to the [0,1] interval, avoiding the excessive influence of extreme values ​​on trend indicators.

[0061] The update strategy for time-series feature summaries varies depending on the index level. For hot-layer indexes, due to frequent data changes, an incremental update strategy is adopted: each time new data is inserted, the minimum, maximum, and mean values ​​are updated using O(1) incremental calculations, while trend indicators are recalculated every fixed number of data points (default 100). For warm-layer and cold-layer indexes, due to relatively stable data, a batch update strategy is adopted: after data sinking or compaction operations are completed, the feature summaries of the entire node are recalculated all at once.

[0062] Preferably, the present invention also supports hierarchical temporal feature summaries. The feature summary of a parent node can be aggregated and calculated using the feature summaries of its child nodes: the parent node's... Get all child nodes The minimum value, Get all child nodes The maximum value, Calculated by weighted average (weights of each child node). value), The summary is calculated using a weighted average and then smoothed. This hierarchical summary structure supports multi-granularity query pruning, allowing for refinement from the root node layer by layer to quickly locate the leaf node containing the target data.

[0063] Step S5: Upon receiving a query request, perform query pruning to skip index segments that are irrelevant to the query conditions.

[0064] Query pruning is a key technique for improving query efficiency by leveraging temporal feature summaries. When the system receives a query request, the query optimizer first parses the query conditions, then matches the query conditions with the temporal feature summaries of each index node. Nodes that are determined not to contain data that meets the conditions are skipped to avoid unnecessary data access.

[0065] In one embodiment of the present invention, the supported query condition types include: value range query (e.g., querying data with values ​​in the range of [100, 200]), time range query (e.g., querying data within the most recent hour), trend query (e.g., querying periods with an upward trend), and combined queries of the above conditions.

[0066] For range queries, the core logic of pruning is: if the range of the queried value is within the range... The range of values ​​of the index node If there is no intersection, the node can be safely skipped. Specifically, when or Nodes are pruned. In one embodiment of the invention, a mean-based probability pruning strategy is also introduced: when the query range intersects with the node value range but the intersection is small, the probability that the node contains the target data is estimated based on the statistical characteristics of the data distribution. Nodes with a probability lower than a threshold (default 0.01) can also be pruned. The probability estimation adopts the normal distribution assumption: ,in: This represents an estimated probability that a node contains the target data. The cumulative distribution function representing the standard normal distribution; This refers to the mean information of the nodes; The standard deviation of the node data can be estimated by... The empirical formula is used for approximate calculation. This approximation is based on the fact that for approximately normally distributed data, about 99.7% of the data points fall within the range of the mean ± 3 standard deviations.

[0067] For trend queries, the pruning logic is: if the query condition requires an upward trend ( However, the node's trend indicator is negative, or the query condition requires a downward trend. However, if a node's trend indicator is positive, then that node can be pruned. Preferably, this invention supports trend matching with tolerance: setting a trend matching threshold. (Default 0.1) A node is only retained for subsequent queries if its trend indicator is consistent with the trend direction of the query condition and its absolute value exceeds the threshold.

[0068] For combined queries, this invention employs a condition-by-condition pruning strategy: first, coarse-grained pruning is performed based on the time range condition, filtering out index layers and segments with mismatched time ranges; then, fine-grained pruning is performed based on the value range condition, filtering out index nodes with mismatched value ranges; finally, final pruning is performed based on the trend condition. This staged pruning strategy can gradually narrow down the candidate set and maximize the pruning effect.

[0069] The effectiveness of query pruning can be quantitatively evaluated using the pruning rate. In one embodiment of this invention, in a typical time-series data query scenario, query pruning based on time-series feature summaries can filter out 60% to 90% of index node accesses, resulting in a 50% to 80% reduction in query latency. The pruning rate is closely related to the selectivity of the query conditions: for highly selective queries (e.g., querying specific outliers), the pruning rate can reach over 90%; for low-selectivity queries (e.g., querying the average of all data), the pruning rate is lower, but such queries can usually be approximated using summary information without accessing the original data.

[0070] Step S6 introduces a pre-write log and checkpoint mechanism to ensure the atomicity of incremental updates and crash recovery capabilities.

[0071] Write-ahead logs (WAL) and checkpointing mechanisms are key components for ensuring data durability and consistency. In the technical solution of this invention, all incremental update operations must be written to the WAL before execution to ensure that committed update operations are not lost even if the system crashes; the checkpointing mechanism periodically persists the index state in memory to the storage medium, shortening the recovery time after a system crash.

[0072] In one embodiment of this invention, the WAL (Write-Ahead Log) is designed with an append-only log structure. Each log record contains the following fields: Log Sequence Number (LSN, a 64-bit unsigned integer, monotonically increasing), Operation Type (INSERT, UPDATE, DELETE, MERGE, etc.), Timestamp (the time the operation occurred), Operation Data (the specific key-value pairs or index modifications), and Checksum (used to detect log corruption). The WAL file is managed in segments, with each segment configured to be between 64MB and 512MB, and a default size of 128MB. When the current segment file is full, it automatically switches to a new segment file. The old segment file can be safely deleted after confirming that all operations within it have been persisted to the index.

[0073] The write process for the pre-write log follows this sequence: First, the operation to be executed is serialized into a log record and a checksum is calculated; then, the log record is appended to the end of the current WAL segment file; next, the fsync or fdatasync system call is called to ensure that the log record has been persisted to the storage medium; finally, the actual index update operation is executed and a success response is returned. This order of writing the log first and then modifying the data ensures that even if a crash occurs during the index update process, the system can recover the unfinished operations based on the records in the WAL.

[0074] Preferably, this invention supports batch writing and group commit optimization for the Write Allocation Array (WAL). When the concurrency of write requests is high, the system merges the log records of multiple requests into a single batch, writes it to the WAL all at once, and calls fsync, thereby reducing the overhead of fsync. The triggering conditions for batch writing include: the batch size reaching a threshold (default 64KB) or the waiting time reaching a threshold (default 1ms). The group commit strategy can improve the write throughput of the WAL by 3 to 5 times.

[0075] The checkpoint mechanism periodically persists the memory state of the hot-layer index and the metadata of each layer of the index to the storage medium. In one embodiment of the present invention, the checkpoint generation cycle... It can be configured from 60s to 300s, with a default of 120s. The selection of this parameter needs to balance recovery time and checkpoint overhead: a shorter cycle can shorten recovery time but increase I / O overhead, while a longer cycle reduces I / O overhead but prolongs recovery time.

[0076] The checkpoint generation process employs a fuzzy checkpoint strategy, allowing new write requests to be received while checkpoints are being generated. The specific process is as follows: First, the current WAL sequence number is recorded as the checkpoint LSN; then, the memory skip list of the hot layer index is serialized to a temporary file as a snapshot; next, the metadata of the warm and cold layer indexes (including SSTable lists, B+ tree segment lists, etc.) is written to the metadata file; finally, the temporary file is atomically renamed to the official checkpoint file, and the checkpoint LSN is recorded in the checkpoint metadata.

[0077] The system recovery process after a crash is as follows: First, read the latest checkpoint file to restore the memory state of the hot layer index and the metadata of each layer index; then, replay the log records in the WAL starting from the checkpoint LSN, re-executing operations that were committed but not persisted before the crash; finally, clean up incomplete temporary files and uncommitted transactions, and the system recovers to a consistent state and begins receiving new requests. In one embodiment of the present invention, under the default configuration (checkpoint period 120s, WAL segment size 128MB), the recovery time after a system crash is typically in the range of 5s to 30s, depending on the number of WAL records that need to be replayed.

[0078] Preferably, the present invention also introduces an incremental checkpointing mechanism, which only persists the index portion that has changed since the last checkpoint, further reducing the I / O overhead of checkpoints. Incremental checkpointing is implemented by maintaining a dirty page bitmap of the index pages: each index page is associated with a dirty flag, which is set when a page is modified; when a checkpoint is generated, only the dirty page is written out and its dirty flag is cleared. Incremental checkpointing can reduce the average I / O amount of checkpoints by 60% to 80%.

[0079] To more clearly illustrate the complete execution process of incremental updates in this invention, a specific example of writing new time-series data will be used for detailed explanation below.

[0080] Suppose that in an industrial IoT sensor monitoring scenario, a temperature sensor generates a new temperature sampling data at time T=1709251200 (corresponding to March 1, 2024, 00:00:00 UTC). The data content is: sensor_id=TS-0042, sampling timestamp=1709251200000ms, and temperature value=78.6 degrees Celsius. The incremental update process for this new data is performed according to the following steps:

[0081] The first step is pre-write logging. After receiving the new data, the system first encapsulates it into a WAL log record of type INSERT. This log record includes: Log Sequence Number (LSN) = current maximum LSN + 1, Operation Type = INSERT, Timestamp = 1709251200000, Operation Data = key-value pair (key = TS-0042:1709251200000, value = 78.6), and Checksum = CRC32 value calculated from the above fields. This log record is appended to the end of the current WAL segment file, and persistence to the storage medium is ensured via the fsync system call. Only after the WAL write is successful will the system execute subsequent index update operations.

[0082] The second step is hot-layer index insertion. After a successful WAL write, the system inserts the data into the hot-layer's memory skip list. The insertion process is as follows: using the composite key (sensor_id + timestamp) as the sort key of the skip list, the data is inserted into the ordered position of the bottom linked list of the skip list; the system determines the level promotion of this node with a probability p=0.25. Assuming the random result is promotion to level 2, index nodes are created in levels 1 and 2 of the skip list and linked to the bottom data nodes. The insertion operation modifies the linked list pointers through CAS atomic operations, without the need for locking. Since this data is newly written, its last access timestamp is initialized to the write timestamp 1709251200. According to the heat evaluation algorithm, its heat value is calculated as follows: the time decay factor is exp(0)=1 (because the write time is the same as the current time), the access frequency factor is 0 after normalization (it has not been queried and accessed yet), and the heat value = 0.5 multiplied by 1 plus 0.5 multiplied by 0 equals 0.5. However, considering that the last access timestamp of the new data is equal to the current time, the time decay factor is 1, and the judgment condition that the heat value is greater than the heat layer heat threshold of 0.7 needs to be dynamically adjusted in combination with the access frequency. In the actual system, newly written data is directly stored in the heat layer index by default.

[0083] The third step is updating the time-series feature summary. After data is inserted into the skip list, the system uses an incremental update strategy to update the time-series feature summary of the index node to which the data belongs. Assume that the node currently contains 99 data points, with a minimum value of 72.1, a maximum value of 81.3, and a mean of 76.8. After inserting a new data point 78.6: the minimum value remains unchanged at 72.1 (78.6 is greater than 72.1), the maximum value remains unchanged at 81.3 (78.6 is less than 81.3), the mean is updated to (76.8 multiplied by 99 plus 78.6) divided by 100, which equals 76.818, and the number of data points is updated to 100. Since the number of data points reaches 100 (equal to the default threshold for recalculating trend indicators), the system performs linear regression fitting on the 100 data points of this node to recalculate the trend indicators.

[0084] Step 4: Incremental Merge Scheduler Evaluation. In the next scheduling cycle (default 1 second later), the incremental merge scheduler calculates the merge trigger score for the hot layer. Assuming the current hot layer data volume is 85% of 30% of available memory (i.e., data volume factor is 0.85), the average hotness of the hot layer is 0.65 (i.e., hotness factor is 0.35), and the current write rate is 80MB / s (i.e., write rate factor is 0.8), then the merge trigger score = 0.4 x 0.85 + 0.3 x 0.35 + 0.3 x 0.8 = 0.34 + 0.105 + 0.24 = 0.685. Since 0.685 is less than the merge trigger threshold of 0.8, data migration from the hot layer to the warm layer is not triggered in this scheduling cycle. When subsequent continuous writes cause the merge trigger score to exceed 0.8, the scheduler will select data blocks with hotness values ​​lower than the hot layer hotness threshold and migrate them in batches to the L0 level of the warm layer LSM tree.

[0085] Step 5: Query Verification. When a user initiates a query request "Query data of sensor TS-0042 with a temperature value exceeding 75 degrees Celsius in the last hour", the query optimization module first locates the thermal layer index based on the time range condition (data from the last hour is highly likely still in the thermal layer). Then, it matches the value range condition with the time-series feature summary of the thermal layer index node: if the lower bound of the query (75) is less than the node's maximum value (81.3) and the upper bound of the query (positive infinity) is greater than the node's minimum value (72.1), and the value ranges overlap, the node is retained through pruning. The system enters the node to perform a precise query, locates the data records that meet the conditions, including the newly written value=78.6, and returns the query results.

[0086] The technical solution of this invention achieves incremental updates of time-series data for adaptive hierarchical indexes through the coordinated operation of six steps. Step S1, which constructs a three-level hierarchical index structure, provides the basic architecture for the entire solution. Step S2, with its incremental merging scheduler, implements intelligent inter-layer data migration decisions. Step S3, with its lock-free concurrency strategy, eliminates the impact of merging operations on front-end performance. Step S4, with its time-series feature summary, provides key information for query optimization. Step S5, with its query pruning mechanism, significantly improves query efficiency. Step S6, with its WAL (Write-Ahead Log) and checkpoint mechanism, ensures data persistence and consistency. These six steps form a deeply coupled closed-loop collaboration: the heat calculation result of Step S1 serves as the key input for the merging decision in Step S2; the merging operation in Step S2 triggers the concurrency control in Step S3; after merging in Step S3, the feature summary in Step S4 is updated; the summary information in Step S4 supports the query pruning in Step S5; the query access statistics in Step S5 are fed back to the heat calculation in Step S1; and Step S6, throughout all steps, ensures data consistency. This closed-loop collaborative mechanism enables the system to adaptively adjust the index structure to adapt to changes in workload and maintain stable query latency under continuous high-throughput write conditions.

[0087] Reference Figure 2 This invention also provides an adaptive hierarchical index time-series data incremental update system for executing the adaptive hierarchical index time-series data incremental update method described in the above method embodiments. In one embodiment of this invention, the system includes a hierarchical index construction module, an incremental merging and scheduling module, a concurrency control module, a feature summarization module, a query optimization module, and a persistence guarantee module.

[0088] The hierarchical index building module is used to construct a three-level hierarchical index structure containing a hot layer index, a warm layer index, and a cold layer index. In one embodiment of the present invention, the hierarchical index building module includes a hot layer management submodule, a warm layer management submodule, and a cold layer management submodule. The hot layer management submodule is responsible for managing the creation, maintenance, and destruction of memory skip lists, maintaining fine-grained timestamp indexes, and calculating the heat value of each data block according to the heat evaluation algorithm described in step S1 of the method embodiment. The warm layer management submodule is responsible for managing the L0 to L3 levels of the LSM tree structure, performing memtable flushing operations and background compaction operations, and maintaining Bloom filters to accelerate existence judgment. The cold layer management submodule is responsible for managing the creation and maintenance of segmented B+ trees, dividing B+ tree segments according to time intervals, and supporting whole-segment deletion to achieve efficient historical data cleanup. The hierarchical index building module corresponds to step S1 of the method embodiment, and its specific implementation details can be found in the description of step S1.

[0089] The incremental merge scheduling module calculates merge trigger conditions based on the data volume, query popularity, and write rate of each layer. When the merge trigger conditions are met, it dynamically triggers inter-layer data sinking and index merging. In one embodiment of the present invention, the incremental merge scheduling module includes an indicator collection submodule, a trigger decision submodule, and a merge execution submodule. The indicator collection submodule is responsible for collecting indicators such as the data volume, average popularity, and system write rate of each layer in real time. The collection period is consistent with the scheduling period, which is 1 second by default. The trigger decision submodule is responsible for calculating the merge trigger score of each layer according to the three-factor joint algorithm described in step S2 of the method embodiment, and comparing it with the merge trigger threshold to determine whether to start the merge operation. The merge execution submodule is responsible for executing specific inter-layer data sinking and index merging operations, including data sinking from hot layers to warm layers, compaction within warm layers, and data archiving from warm layers to cold layers. The incremental merge scheduling module corresponds to step S2 of the method embodiment, and its specific implementation details can be found in the description of step S2.

[0090] The concurrency control module employs a lock-free concurrency strategy during index merging, achieving concurrent execution of the merging thread and the foreground read / write thread through a version number mechanism and atomic operations. In one embodiment of the invention, the concurrency control module includes a version management submodule, an atomic operation submodule, and a memory reclamation submodule. The version management submodule maintains the version number of each index node, generating a new version and atomically switching when a node is modified. The atomic operation submodule encapsulates the underlying CAS atomic operation interface, providing lock-free data update capabilities for upper-layer modules. The memory reclamation submodule implements an epoch-based safe memory reclamation mechanism, ensuring that old version nodes are only reclaimed after all references have been released. The concurrency control module corresponds to step S3 in the method embodiment, and its specific implementation details can be found in the description of step S3.

[0091] The feature summarization module is used to embed time-series feature summaries into the index nodes of each layer of the hot layer index, the warm layer index, and the cold layer index. The time-series feature summaries include extreme value information, mean information, and trend information. In one embodiment of the present invention, the feature summarization module includes a summary calculation submodule, a summary storage submodule, and a summary update submodule. The summary calculation submodule is responsible for calculating the time-series feature summaries of each index node according to the formula described in step S4 of the method embodiment, including minimum value, maximum value, mean value, and trend index. The summary storage submodule is responsible for managing the storage of time-series feature summaries and embedding the summary data into the metadata area of ​​the index nodes. The summary update submodule is responsible for updating the time-series feature summaries of the corresponding nodes when data is inserted, deleted, or merged. An incremental update strategy is used for the hot layer, and a batch update strategy is used for the warm and cold layers. The feature summarization module corresponds to step S4 of the method embodiment, and its specific implementation details can be found in the description of step S4.

[0092] The query optimization module is used to match and determine the query conditions with the temporal feature summaries of each index node, and to perform query pruning to skip irrelevant index segments. In one embodiment of the present invention, the query optimization module includes a query parsing submodule, a pruning determination submodule, and an execution routing submodule. The query parsing submodule is responsible for parsing the user's query request and extracting query parameters such as time range conditions, value range conditions, and trend conditions. The pruning determination submodule is responsible for matching the query conditions with the temporal feature summaries of each index node according to the pruning logic described in step S5 of the method embodiment, and determining whether the node can be pruned. The execution routing submodule is responsible for organizing the index nodes that have passed the pruning determination into a query execution plan, querying and merging the results sequentially from hot layers to cold layers. The query optimization module corresponds to step S5 of the method embodiment, and its specific implementation details can be found in the description of step S5.

[0093] The persistence guarantee module ensures the atomicity and crash recovery capabilities of incremental updates through pre-write logs and checkpointing mechanisms. In one embodiment of the present invention, the persistence guarantee module includes a WAL management submodule, a checkpointing submodule, and a recovery submodule. The WAL management submodule is responsible for managing the writing, flushing, and segment file switching of the pre-write log, supporting batch writing and group commit optimizations. The checkpointing submodule is responsible for periodically generating checkpoints, persisting the memory state of hot-layer indexes and the metadata of each layer of indexes, and supporting fuzzy checkpointing and incremental checkpointing strategies. The recovery submodule is responsible for detecting whether crash recovery is needed at system startup, performing checkpoint loading and WAL replay to restore the system to a consistent state. The persistence guarantee module corresponds to step S6 of the method embodiment, and its specific implementation details can be found in the description of step S6.

[0094] In this system embodiment, the collaborative relationship between the six modules is as follows: the hierarchical index building module provides the basic index data structure for other modules; the incremental merge scheduling module makes merge decisions based on the indicator data provided by the hierarchical index building module and calls the concurrency control module to perform lock-free merge operations; the concurrency control module provides security guarantees for concurrent access for the hierarchical index building module and the incremental merge scheduling module; the feature summary module updates the summary data after the hierarchical index building module and the incremental merge scheduling module complete the index update; the query optimization module uses the summary data provided by the feature summary module to perform query pruning and retrieves data from the third-level index managed by the hierarchical index building module; the persistence guarantee module runs through all modules, ensuring the persistence and consistency of all index update operations.

[0095] The adaptive hierarchical index incremental update system for time-series data provided in this invention, through the coordinated operation of six functional modules, can maintain stable query performance in scenarios where massive time-series data is continuously written, effectively solving the technical problem of system performance fluctuations caused by frequent reconstruction of traditional indexes.

[0096] The embodiments of the present invention are not limited to the specific embodiments described above. Those skilled in the art can make various equivalent changes or substitutions based on the technical solutions of the present invention, and all such changes or substitutions should be included within the protection scope of the present invention.

Claims

1. An incremental update method for time-series data using an adaptive hierarchical index, characterized in that, Includes the following steps: Step S1: Construct a three-level hierarchical index structure including a hot layer index, a warm layer index, and a cold layer index. The hot layer index uses a memory skip list to receive real-time writes and maintain a fine-grained timestamp index. The warm layer index uses an LSM tree structure to store compressed index blocks of recent data. The cold layer index uses a segmented B+ tree to manage historical archived data and calculates the heat value of each data block based on a time decay data heat evaluation algorithm to determine its index level. Step S2: Configure the incremental merge scheduler to calculate the merge trigger conditions based on the data volume, query popularity and write rate of each layer. When the merge trigger conditions meet the preset merge trigger threshold, dynamically trigger inter-layer data sinking and index merging. The inter-layer data sinking includes data sinking from hot layers to warm layers and data archiving from warm layers to cold layers. Step S3: During the index merging process, a lock-free concurrency strategy is adopted. The merging thread and the foreground read / write thread are executed concurrently through the version number mechanism and CAS atomic operation, so as to avoid the index merging operation blocking the foreground read / write. Step S4: Embed a time-series feature summary into each index node of the thermal layer index, the warm layer index, and the cold layer index. The time-series feature summary includes the extreme value information, mean information, and trend information of the data within the corresponding time period of the index node. Step S5: Upon receiving a query request, match the query conditions with the time-series feature summaries of each index node, and perform query pruning to skip index segments that are irrelevant to the query conditions. Step S6 introduces a pre-write log and checkpoint mechanism. Incremental update operations are first written to the pre-write log, and checkpoints are generated periodically to ensure the atomicity of incremental updates and the data recovery capability after system crash.

2. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, In step S1, the memory skip list adopts a 32-level structure, with each node being promoted upwards with a probability of 0.25, and the timestamp index granularity is configured to be 1ms to 100ms; the LSM tree structure contains four levels from L0 to L3, with a size ratio of 10:1 between levels, and the size of each SSTable file is configured to be 64MB to 256MB; the time span covered by each B+ tree segment of the segmented B+ tree is configured to be 1 day, 7 days, or 30 days, the node size is configured to be 4KB to 16KB, and the fan-out coefficient is configured to be 100 to 500.

3. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, In step S1, the data popularity value is calculated as follows: a weighted sum of a time decay factor and an access frequency factor is performed, wherein the time decay factor is calculated using an exponential decay function, the time decay constant is configured to be between 3600s and 86400s, and the sum of the weights of the time decay factor and the access frequency factor is 1; when the popularity value of a data block is higher than the hot layer popularity threshold, it is stored in the hot layer index; when the popularity value of a data block is between the warm layer popularity threshold and the hot layer popularity threshold, it is stored in the warm layer index; when the popularity value of a data block is lower than the warm layer popularity threshold, it is stored in the cold layer index. The hot layer popularity threshold is configured to be between 0.6 and 0.8, and the warm layer popularity threshold is configured to be between 0.3 and 0.

5.

4. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, In step S2, the merge trigger condition is calculated as follows: the data volume factor, the heat factor, and the write rate factor are weighted and summed, wherein the data volume factor is the ratio of the current layer's data volume to the maximum capacity threshold of the layer, the heat factor is the difference between 1 and the average heat value of the layer, the write rate factor is the ratio of the current write rate to the baseline write rate and does not exceed 1, the sum of the weights of the three factors is 1, and the merge trigger threshold is configured to be 0.7 to 0.

9.

5. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, In step S3, the version number mechanism uses a 64-bit version number, with the high 32 bits representing the modification sequence number of the index node and the low 32 bits representing the status flag of the index node. When the merging thread modifies the index node, it first creates a copy of the index node and performs the modification operation on the copy. Then, it switches the index node pointer from the old version to the new version through the CAS atomic operation. An epoch-based memory reclamation mechanism is used to manage the memory release of the old version index node, and the epoch advancement period is configured to be 10ms to 100ms.

6. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, Step S3 further includes: using an exponential backoff strategy to handle write-write conflicts, with an initial backoff time of 1μs, a maximum backoff time of 1ms, and the backoff time doubling each time; when the frequency of write-write conflicts exceeds a preset threshold, automatically reducing the parallelism of the index merging operation to reduce conflict overhead.

7. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, In step S4, the data structure of the time-series feature summary includes: minimum value, maximum value, arithmetic mean, trend index, start timestamp, end timestamp, and number of data points within the time period; the trend index is obtained by performing linear regression fitting on the data points within the time period to obtain the slope, and the slope is normalized to the range of -1 to 1 using the hyperbolic tangent function; the time-series feature summary is updated using an incremental update strategy for the hot layer index, and a batch update strategy is used to update the time-series feature summary for the warm layer index and the cold layer index.

8. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, In step S5, the query pruning includes: for value range queries, pruning the node when the value range of the query does not intersect with the value range of the index node; for trend queries, pruning the node when the trend direction required by the query conditions is opposite to the trend direction of the index node; and for combined queries, pruning is performed in stages according to the time range conditions, value range conditions, and trend conditions.

9. The time-series data incremental update method for adaptive hierarchical indexing according to claim 1, characterized in that, In step S6, the pre-write log is managed in segments, with each segment file size configured to be 64MB to 512MB. It supports batch writing and group commit optimization. The triggering conditions for batch writing include the batch size reaching a threshold or the waiting time reaching a threshold. The checkpoint generation cycle is configured to be 60s to 300s. The fuzzy checkpoint strategy allows new write requests to be received while checkpoints are being generated. The incremental checkpoint mechanism only persists the index portion that has changed since the last checkpoint.

10. An adaptive hierarchical indexing system for incremental time-series data updates, used to implement the adaptive hierarchical indexing method for incremental time-series data updates according to any one of claims 1-9, characterized in that, include: The hierarchical index building module is used to build a three-level hierarchical index structure containing hot layer index, warm layer index and cold layer index. The hot layer index adopts a memory skip list, the warm layer index adopts an LSM tree structure, and the cold layer index adopts a segmented B+ tree. The heat value of each data block is calculated based on the time decay data heat evaluation algorithm to determine its index level. The incremental merge scheduling module is used to calculate the merge triggering conditions based on the data volume, query popularity and write rate of each layer. When the merge triggering conditions meet the preset merge triggering threshold, the inter-layer data sinking and index merging are dynamically triggered. The concurrency control module is used to employ a lock-free concurrency strategy during the index merging process, and achieves concurrent execution of the merging thread and the foreground read / write thread through version number mechanism and CAS atomic operation; The feature summary module is used to embed time-series feature summaries into each index node of the thermal layer index, the temperature layer index, and the cold layer index. The time-series feature summaries include extreme value information, mean information, and trend information. The query optimization module is used to match and judge the query conditions with the time-series feature summaries of each index node, and perform query pruning to skip irrelevant index segments; The persistence guarantee module is used to ensure the atomicity of incremental updates and crash recovery capabilities through pre-write logging and checkpointing mechanisms.

Images