Self-adaptive index optimization system based on timing context driving

Abstract

The invention discloses a self-adaptive index optimization system based on timing context driving, and relates to the field of computer databases. The context extraction unit is used for initializing a lock-free circular queue by applying a sliding window accumulation mechanism, calculating feature data in combination with a time sequence data stream and a query log, performing normalization processing on the feature data and constructing a context feature vector; the self-adaptive decision-making unit is used for taking the context feature vector and an algorithm library as input and outputting an optimal algorithm and a corresponding weight through an attention network; and the index reconstruction execution unit is used for executing a dynamic adjustment mechanism of the index structure in a linkage manner according to the output optimal algorithm and the corresponding weight, evaluating and updating the current state based on a dynamic adjustment result, and feeding back an updating result to the context extraction unit. According to the method, high-precision spatial-temporal feature modeling can be realized, the problem of mismatching of a strategy and a load is solved, and index expansion and delay peak are avoided.

Description

Technical Field

[0001] This invention relates to the field of computer databases, and more specifically, to an adaptive index optimization system based on time-series context-driven optimization. Background Technology

[0002] With the rapid development of real-time data systems such as the Internet of Things, industrial monitoring, and financial transactions, time-series data has become one of the fastest-growing and largest data types in modern database systems. This type of data is typically generated in a high-throughput, continuous streaming manner, exhibiting significant time correlation, local stationarity, and context dependence. Application scenarios such as sensor monitoring, equipment status tracking, and market quotation systems all place stringent demands on efficient data writing, low-latency querying, and long-term storage management.

[0003] In time-series database systems, the indexing mechanism is a core component for achieving fast data location and retrieval. Traditional index structures, such as B+ trees, LSM-trees, and hash indexes, are generally designed based on the assumption of static data distribution and rely on fixed key-value sorting or partitioning strategies, making them difficult to adapt to the dynamically evolving access patterns of time-series data. Especially when facing complex scenarios such as periodic fluctuations (e.g., diurnal load changes), sudden events (e.g., equipment failure alarms), or data lifecycle evolution (e.g., cold and hot data migration), traditional indexes expose significant performance bottlenecks and resource waste.

[0004] Existing index optimization techniques have three main limitations: First, the index structure is static and fixed, making it impossible to dynamically adjust according to the time evolution characteristics of data access patterns. This results in hot data not being accelerated in time and cold data continuously occupying index space, causing the dual pressure of increased query latency and increased storage overhead. Second, the tuning process relies heavily on manual experience or offline analysis, lacking millisecond-level response capabilities and making it difficult to meet the adaptive requirements under real-time streaming loads. Third, there is insufficient modeling of time-series context features, failing to effectively integrate multi-dimensional time-series context information such as access frequency fluctuations, data lifecycle stages, and query time windows. This leads to a lack of intelligent decision-making basis for index resource configuration and low overall resource utilization.

[0005] To address these challenges, existing research has attempted to incorporate machine learning or multi-level indexing mechanisms for optimization. For example, patent CN114328519A proposes a machine learning-based index optimization method that improves the query efficiency of primary keys and secondary indexes by building a predictive model. However, this approach fixes the model parameters, making online updates impossible when data distribution shifts, leading to accumulated prediction errors. Furthermore, its feature engineering is limited to the key values ​​themselves, lacking modeling of temporal contexts such as timestamps and access cycles, resulting in weak differentiation between hot and cold data and difficulty in dynamic resource scheduling. In addition, index reconstruction requires offline retraining, limiting system real-time performance and availability. Another approach, such as patent CN117493393A, proposes building an inverted index in memory to support multi-dimensional queries and achieving persistence through WAL logs. However, this method employs a fixed "write-as-you-write" strategy, with index granularity statically determined by "table name + dimension value," failing to perceive the temporal evolution of access patterns, leading to cold dimensions residing in memory for extended periods and significant resource waste. Simultaneously, its batch persistence mechanism is prone to I / O jitter under high write loads, affecting query stability. The algorithm has a single strategy and lacks the ability to adaptively compress and switch indexes for heterogeneous data modalities (such as numerical and textual data).

[0006] Furthermore, patent CN115374240A proposes a multi-level index architecture based on TSM files and TSI indexes, which improves query efficiency through the synergy of timestamp indexes and trie indexes. However, once the index hierarchy is persisted to disk, it cannot be changed, and the index depth and granularity cannot be adjusted according to the dynamic fluctuations of query hotspots. In scenarios with sudden queries or changes in data distribution, it is still necessary to address across multiple levels of the structure, resulting in query latency spikes. At the same time, the independent persistence of multi-level indexes brings significant write amplification problems, and lacks awareness of contextual characteristics such as access frequency and lifecycle, making it difficult to achieve fine-grained hot and cold data separation and resource reclamation.

[0007] In summary, current index optimization techniques generally suffer from problems such as a lack of time-series context awareness, static and unadjustable structure, updates lagging behind data evolution, and inefficient resource utilization. Especially when dealing with time-series data scenarios with strong time correlation and dynamic access patterns, it is difficult to balance query performance, storage efficiency, and system real-time performance.

[0008] There are currently no effective solutions to the problems in the relevant technologies. Summary of the Invention

[0009] To address the problems in related technologies, this invention proposes an adaptive index optimization system based on time-series context driving, in order to overcome the aforementioned technical problems existing in the prior art.

[0010] Therefore, the specific technical solution adopted by the present invention is as follows:

[0011] A temporal context-driven adaptive indexing optimization system, comprising:

[0012] The data access unit is used to build a dataset based on the real-time received time-series data stream and query logs, and to perform sharding and alignment processing on the dataset. Based on the sharding and alignment processing results, it performs preprocessing to generate data blocks and query blocks.

[0013] The context extraction unit is used to initialize a lock-free circular queue using a sliding window accumulation mechanism, calculate feature data by combining time-series data streams and query logs, and normalize the feature data to construct a context feature vector.

[0014] The adaptive decision unit takes the context feature vector and the algorithm library as input, and outputs the optimal algorithm and corresponding weights through the attention network.

[0015] The index reconstruction execution unit is used to execute the dynamic adjustment mechanism of the index structure in conjunction with the output optimal algorithm and corresponding weights, evaluate and update the current state based on the dynamic adjustment results, and feed the update results back to the context extraction unit.

[0016] Furthermore, the data access unit includes:

[0017] The data receiving module is used to receive time-series data streams from sensors, gateways, or message queues in real time using zero-copy technology, collect query logs generated in real time on the user side in parallel, and combine the time-series data streams with the query logs to form a dataset.

[0018] The sharding and alignment module is used to construct a sliding window based on preset time window parameters, use the sliding window to shard the dataset, and synchronously perform alignment processing of the time-series data stream and query log within the sliding window.

[0019] The data block encapsulation module is used to perform preprocessing operations on the dataset after it has been fragmented and aligned, and to encapsulate the preprocessing results into standardized data blocks.

[0020] The query block generation module is used to perform parsing operations in parallel based on the query statements in the same sliding window in the data block, build the corresponding syntax tree, prune the syntax tree, and generate query blocks corresponding to the sliding window based on the statistical query access patterns of the pruned syntax tree.

[0021] The first and restart limiting module is used to load the current index, algorithm library and reinforcement learning hyperparameters from the persistent disk when the first acquisition of time series data stream is triggered or the time series data stream is re-acquired, and to initialize the sliding window to an empty queue.

[0022] Furthermore, the context extraction unit includes:

[0023] The sliding window configuration module is used to configure sliding window parameters using the sliding window accumulation mechanism, and initialize a lock-free circular queue as a storage carrier for time-series data and query logs. The lock-free circular queue includes a data queue and a query queue.

[0024] The queue processing module is used to perform data eviction based on a preset sliding window minimum period parameter during the continuous writing of time-series data and query logs into the lock-free circular queue, and to perform dynamic updates of the lock-free circular queue based on the eviction result.

[0025] The feature calculation module is used to calculate feature data in parallel based on the time-series data and query logs within the sliding window. The feature data includes query frequency, lifecycle, burst coefficient, and data entropy.

[0026] The normalization and feature vector generation module is used to normalize the calculated feature data, construct context feature vectors based on the normalization results, and transmit the context feature vectors to the attention network.

[0027] Furthermore, the queue processing module includes:

[0028] The expired data removal submodule is used to identify and remove expired data from the lock-free circular queue that exceeds the preset minimum period parameter of the sliding window during the continuous writing of time-series data and query logs.

[0029] The queue update module is used to detect whether the current occupied capacity of the lock-free circular queue after removal has reached the preset limit. If it has not reached the preset limit, the newly arrived time-series data and query log increments are written to the lock-free circular queue to perform dynamic updates to the lock-free circular queue.

[0030] Furthermore, the expression for query frequency is:

[0031] ;

[0032] In the formula, f1 represents the query frequency; i represents the index value; W represents the number of sampling points within the sliding window; q i This indicates the total number of queries within the sliding window;

[0033] The expression for lifecycle is:

[0034] ;

[0035] In the formula, f2 represents the life cycle stage; t now Indicates the current time point; T i Indicates the most recent usage time of the current data; ε represents the hyperparameter;

[0036] The expression for the burst coefficient is:

[0037] ;

[0038] In the formula, f3 represents the burst coefficient; σ represents the standard deviation of the number of queries within the sliding window; This represents the output of the sliding window; μ represents the average number of queries within the sliding window.

[0039] The expression for data entropy is:

[0040] ;

[0041] In the formula, f4 represents the data value entropy; v represents a discretized bucket corresponding to each interval; V represents the set of discretized buckets of value within the sliding window; p(v) represents the bucket frequency probability.

[0042] Furthermore, the algorithm library includes: merging algorithms, splitting algorithms, data compression algorithms, differential algorithms, and simplified algorithms.

[0043] Furthermore, the index reconstruction execution unit includes:

[0044] The index reconstruction module is used to extract the current data segment based on the encapsulated data block, and combine the output optimal algorithm with the corresponding weight to execute the dynamic adjustment mechanism of the index structure.

[0045] The evaluation and update module is used to collect query extension and storage increment as key indicators based on the dynamic adjustment results, calculate the corresponding reward value based on the key indicators, update the reinforcement learning parameters according to the reward value, clear expired data synchronously, and feed back to the context extraction unit.

[0046] Furthermore, the index reconstruction module includes:

[0047] The merge algorithm submodule is used to evaluate in parallel whether the sum of the sizes of adjacent data segments is less than a preset minimum threshold and whether the weight of the merge algorithm is greater than the reinforcement learning hyperparameter when the optimal algorithm is the merge algorithm. If both conditions are met, the adjacent data segments are merged into a new data segment, loaded into the system memory for merge sorting, and the new data segment is compressed using the current optimal compression algorithm.

[0048] The splitting algorithm submodule is used to evaluate in parallel whether the temporal key distribution entropy of the current data segment is greater than the maximum splitting threshold and whether the weight of the splitting algorithm is greater than the reinforcement learning hyperparameter when the optimal algorithm is the splitting algorithm. If both conditions are met, the median of the temporal dimension of the current data segment is calculated and the current data segment is split into two new data segments based on the median.

[0049] The compression algorithm submodule is used to evaluate whether the timestamp difference of the current data segment does not exceed the preset inflection point of revenue when the compression algorithm is a data compression algorithm. If the condition is met, the current data segment is compressed using the data compression algorithm.

[0050] The hot and cold storage submodule is used to evaluate the lifecycle and query frequency of the current data segment, determine the storage attributes of the current data segment based on the evaluation results, and perform corresponding hot and cold storage.

[0051] Furthermore, assess the lifecycle and query frequency of the current data segment, determine its storage attributes based on the assessment results, and perform appropriate hot and cold storage operations, including:

[0052] Assess the lifecycle and query frequency of the current data segment. If the lifecycle of the current data segment is less than a preset threshold and the query frequency is higher than a preset threshold, it is determined to be a hot segment. Use zero-copy technology to migrate the current data segment to the system memory.

[0053] If the lifecycle exceeds a preset threshold, it is identified as a cold segment, and the current data segment is migrated to the system's cold storage.

[0054] Furthermore, the formula for calculating the reward value is as follows:

[0055] ;

[0056] In the formula, r t L represents the reward value; λ represents the hyperparameter; L represents the reward value. query Δ indicates query latency; ΔStorage indicates storage increment.

[0057] The beneficial effects of this invention are as follows:

[0058] 1. This invention relies on the temporal context-driven modeling logic to couple RNN local temporal extraction with temporal context attention global weight allocation, enabling the model to learn the long-term mapping of context, action and reward online, accurately fit the three-dimensional surface of compression rate, query latency and resource consumption, achieve high-precision spatiotemporal feature modeling, solve the problem of policy and load mismatch, and avoid index bloat and latency spikes.

[0059] 2. This invention utilizes temporal context-driven prediction capabilities to predict compression gains and resource consumption within future windows in advance through an attention network, proactively triggering Merge, Split, or encoding switching actions. This eliminates the periodic limitations of passive adjustments, saving computation time and enabling the system to handle sudden traffic surges with ease, reducing instantaneous system load and resource fluctuations.

[0060] 3. This invention significantly reduces ineffective adjustment actions through high-precision online prediction driven by temporal context. By combining the reinforcement learning reward function with real-time CPU and memory usage as optimization objectives, the system maintains a low-latency state during operation. Furthermore, the cold data sinking and efficient compression mechanisms further reduce storage space usage, achieving a dual improvement in system stability and reliability, and reducing overall costs. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a block diagram of a temporal context-driven adaptive index optimization system according to an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of the adaptive index optimization framework of the adaptive index optimization system based on time-series context driving according to an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of the context extraction layer of an adaptive index optimization system based on temporal context driving according to an embodiment of the present invention;

[0065] Figure 4 This is a block diagram of the data access unit of an adaptive index optimization system based on time-series context driving according to an embodiment of the present invention;

[0066] Figure 5 This is a block diagram of the context extraction unit of an adaptive index optimization system based on time-series context driving according to an embodiment of the present invention.

[0067] In the picture:

[0068] 1. Data access unit; 2. Context extraction unit; 3. Adaptive decision-making unit; 4. Index reconstruction execution unit. Detailed Implementation

[0069] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention.

[0070] According to an embodiment of the present invention, an adaptive index optimization system based on time-series context driving is provided.

[0071] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, an adaptive indexing optimization system based on time-series context driving includes:

[0072] Data access unit 1 is used to establish a dataset based on the real-time received time-series data stream and query logs, and to perform sharding and alignment processing on the dataset. Based on the sharding and alignment processing results, it performs preprocessing to generate data blocks and query blocks.

[0073] In this optional embodiment, the data access unit 1 includes:

[0074] The data receiving module 101 is used to receive time-series data streams from sensors, gateways, or message queues in real time using zero-copy technology, collect query logs generated in real time on the user side in parallel, and combine the time-series data streams with the query logs to form a dataset.

[0075] The sharding and alignment module 102 is used to construct a sliding window based on preset time window parameters, use the sliding window to shard the dataset, and synchronously perform alignment processing of the time-series data stream and query log within the sliding window.

[0076] The data block encapsulation module 103 is used to perform preprocessing operations on the dataset after it has been fragmented and aligned, and to encapsulate the preprocessing results into standardized data blocks.

[0077] The query block generation module 104 is used to perform parsing operations in parallel based on the query statements in the same sliding window in the data block, construct the corresponding syntax tree, prune the syntax tree, and generate query blocks corresponding to the sliding window based on the statistical query access patterns of the pruned syntax tree.

[0078] The first-time and restart limiting module 105 is used to load the current index, algorithm library and reinforcement learning hyperparameters from the persistent disk when the first acquisition of time series data stream is triggered or the time series data stream is re-acquired, and to initialize the sliding window to an empty queue.

[0079] It should be added that, such as Figure 4As shown, this invention accurately captures the temporal correlation and dynamic change characteristics of time-series data and query logs through sliding window segmentation. Data cleaning effectively removes noise and redundant information to ensure data quality. Based on feature extraction, key effective information such as query frequency and data entropy is mined. Combined with the reasonable construction and scientific division of the context dataset, it provides high-quality, highly relevant standardized input data for subsequent attention network feature learning and reinforcement learning model training. It can also adapt to the dynamic evolution characteristics of time-series data, helping the model to quickly capture the load change pattern. This lays a solid data foundation for real-time optimization decisions of the index structure, thereby improving the adaptive adjustment efficiency and decision accuracy of the entire system.

[0080] Context extraction unit 2 is used to initialize a lock-free circular queue using a sliding window accumulation mechanism, calculate feature data by combining time-series data streams and query logs, and normalize the feature data to construct a context feature vector.

[0081] In this optional embodiment, the context extraction unit 2 includes:

[0082] The sliding window configuration module 201 is used to configure the sliding window parameters using the sliding window accumulation mechanism, and initialize the lock-free circular queue as the storage carrier for time-series data and query logs. The lock-free circular queue includes a data queue and a query queue.

[0083] The queue processing module 202 is used to perform data eviction based on a preset sliding window minimum period parameter during the continuous writing of time-series data and query logs into the lock-free circular queue, and to perform dynamic updates of the lock-free circular queue based on the eviction result.

[0084] In this optional embodiment, the queue processing module 202 includes:

[0085] The expired data removal submodule is used to identify and remove expired data from the lock-free circular queue that exceeds the preset minimum period parameter of the sliding window during the continuous writing of time-series data and query logs.

[0086] The queue update module is used to detect whether the current occupied capacity of the lock-free circular queue after removal has reached the preset limit. If it has not reached the preset limit, the newly arrived time-series data and query log increments are written to the lock-free circular queue to perform dynamic updates to the lock-free circular queue.

[0087] The feature calculation module is used to calculate feature data in parallel based on the time-series data and query logs within the sliding window. The feature data includes query frequency, lifecycle, burst coefficient, and data entropy.

[0088] In this optional embodiment, the expression for query frequency is:

[0089] ;

[0090] In the formula, f1 represents the query frequency; i represents the index value; W represents the number of sampling points within the sliding window; q i This indicates the total number of queries within the sliding window;

[0091] The expression for lifecycle is:

[0092] ;

[0093] In the formula, f2 represents the life cycle stage; t now Indicates the current time point; T i Indicates the most recent usage time of the current data; ε represents the hyperparameter;

[0094] The expression for the burst coefficient is:

[0095] ;

[0096] In the formula, f3 represents the burst coefficient; σ represents the standard deviation of the number of queries within the sliding window; This represents the output of the sliding window; μ represents the average number of queries within the sliding window.

[0097] The expression for data entropy is:

[0098] ;

[0099] In the formula, f4 represents the data value entropy; v represents a discretized bucket corresponding to each interval; V represents the set of discretized buckets of value within the sliding window; p(v) represents the bucket frequency probability.

[0100] The normalization and feature vector generation module is used to normalize the calculated feature data, construct context feature vectors based on the normalization results, and transmit the context feature vectors to the attention network.

[0101] It should be further explained that the normalization process for the calculated feature data, the construction of a context feature vector based on the normalization result, and the transmission of the context feature vector to the attention network specifically include:

[0102] 1. Construct a graph structure based on the relationship between time-series data and devices (such as data exchange frequency, device dependencies, etc.). In this graph structure, each node represents a device or data segment, while the edges represent the temporal or logical relationships between these nodes. This structure is used to capture the correlation between devices and time-series data.

[0103] 2. Use a spatiotemporal graph convolutional network to perform computational modeling on the feature data before normalization, including extracting spatial features (i.e., the relationship between devices) through graph convolution operations and extracting temporal features (i.e., the change of data over time) through temporal convolution operations, in order to capture global temporal changes and device interaction information.

[0104] 3. When the system detects a specific external event (such as a high temperature alarm), it will activate the corresponding event trigger. This trigger dynamically adjusts the feature data that has been calculated by the spatiotemporal graph convolutional network but has not yet been normalized, based on changes in the environment or system state. Specifically, when an event such as a high temperature alarm occurs, the system will generate a new event embedding (such as a scalar or vector representation of high temperature) and integrate it into the existing feature data.

[0105] 4. Normalize the feature data that incorporates event information. Based on the result of this normalization, construct a context feature vector and transmit it to the attention network for further analysis and decision-making. This ensures that the system can not only respond to the current data pattern but also adapt to unexpected situations and influence future decisions.

[0106] like Figure 5 As shown, this invention configures parameters and initializes a lock-free circular queue containing a data queue and a query queue through a sliding window accumulation mechanism. During the continuous writing of time-series data and query logs, the queue processing module performs data eviction and dynamically updates the queue according to the preset minimum sliding window period. The two work together to achieve a lock-free design, reduce concurrent contention, improve data read / write and processing efficiency, and filter out expired data in a timely manner to ensure the timeliness of time-series data and query logs. This optimizes storage resource usage, avoids data redundancy, and provides accurate and efficient data support for subsequent context extraction and model decision-making, thereby improving the operational stability and response speed of the entire time-series context-driven adaptive index optimization system.

[0107] The adaptive decision unit 3 takes the context feature vector and the algorithm library as input and outputs the optimal algorithm and corresponding weights through the attention network.

[0108] In this optional embodiment, the algorithm library includes: merging algorithm, splitting algorithm, data compression algorithm, differential algorithm, and simplified algorithm.

[0109] The index reconstruction execution unit 4 is used to execute the dynamic adjustment mechanism of the index structure in conjunction with the output optimal algorithm and corresponding weight, evaluate and update the current state based on the dynamic adjustment result, and feed the update result back to the context extraction unit.

[0110] In this optional embodiment, the index reconstruction execution unit 4 includes:

[0111] The index reconstruction module is used to extract the current data segment based on the encapsulated data block, and combine the output optimal algorithm with the corresponding weight to execute the dynamic adjustment mechanism of the index structure.

[0112] In this optional embodiment, the index reconstruction module includes:

[0113] The merge algorithm submodule is used to evaluate in parallel whether the sum of the sizes of adjacent data segments is less than a preset minimum threshold and whether the weight of the merge algorithm is greater than the reinforcement learning hyperparameter when the optimal algorithm is the merge algorithm. If both conditions are met, the adjacent data segments are merged into a new data segment, loaded into the system memory for merge sorting, and the new data segment is compressed using the current optimal compression algorithm.

[0114] The splitting algorithm submodule is used to evaluate in parallel whether the temporal key distribution entropy of the current data segment is greater than the maximum splitting threshold and whether the weight of the splitting algorithm is greater than the reinforcement learning hyperparameter when the optimal algorithm is the splitting algorithm. If both conditions are met, the median of the temporal dimension of the current data segment is calculated and the current data segment is split into two new data segments based on the median.

[0115] The compression algorithm submodule is used to evaluate whether the timestamp difference of the current data segment does not exceed the preset inflection point of revenue when the compression algorithm is a data compression algorithm. If the condition is met, the current data segment is compressed using the data compression algorithm.

[0116] The hot and cold storage submodule is used to evaluate the lifecycle and query frequency of the current data segment, determine the storage attributes of the current data segment based on the evaluation results, and perform corresponding hot and cold storage.

[0117] In this optional embodiment, the lifecycle and query frequency of the current data segment are evaluated, and the storage attributes of the current data segment are determined based on the evaluation results. The corresponding hot and cold storage operations are then performed, including:

[0118] Assess the lifecycle and query frequency of the current data segment. If the lifecycle of the current data segment is less than a preset threshold and the query frequency is higher than a preset threshold, it is determined to be a hot segment. Use zero-copy technology to migrate the current data segment to the system memory.

[0119] If the lifecycle exceeds a preset threshold, it is identified as a cold segment, and the current data segment is migrated to the system's cold storage.

[0120] The evaluation and update module is used to collect query extension and storage increment as key indicators based on the dynamic adjustment results, calculate the corresponding reward value based on the key indicators, update the reinforcement learning parameters according to the reward value, clear expired data synchronously, and feed back to the context extraction unit.

[0121] It should be further explained that, based on the dynamic adjustment results, the collection and query extension and storage increment are used as key indicators. The corresponding reward values ​​are calculated based on these key indicators, and the reinforcement learning parameters are updated according to the reward values. Expired data is simultaneously cleared, and feedback is sent to the context extraction unit. Specifically, this includes:

[0122] 1. Based on the adjustment results of the dynamic adjustment mechanism, continuously collect two key indicators: query latency and storage increment. These indicators reflect the efficiency of the system in processing query requests and the growth rate of data storage.

[0123] 2. Based on the collected key metrics, the system calculates a reward value, which is used to measure the effectiveness of the current configuration and index structure. If the query latency is low and the storage increment is reasonable, the reward value is high, indicating that the system is performing well. Conversely, if the query latency is high and the storage increment is large, the reward value is low, indicating that optimization is needed.

[0124] 3. Based on the calculated reward value, the system updates its parameters and adjusts its optimization strategy through a reinforcement learning mechanism. This includes, but is not limited to, deciding when to migrate index segments from memory to disk or the cloud, and how to copy index segments across nodes to balance query load. In this process, in order to ensure the consistency of read and write operations during data migration and replication, a lightweight transaction protocol is applied to ensure the atomicity and consistency of data migration.

[0125] 4. After completing the index segment migration and copying operations, the system synchronously performs the clearing of expired data, that is, deletes data segments whose life cycle has ended or are no longer used. This not only frees up valuable storage space, but also reduces unnecessary storage pressure and helps maintain the efficient operation of the system.

[0126] 5. All adjustment results (such as index migration, cross-node replication, etc.) and the latest operating status of the system will be fed back to the context extraction unit. This unit will update the context features according to the latest operating status and provide them to subsequent modules, forming a closed-loop optimization process. This feedback mechanism allows the system to re-evaluate performance after each adjustment cycle and adjust the strategy accordingly, thereby achieving continuous optimization of query load balancing, storage allocation and data migration strategies.

[0127] This invention uses an attention network to extract temporal context features in real time, including query frequency, lifecycle, burst coefficient, and data entropy. Within a millisecond-level closed loop, it dynamically filters and executes a triple index adjustment strategy of granularity, layout, and compression to achieve globally optimal adaptation of the index structure to the operating environment and load characteristics. At the same time, it embeds real-time system indicators such as CPU utilization, memory utilization, and I / O jitter into the design of a reinforcement learning reward function, and continuously optimizes the decision model through online gradient updates, ultimately achieving cross-scenario adaptive effects with high throughput, low latency, and low storage.

[0128] Furthermore, by using real-time system loads such as CPU, memory, and I / O as core feedback signals, the system can dynamically adapt to its operating status and load fluctuations, and precisely adjust index granularity, layout, and compression coding strategies. This avoids resource waste or performance bottlenecks caused by mismatch between fixed index configurations and dynamic loads, optimizes CPU computing power allocation, reduces memory usage and I / O overhead, and ensures low latency and high throughput for index queries. It also improves the stability and resource utilization efficiency of the system under complex load scenarios, achieves a dynamic balance between index performance and system resource consumption, and enhances cross-scenario adaptability.

[0129] In this optional embodiment, the formula for calculating the reward value is:

[0130] ;

[0131] In the formula, r t L represents the reward value; λ represents the hyperparameter; L represents the reward value. query Δ indicates query latency; ΔStorage indicates storage increment.

[0132] It should be further explained that this invention, by capturing key temporal context features such as access frequency, lifecycle stage, and burst coefficient in real time, drives the attention network to dynamically select multi-level index adjustment actions such as merging, splitting, moving up, moving down, and compressing within a millisecond-level closed loop, thereby achieving online joint optimization of index granularity, hierarchy, and compression strategies. At the same time, a reinforcement learning mechanism is introduced to continuously update model parameters, effectively eliminating the index mismatch problem caused by distribution drift. This ensures that query latency remains stable in the low-water mark while maximizing the utilization of memory and disk resources, solving the problem of "temporal drift and heterogeneous modalities" that traditional multi-level indexing schemes struggle to address.

[0133] In a specific embodiment, such as Figure 2 As shown, the adaptive index optimization system based on temporal context driving consists of five layers from top to bottom: data access layer, context extraction layer, adaptive decision layer, index reconstruction layer, and online evaluation. The data access layer receives the raw temporal stream in real time, performs cleaning, alignment, and windowing preprocessing to form data batches in a unified format. The context extraction layer encodes the preprocessed batches into high-dimensional temporal features and inputs them into the context model in chronological order; the model outputs the context feature vector of the current window. The adaptive decision layer uses the context features as input and evaluates the scores of each strategy in the algorithm library in real time through an attention network to select the optimal algorithm A*. The index reconstruction layer performs granular adjustment, layout migration, and compression encoding on the index within milliseconds based on A*, completing a full adaptive closed loop. Specifically, as follows:

[0134] I. Data Access Layer:

[0135] As the bottom layer of the entire adaptive indexing optimization framework, the data access layer undertakes the triple responsibility of "data acquisition, cleaning, and sharding." First, it continuously receives raw time-series data streams (Data-Stream) from sensors, gateways, or message queues in a zero-copy manner, while simultaneously collecting query logs generated in real time by the user side to ensure that data corresponds one-to-one with queries and that no data is lost. Then, the system shards and aligns the data using a 5-second sliding window (W) as the default: preprocessing operations such as denoising, linear imputation of missing values, timestamp normalization, and batch compression are performed on the raw points within each window, and finally, the data is encapsulated into standardized data blocks (D). t ; Parse the query statements within the same window, perform syntax tree pruning and access pattern statistics, and generate query block Q. t Simultaneously, upon initial startup or restart, the data access layer loads the current index I0, the algorithm library Д containing merging / splitting / encoding strategies, and the reinforcement learning reward weight λ and segment merging threshold θ from the persistent disk all at once. merge Segment splitting threshold θ split The key hyperparameters are set, and the context window W is initialized to an empty queue to provide clean, complete, and low-latency input for the subsequent context extraction layer.

[0136] II. Context Extraction Layer:

[0137] The context extraction layer is located at the second level of the adaptive indexing optimization framework, and its task is to extract the original data block D. t ∈R L ×d (L tokens, d-dimensional latent vectors) and query block Q at time t t ∈R M×d The features extracted within the sliding window W (sliding window width, a hyperparameter, in seconds) yield the output. :

[0138] X t =SliceConcat(D t Q t ,W);

[0139] =LayerNorm(X t +P);

[0140] Where, P∈R W×d It is a learnable positional encoding.

[0141] like Figure 3 As shown, the features of the four outputs (query frequency, lifetime, burst coefficient, and data entropy) are compressed into a 4-dimensional, low-redundancy, and high-discrimination context feature vector F. t ∈R 4It is then delivered to the upper-level attention decision network within milliseconds.

[0142] 1. Sliding window accumulation (time and memory dual optimization):

[0143] The window length Δ defaults to 5 seconds and can be hot-modified online via the configuration center; it internally maintains two lock-free circular queues:

[0144] 1) Data queue: Elements are <timestamp, data value>, arranged in a compact 64-bit format;

[0145] 2) Query queue: Elements are <time, q> type ,key range payload len > Record the query type and key field.

[0146] The queue is advanced using head and tail pointers to ensure that |W| is constant, with a time complexity of O(1). Whenever a new tuple arrives, the context extraction layer first performs expulsion according to the minimum period parameter σ, and performs incremental update when the queue is not full.

[0147] 2. Query frequency f1:

[0148] f1=Σ i∈W q i / |W|;(1)

[0149] Molecular q i The total number of queries within the window is f1, and the denominator |W| is the number of sampling points within the window, where f1∈[0,+∞). When f1>10, the system determines it to be a hot window, and the subsequent decision layer will prioritize increasing the index granularity.

[0150] 3. Lifecycle stage f2:

[0151] f2=(t now -min(T i )) / (max(T i )-min(T i )+ε);(2)

[0152] Among them, T i t represents the most recent usage time of the current data, ε is a hyperparameter set to 1µs to prevent division by zero, and t now f2 represents the current time point; f2∈[0,1], where 0 represents newly generated data and 1 represents data nearing obsolescence; this value is used to guide hot and cold stratification: segments with f2 close to 1 will be submerged to the lower medium.

[0153] 4. Burst coefficient f3:

[0154] f3=σ( ) / μ( (3)

[0155] σ and μ are the standard deviation and mean of the number of queries within the window, and the output of the sliding window is... If f3 > 2, the system considers a query spike to have occurred, and the subsequent strategy tends to be split segment + memory shift.

[0156] 5. Data value entropy f4:

[0157] f4=-Σ v∈V p(v)log2p(v); (4)

[0158] v is a discretized bucket corresponding to each interval, V is the set of discretized buckets of value within the window, the bucket width adapts to the global maximum and minimum values, p(v) is the bucket frequency probability; the higher the entropy, the more dispersed the values ​​are, and the greater the compression benefit; when f4 < 3 bits, RLE or Delta-of-Delta encoding can be used directly.

[0159] 6. The four-dimensional features are normalized to the [0,1] interval using Min-Max:

[0160] f i ′=(f i -min i ) / (max i -min i ), i∈{1,2,3,4};(5)

[0161] min i and max i It represents the extreme values ​​of all query frequencies within the window; ultimately, the context vector F is obtained. t .

[0162] F t =[f1′,f2′,f3′,f4′] T (6)

[0163] T is the feature transpose, and f1′, f2′, f3′, f4′ are the outputs of formula (5). Through three steps of sliding window, parallel statistics and normalization, the context extraction layer condenses "time series data + query behavior" into a 4-dimensional context vector F. t It retains four key pieces of information: query frequency, lifecycle stage, burst intensity, and data value entropy. It also delivers the information to the upper-layer attention network with sub-millisecond latency, laying a real-time, accurate, and lightweight decision-making foundation for millisecond-level index adaptation.

[0164] III. Adaptive Decision-Making Layer:

[0165] The core of the adaptive decision layer is a lightweight attention network gθ, whose input is the context feature vector F. t ∈R 4The embedding vector A of the five algorithms in the algorithm library Д={Merge,Split,Gorilla,Delta,Simple8b} iemb ∈R 8 (Pre-trained and fixed), the output is the real-time weight α for each algorithm. i .

[0166] F in formula (6) t and each algorithm embedding vector A iemb Input Attention Network:

[0167] e i =v T tanh(W f F t +W a A iemb +b); (7)

[0168] Among them, W f ∈R 16×4 W a ∈R 16×8 ,b∈R 16 ,v∈R 16 All are learnable weights, tanh introduces nonlinearity, and v achieves compression from 16 to 1; the output e in formula (7) is... i Perform Softmax normalization, where j represents the five algorithms in the library:

[0169] α i =exp(e i ) / Σ 5 j=1 exp(e j (8)

[0170] The optimal algorithm A for hard decision selection * and the weights α of each algorithm * :

[0171] α * =argmax(α i (9)

[0172] A * =argmax(α i (10)

[0173] IV. Index Reconstruction Execution Layer:

[0174] The α output according to formula (9) * The corresponding actions are triggered to complete the triple adjustment of index granularity, layout, and compression; at the same time, real-time performance indicators are sent back to gθ to form a closed loop of "perception-decision-execution-feedback".

[0175] The index reconstruction layer receives the optimal algorithm A * Then, immediately execute the triple linkage of "granularity-layout-compression":

[0176] 1. If A * =Merge and the sum of the sizes of the two segments|S i |+|S j | <N min And α merge >θ merge Then, the two adjacent segments S i S j The physical segments are merged into a new segment S′. During the merge, a merge sort is performed in memory, and the current optimal compression algorithm is used to compress the data again before writing it to disk. Writing the new file first, then performing atomic modifications, and finally deleting the two old segments can reduce metadata overhead; N min This is a system-level constant, with a default value of 8MB; θ merge The threshold is dynamic, output online by gθ, initially 0.65, and automatically drifts based on feedback; α merge Formula (9) is α * The higher the value, the greater the "post-merger benefit".

[0177] 2. If A * =Split and segment entropy(S)>Hmax and α split >θ split Then, divide the segment into two parts based on the median of the timestamps, and calculate the median t of the timestamps within the segment. m According to t≤t m With t>t m Cut into two new segments S1 and S2 r Reduce scan volume; entropy(S) is the entropy of the time bond distribution within the segment, α split Split confidence level. θ split Splitting threshold, H max is the upper limit of the splitting entropy, and both are hyperparameters.

[0178] 3. Compression side: A * =Gorilla and Δ-value≤7-bit, rewrite the entire segment with Gorilla encoding; otherwise, backtrack to Delta-of-Delta+RLE; Δ-value: take the adjacent difference Δt for the timestamp column, take the adjacent difference Δv for the value column, and then calculate the 95th percentile; 7-bit: the "inflection point" of Gorilla, when >7 bits, the XOR compression benefit drops sharply.

[0179] 4. Layout side: Hot segments (f2<0.3 and f1>10) are moved up to the memory Bloom filter through zero copy; cold segments (f2>0.8) are moved down to the SSD cold layer.

[0180] V. Online Assessment and Reinforcement Learning Updates:

[0181] Calculate the query latency L after this adjustment. query With incremental storage ΔStorage; the performance closed-loop process is: execute action → sample L query ΔStorage → Calculate reward r t → Update the set of trainable parameters θ;

[0182] Reward calculation formula:

[0183] r t =λ(1 / L query )-(1-λ)ΔStorage,λ∈[0,1]; (11)

[0184] λ is the hyperparameter 1 / L query The unit is 1 / ms, initially 0.2; ΔStorage is in MB / s, initially 2; the system will multiply ΔStorage by a scaling factor κ=0.01 to ensure both values ​​are in the 0–1 range, facilitating gradient stabilization; using r t Gradient updates are performed on θ, and the loss function is:

[0185] L(θ)=-r t logπ θ (A*|F t (12)

[0186] Clear expired data in the W header and return to Step-1; an online update takes only a few milliseconds, then clear expired data in the window header and immediately return to the context extraction layer, achieving a complete closed loop of "perception-decision-execution-feedback" at the millisecond level.

[0187] Furthermore, this invention enables high-precision spatiotemporal feature modeling, accurately adapts to dynamic loads, supports forward-looking resource scheduling, effectively handles sudden traffic surges, significantly improves system stability and reliability, and substantially reduces overall costs, as detailed below:

[0188] 1. Traditional B+ trees, inverted indexes, and other solutions rely solely on static key-value distribution, failing to capture long-term spatiotemporal dependencies such as query popularity shifts between day and night and sudden changes in data lifecycle. This leads to a mismatch between compression, merging, and splitting strategies and the actual load, resulting in index bloat and query latency spikes. This invention relies on time-series context-driven modeling logic, coupling RNN local time-series extraction with global weight allocation for time-series context attention. This enables the model to learn the long-term mapping of context, actions, and rewards online, accurately fitting the three-dimensional surface of compression rate, query latency, and resource consumption. This achieves high-precision spatiotemporal feature modeling, solving the problem of strategy-load mismatch and effectively avoiding index bloat and latency spikes.

[0189] 2. Existing systems mostly adopt passive adjustment strategies triggered by fixed periods and thresholds. When faced with sudden traffic surges, the index cannot complete the merging, splitting, and other adaptation operations in a timely manner, which can easily cause CPU peaks and I / O jitter. This invention uses the predictive capabilities driven by temporal context to predict the compression benefits and resource consumption in the next 1-2 windows in advance through an attention network, and actively triggers Merge, Split, or encoding switching actions. It gets rid of the periodic limitations of passive adjustment, not only saving computation time, but also being able to cope with sudden traffic surges, significantly reducing the instantaneous load and resource fluctuations of the system.

[0190] 3. Traditional solutions often lead to frequent invalid index rebuilds due to large prediction errors, resulting in significant waste of CPU, memory, and disk bandwidth and increasing the risk of system overload. Furthermore, reliance on manual parameter tuning leads to a linear increase in maintenance costs. This invention significantly reduces invalid adjustment actions through high-precision online prediction driven by time-series context. Combined with a reinforcement learning reward function, it incorporates real-time CPU and memory usage into the optimization objective, ensuring the system maintains low latency throughout operation with zero manual intervention. The addition of cold data sinking and efficient compression mechanisms further reduces storage space usage, ultimately achieving a dual improvement in system stability and reliability, and a significant reduction in overall cost.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive index optimization system based on time-series context-driven approach, characterized in that, The system includes: The data access unit is used to build a dataset based on the real-time received time-series data stream and query logs, and to perform sharding and alignment processing on the dataset. Based on the sharding and alignment processing results, it performs preprocessing to generate data blocks and query blocks. The context extraction unit is used to initialize a lock-free circular queue using a sliding window accumulation mechanism, calculate feature data by combining time-series data streams and query logs, and normalize the feature data to construct a context feature vector. The adaptive decision unit takes the context feature vector and the algorithm library as input, and outputs the optimal algorithm and corresponding weights through the attention network. The index reconstruction execution unit is used to execute the dynamic adjustment mechanism of the index structure in conjunction with the output optimal algorithm and corresponding weights, evaluate and update the current state based on the dynamic adjustment results, and feed the update results back to the context extraction unit.

2. The adaptive index optimization system based on time-series context driving according to claim 1, characterized in that, The data access unit includes: The data receiving module is used to receive time-series data streams from sensors, gateways, or message queues in real time using zero-copy technology, collect query logs generated in real time on the user side in parallel, and combine the time-series data streams with the query logs to form a dataset. The sharding and alignment module is used to construct a sliding window based on preset time window parameters, use the sliding window to shard the dataset, and synchronously perform alignment processing of the time-series data stream and query log within the sliding window. The data block encapsulation module is used to perform preprocessing operations on the dataset after it has been fragmented and aligned, and to encapsulate the preprocessing results into standardized data blocks. The query block generation module is used to perform parsing operations in parallel based on the query statements in the same sliding window in the data block, build the corresponding syntax tree, prune the syntax tree, and generate query blocks corresponding to the sliding window based on the statistical query access patterns of the pruned syntax tree. The first and restart limiting module is used to load the current index, algorithm library and reinforcement learning hyperparameters from the persistent disk when the first acquisition of time series data stream is triggered or the time series data stream is re-acquired, and to initialize the sliding window to an empty queue.

3. The adaptive index optimization system based on time-series context driving according to claim 1, characterized in that, The context extraction unit includes: The sliding window configuration module is used to configure sliding window parameters using a sliding window accumulation mechanism, and initialize a lock-free circular queue as a storage carrier for time-series data and query logs. The lock-free circular queue includes a data queue and a query queue. The queue processing module is used to perform data eviction based on a preset sliding window minimum period parameter during the continuous writing of time-series data and query logs into the lock-free circular queue, and to perform dynamic updates of the lock-free circular queue based on the eviction result. The feature calculation module is used to calculate feature data in parallel based on the time-series data and query logs within the sliding window. The feature data includes query frequency, lifecycle, burst coefficient, and data entropy. The normalization and feature vector generation module is used to normalize the calculated feature data, construct context feature vectors based on the normalization results, and transmit the context feature vectors to the attention network.

4. The adaptive index optimization system based on time-series context driving according to claim 3, characterized in that, The queue processing module includes: The expired data removal submodule is used to identify and remove expired data from the lock-free circular queue that exceeds the preset minimum period parameter of the sliding window during the continuous writing of time-series data and query logs. The queue update module is used to detect whether the current occupied capacity of the lock-free circular queue after removal has reached the preset limit. If it has not reached the preset limit, the newly arrived time-series data and query log increments are written to the lock-free circular queue to perform dynamic updates to the lock-free circular queue.

5. The adaptive index optimization system based on time-series context driving according to claim 4, characterized in that, The expression for the query frequency is: ； In the formula, f1 represents the query frequency; i represents the index value; W represents the number of sampling points within the sliding window; q i This indicates the total number of queries within the sliding window; The expression for lifecycle is: ； In the formula, f2 represents the life cycle stage; t now Indicates the current time point; T i Indicates the most recent usage time of the current data; ε represents the hyperparameter; The expression for the burst coefficient is: ； In the formula, f3 represents the burst coefficient; σ represents the standard deviation of the number of queries within the sliding window; This represents the output of the sliding window; μ represents the average number of queries within the sliding window. The expression for data entropy is: ； In the formula, f4 represents the data value entropy; v represents a discretized bucket corresponding to each interval; V represents the set of discretized buckets of value within the sliding window; p(v) represents the bucket frequency probability.

6. The adaptive index optimization system based on time-series context driving according to claim 1, characterized in that, The algorithm library includes: merging algorithm, splitting algorithm, data compression algorithm, differential algorithm, and simplified algorithm.

7. The adaptive index optimization system based on time-series context driving according to claim 6, characterized in that, The index reconstruction execution unit includes: The index reconstruction module is used to extract the current data segment based on the encapsulated data block, and combine the output optimal algorithm with the corresponding weight to execute the dynamic adjustment mechanism of the index structure. The evaluation and update module is used to collect query extension and storage increment as key indicators based on the dynamic adjustment results, calculate the corresponding reward value based on the key indicators, update the reinforcement learning parameters according to the reward value, clear expired data synchronously, and feed back to the context extraction unit.

8. The adaptive index optimization system based on time-series context driving according to claim 7, characterized in that, The index reconstruction module includes: The merge algorithm submodule is used to evaluate in parallel whether the sum of the sizes of adjacent data segments is less than a preset minimum threshold and whether the weight of the merge algorithm is greater than the reinforcement learning hyperparameter when the optimal algorithm is the merge algorithm. If both conditions are met, the adjacent data segments are merged into a new data segment, loaded into the system memory for merge sorting, and the new data segment is compressed using the current optimal compression algorithm. The splitting algorithm submodule is used to evaluate in parallel whether the temporal key distribution entropy of the current data segment is greater than the maximum splitting threshold and whether the weight of the splitting algorithm is greater than the reinforcement learning hyperparameter when the optimal algorithm is the splitting algorithm. If both conditions are met, the median of the temporal dimension of the current data segment is calculated and the current data segment is split into two new data segments based on the median. The compression algorithm submodule is used to evaluate whether the timestamp difference of the current data segment does not exceed the preset inflection point of revenue when the compression algorithm is a data compression algorithm. If the condition is met, the current data segment is compressed using the data compression algorithm. The hot and cold storage submodule is used to evaluate the lifecycle and query frequency of the current data segment, determine the storage attributes of the current data segment based on the evaluation results, and perform corresponding hot and cold storage.

9. The adaptive index optimization system based on time-series context driving according to claim 8, characterized in that, The process of assessing the lifecycle and query frequency of the current data segment, determining the storage attributes of the current data segment based on the assessment results, and performing corresponding hot and cold storage includes: Assess the lifecycle and query frequency of the current data segment. If the lifecycle of the current data segment is less than a preset threshold and the query frequency is higher than a preset threshold, it is determined to be a hot segment. Use zero-copy technology to migrate the current data segment to the system memory. If the lifecycle exceeds a preset threshold, it is identified as a cold segment, and the current data segment is migrated to the system's cold storage.

10. The adaptive index optimization system based on time-series context driving according to claim 9, characterized in that, The formula for calculating the reward value is: ； In the formula, r t L represents the reward value; λ represents the hyperparameter; L represents the reward value. query Δ indicates query latency; ΔStorage indicates storage increment.

Images