Time sequence AI data management and loading method and system

Abstract

The invention relates to the technical field of artificial intelligence and computer data management, discloses a time sequence AI data management and loading method and system, and aims to solve the problems of low I / O efficiency, memory waste, high access delay and the like caused by time sequence data storage fragmentation, loading staticization, index coarseness and high system coupling degree in the prior art. The method comprises the following steps: carrying out standardization processing and distributed column type storage on received original time sequence data, constructing a three-level index system based on time, equipment and a characteristic mode, extracting time sequence characteristics by utilizing an auto-encoder, and establishing a mode index; according to the scheme, the complex query response speed is remarkably improved, the prefetching hit rate reaches 87%, the I / O efficiency is improved by 3.5 times, the end-to-end delay is controlled within 150 milliseconds, the training throughput and the memory utilization rate are effectively improved, and efficient cooperation of data management and AI calculation is achieved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer data management technology, specifically relating to a time-series AI data management and loading method and system. Background Technology

[0002] With the deep integration of artificial intelligence and big data technologies, time-series data is playing an increasingly important role in key areas such as industrial monitoring, financial analysis, intelligent transportation and the Internet of Things. This type of data is characterized by high frequency, long cycle and continuity, and carries the core information of the dynamic evolution of the system. Its management and processing efficiency directly affects the inference accuracy and response speed of the upper-level intelligent model.

[0003] Among them, time-series data management and loading technology for machine intelligence has become a key infrastructure to support AI model training and inference. This technology aims to achieve efficient storage, fast retrieval and low-latency supply of massive time-series data, and ensure high throughput and low latency collaboration between data flow and computing tasks, so as to meet the stringent requirements of real-time performance and stability for complex AI applications.

[0004] Existing technologies exhibit multiple systemic bottlenecks when processing large-scale time-series AI data: First, data organization generally employs general-purpose files or databases for storage, lacking collaborative optimization for time-series characteristics and AI training access patterns, resulting in fragmented data reading and low I / O efficiency; second, the data loading process is statically fixed, unable to adaptively schedule according to the dynamic needs of model training (such as batch sampling strategies and time window changes), causing wasted memory resources and cumulative loading latency; third, metadata management is coarse-grained, making it difficult to support efficient indexing and subsequence location under complex query conditions, significantly increasing data preprocessing overhead; finally, the system architecture is tightly coupled, with a lack of standardized interfaces and semantic understanding capabilities between the data management module and the AI ​​framework, resulting in high integration complexity and poor scalability. These problems are particularly prominent in typical scenarios such as high concurrency, large models, and real-time inference, severely restricting the overall performance and resource utilization of end-to-end AI systems, urgently requiring a new time-series data management and loading mechanism tailored to the characteristics of AI workloads. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for managing and loading time-series AI data, which can effectively solve the problems in the background technology. Currently, time-series data in artificial intelligence applications faces multiple challenges, including continuously growing data scale, diverse and heterogeneous data sources, high data update frequency, and complex data semantics. This leads to significant bottlenecks in traditional data management architectures in terms of data organization, storage efficiency, access latency, and model training throughput. Existing systems generally adopt static data partitioning strategies and fixed loading processes, which cannot adapt to the dynamic access patterns of time-series data required by AI models. This results in wasted memory resources, uneven I / O load, and increased feature extraction latency, thereby affecting the real-time performance and accuracy of model inference and training. Furthermore, traditional solutions lack systematic modeling of the inherent spatiotemporal correlations of time-series data, making it difficult to support multi-granularity, multi-modal joint analysis and efficient feature pre-fetching. This invention proposes a collaborative management architecture for the entire lifecycle of time-series data, based on the integration of big data management and machine intelligence technologies, for AI workloads. By constructing a dynamically perceptive data hierarchical indexing mechanism, a prediction-based intelligent preloading strategy, and a data compression and decoding pipeline that supports time-series semantic awareness, it achieves synergistic optimization of data management efficiency and AI model performance.

[0006] To achieve the above objectives, the present invention provides the following technical solution: On one hand, a time-series AI data management system, comprising the following components: a data access module, used to receive raw time-series data streams from multiple data sources, the raw time-series data streams containing timestamps, device identifiers, measurement values, and metadata fields, and to perform protocol parsing and format standardization processing on the raw time-series data streams, outputting standardized time-series data packets in a unified format; a data storage module, used to perform distributed persistent storage of the received standardized time-series data packets, the data storage module adopting a columnar storage structure and constructing a two-level index structure based on time intervals and device dimensions, supporting efficient data retrieval based on time ranges and device combinations as query conditions; a data index module, used to establish a dynamically updated multi-level index system for the stored time-series data, the multi-level index system including a basic time index, a device clustering index, and a feature pattern index, wherein the feature pattern index identifies and marks time-series segments with similar fluctuation patterns through a machine learning model, used to accelerate pattern matching queries; data The loading module responds to data requests from AI model training or inference tasks, extracts the required time-series data subset from the data storage module, and loads it into the memory cache of the computing node. This loading module integrates an intelligent prefetching engine, capable of predicting potentially accessed data regions based on historical access patterns and the current task context. The cache management module manages the lifecycle of the time-series data cache in memory, employing a hybrid eviction strategy based on access frequency and timeliness to ensure priority retention of frequently accessed and recently used data. The task scheduling module receives execution instructions from AI tasks, parses their data dependencies, and coordinates the collaborative work of the loading module and computing resources to ensure the timing alignment of data supply and model computation. The model feedback interface receives performance monitoring data generated by the AI ​​model during training or inference, including data loading latency, cache hit rate, and feature extraction time. This interface feeds back the performance monitoring data to the data indexing module and the data loading module to dynamically adjust the index update frequency and prefetching strategy.

[0007] In addition, when performing format standardization processing, the data access module performs linear interpolation to complete missing timestamps and automatically corrects abnormal measurement values ​​based on sliding window statistics, ensuring that the output standardized time-series data packets have a complete time-series structure and data consistency.

[0008] Preferably, when performing distributed persistent storage, the data storage module divides the time-series data into multiple data blocks according to a fixed time window. Each data block is compressed and encoded independently. The compression algorithm adopts a hybrid compression strategy based on differential encoding and dictionary encoding, with a compression ratio of not less than 3:1. It also supports time range querying and aggregation calculation of compressed data blocks without decompressing the full amount of data.

[0009] Furthermore, the feature pattern index construction process in the data index module includes: first, normalizing the time-series data segments to eliminate dimensional differences; then, using an autoencoder model to extract low-dimensional latent feature vectors from the time-series segments; subsequently, performing cluster analysis on the latent feature vectors to generate several feature pattern clusters; finally, labeling each time-series data segment with its corresponding feature pattern cluster number, and establishing a reverse mapping index table from feature pattern clusters to data storage addresses to support fast retrieval based on pattern similarity.

[0010] In addition, the intelligent prefetch engine has a built-in long short-term memory network model to learn the data access sequence patterns of AI tasks at different execution stages. The long short-term memory network model takes the time series of historical data requests as input and outputs the probability distribution of data blocks that may be accessed within the next 1 to 5 time steps. The data loading module loads high-probability data blocks from persistent storage to the memory cache in advance according to the probability distribution, with a prefetch hit rate of no less than 85%.

[0011] Preferably, when executing the hybrid eviction policy, the cache management module maintains two weight parameters for each cached data item: access frequency weight and time decay weight. The access frequency weight increases with each access, while the time decay weight decreases exponentially over time. The cache eviction decision is based on the weighted sum of the two parameters, prioritizing the eviction of data items with the lowest weighted sum. The weight coefficients are dynamically adjusted based on the cache hit rate using an online learning algorithm.

[0012] Furthermore, when coordinating data loading and computing resources, the task scheduling module adopts a priority-based task queue mechanism, assigning higher priority to inference tasks with high real-time requirements to ensure that their data requests are responded to within 100 milliseconds. For batch training tasks, a batch processing mode is adopted to merge data requests from multiple adjacent time windows to reduce I / O overhead.

[0013] In addition, the performance monitoring data received by the model feedback interface is periodically aggregated and input into the index optimizer. The index optimizer evaluates the impact of different index update strategies on the overall system latency based on reinforcement learning algorithms, and autonomously decides the optimal index reconstruction cycle and feature pattern clustering granularity to achieve a dynamic balance between index maintenance cost and query performance.

[0014] On the other hand, a time-series AI data management and loading method includes the following steps: Step S110: Receive raw time-series data streams from sensors, log systems, and external databases through a data access module; perform protocol parsing and format standardization on the raw time-series data streams to generate standardized time-series data packets with a unified format. The standardization process includes timestamp alignment, missing value imputation, and outlier correction. Step S120: Write the standardized time-series data packets into a distributed columnar storage system, segment the data according to time windows, and establish a two-level index structure based on time and device dimensions. Simultaneously, perform hybrid compression encoding on the data blocks to improve storage density. Step S130: Use a machine learning model to perform feature pattern recognition on the stored time-series data, extract time-series segment groups with similar dynamic behaviors, construct a feature pattern index table, and maintain its mapping relationship with the data storage address. Step S140: Respond to the data reading request initiated by the AI ​​model, and parse the request... The time range, device list, and feature type conditions are combined with the basic time index, device clustering index, and feature pattern index to perform joint query planning and generate the optimal data retrieval path; Step S150: Based on the current task type and historical access patterns, the Long Short-Term Memory (LSTM) network model is used to predict the time series data regions that may be accessed subsequently, driving the data loading module to load the predicted data into the memory cache in advance; Step S160: During the data loading process, a hybrid cache eviction strategy based on access frequency and timeliness is implemented to dynamically maintain the validity and hotspot of the time series data in memory, ensuring efficient utilization of cache resources; Step S170: The loaded time series data is supplied to the AI ​​model for feature engineering and model calculation, and data access performance indicators are collected through the model feedback interface; Step S180: Based on the collected performance indicators, the clustering parameters of the feature pattern index, the prediction window length of the prefetch model, and the weight coefficients of the cache eviction strategy are dynamically adjusted to form a closed-loop optimization mechanism.

[0015] Preferably, the missing value imputation in step S110 adopts a linear interpolation algorithm. When the number of consecutive missing points does not exceed 5, linear reconstruction is performed based on the valid data points before and after. When the number of consecutive missing points exceeds 5, a data quality alarm is triggered and the segment is marked as an untrusted region.

[0016] Furthermore, the hybrid compression encoding process in step S120 includes: first, differential encoding is performed on the continuous measurement value sequence of the same device to obtain the incremental sequence; then, a static dictionary is established for the high-frequency values ​​in the incremental sequence, and short codes are used to represent common increments; for abnormal increment values ​​not covered by the dictionary, variable-length encoding is used; finally, the differential code stream and dictionary information are packaged and stored, and the compressed data supports the direct execution of aggregation operations such as summation, mean, and maximum value within the compressed domain.

[0017] Furthermore, in step S130, the autoencoder model uses a sliding window to extract time series segments of length 128 as samples during the training phase. The number of neurons in the input layer is consistent with the sequence length, the hidden layer dimension is set to 32, the loss function is the mean squared error, and after training, some parameters of the encoder are fixed for online extraction of feature vectors of new data segments.

[0018] Preferably, the long short-term memory network model in step S150 contains two hidden layers, each containing 64 memory units. The input sequence is the data request records of the past 20 time steps, and the output is the access probability of each data block in the next 5 time steps. The model uses the latest data for incremental training every 24 hours to maintain prediction accuracy.

[0019] Furthermore, in the hybrid cache eviction policy in step S160, the access frequency weight is initially set to 1, and is multiplied by a growth factor of 1.2 after each access. The time decay weight is initially set to 1, and is multiplied by a decay factor of 0.99 every minute. The final eviction score is calculated by multiplying the two weights. The system performs an eviction assessment every 10 seconds, and evictions 5% of the total cache size each time.

[0020] Furthermore, in the closed-loop optimization mechanism in step S180, the state space of the reinforcement learning agent includes three dimensions: cache hit rate, average loading latency, and index update overhead. The action space includes adjusting the index update cycle, modifying the prefetch window size, and switching the cache strategy mode. The reward function is designed as the reciprocal of the overall latency. The agent learns the optimal control strategy through the Q-learning algorithm.

[0021] Compared with the prior art, the present invention has the following beneficial effects: By constructing a three-level index system that integrates time, device, and feature patterns, the response speed of complex queries has been significantly improved. Compared with the traditional two-level index scheme, the average latency of pattern matching queries has been reduced by 62%. By introducing an intelligent prefetching mechanism based on a long short-term memory network, the prefetching hit rate reached 87%, effectively reducing the idle time of the AI ​​model waiting for data and improving the overall training throughput by 41%. It adopts a hybrid encoding strategy that supports compressed domain aggregation, achieving fast statistical calculations without decompression while maintaining a compression ratio of over 98%, thus improving I / O read efficiency by 3.5 times. A closed-loop optimization mechanism based on model feedback was established, enabling the system to adapt to the data access patterns of different AI tasks. The cache hit rate remained above 90% under dynamic load, and the memory resource utilization was improved by 2.8 times. It achieves deep collaboration between time-series data management and AI computing needs, controlling the end-to-end latency from data access to model supply within 150 milliseconds, meeting the needs of high real-time AI applications. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall technical architecture of a time-series AI data management and loading method and system proposed in this invention. Detailed Implementation

[0023] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0024] Example 1 Please refer to Figure 1 This embodiment takes the training and inference scenario of a city-level road network traffic prediction AI model in an intelligent transportation system as its background, and constructs and deploys a time-series AI data management and loading system. This system needs to process multi-dimensional indicators such as traffic flow, average vehicle speed, and occupancy rate reported every 30 seconds from 50,000 traffic monitoring points throughout the city, forming a high-concurrency time-series data stream of over 4.3 billion records daily. Traditional architectures in this scenario suffer from high data loading latency and low cache hit rate, resulting in model training cycles as long as 14 hours, making it difficult to meet the timeliness requirement of completing nighttime batch training before the morning rush hour. This embodiment, by introducing the method and system described in this invention, achieves end-to-end data supply latency control within 120 milliseconds, and increases model training throughput to 2.4 times that of the original system.

[0025] After system startup, the data access module first establishes a multi-protocol communication channel with the traffic signal control platform, the floating car GPS data acquisition network, and the third-party map API. For Modbus TCP protocol data streams from the signal control platform, the module's built-in protocol parser identifies the message header identifier 0x4856 and extracts the raw data fields containing timestamps (UTC milliseconds), intersection IDs (6-digit hexadecimal codes), lane numbers (1-8), traffic flow (integer), average vehicle speed (floating-point, unit km / h), and device status codes (enumerated values). For floating car data, the gRPC streaming protocol based on HTTP / 2 is used to receive JSON format messages and parse out the vehicle VIN code, positioning time, latitude and longitude coordinates, instantaneous speed, and heading angle. All raw data streams enter the format standardization processing unit, which performs unified time base alignment: converting local timestamps to UTC standard time, and using a linear interpolation algorithm with a basic alignment granularity of 30 seconds to fill in missing values ​​caused by network jitter. When a detection point fails to report data for two consecutive cycles, the system performs linear interpolation based on the data from the preceding and following cycles. If more than 5 consecutive cycles (i.e., 2.5 minutes) are missing, the data segment for that device ID within the corresponding time period is marked as an "unreliable area," triggering an operation and maintenance alarm. Simultaneously, the quality flag "Q=0" is written to the metadata field. Outlier correction uses a sliding window statistical method. A sliding window of length 20 is maintained for the vehicle speed sequence at each detection point. The mean μ and standard deviation σ of the data within the window are calculated. If newly arrived data exceeds the interval […], […]. -3 μ+3 If the value is not found, it is considered abnormal, and the window value is used to replace the original value.

[0026] After standardization, a standardized time-series data packet with a unified format is generated. Its data structure is defined as a fixed-length binary structure: the first 8 bytes are a 64-bit Unix timestamp, the next 4 bytes are a 32-bit device identifier, the following 16 bytes are four floating-point measurement values ​​(arranged in the order of traffic flow, vehicle speed, occupancy rate, and queue length), and the last 8 bytes are metadata fields, including a 2-byte quality identifier, a 2-byte data source encoding, and a 4-byte checksum. This structure is then serialized and output to the data storage module.

[0027] The data storage module adopts a columnar storage format based on Apache Parquet, deployed on a 12-node HDFS cluster, with a total available storage capacity of 1.2PB. All written data is divided into independent data blocks every 15 minutes according to UTC time. Each data block corresponds to a Parquet file, named "traffic_data_yyyymmddHHMM.parquet". The storage path is organized according to the partition structure " / data / year=YYYY / month=MM / day=DD / hour=HH / ". Inside the file, each measurement value field is stored as an independent column, and block-level compression is performed using the Snappy compression algorithm. Based on this, a two-level index structure is constructed: the first-level index is a time index, which uses a B+ tree structure to maintain the mapping relationship between the start time of all data blocks and the file storage path, supporting time range queries with O(log n) time complexity; the second-level index is a device clustering index, which uses the DBSCAN clustering algorithm to divide the detection points into 200 spatial clusters based on their geographical coordinates (latitude and longitude). Each cluster corresponds to a device group, and the index table records the cluster number to which each device ID belongs. These two-level indexes are stored in a Redis cluster and support millisecond-level response times.

[0028] Simultaneously with data writing, the data indexing module initiates the feature pattern index construction process. First, it extracts traffic flow time-series data from the storage system in daily batches. Then, it normalizes the 15-minute granular data from each detection point over the past 7 days using the Z-score normalization formula: in This is the 7-day average of the sequence. The standard deviation was used to eliminate the dimensionality impact of traffic volume differences across different road segments. Subsequently, time segments were extracted as training samples using a window of 128 time points (32 hours) and a sliding step of 16 points (4 hours). The autoencoder model consisted of a fully connected neural network with 128-dimensional input and output layers. The hidden layer structure was 128→64→32→64→128, using ReLU as the activation function, mean squared error (MSE) as the loss function, Adam as the optimizer, and a learning rate of 0.001. The model converged after 200 epochs of training on a GPU cluster. The encoder parameters (128→64→32) were fixed for online feature vector extraction. For each normalized 128-point segment, a 32-dimensional low-dimensional latent feature vector was obtained through the encoder. Subsequently, K-means clustering was performed on all feature vectors. The initial number of clusters, k=50, was determined to be optimal at 42 using the elbow rule, generating 42 feature pattern clusters. Each cluster represents a typical traffic flow fluctuation pattern, such as "single-peak morning rush hour," "double-peak commuting," "stable nighttime," and "holiday off-peak." Finally, each time-series data segment was labeled with its cluster number (1-42), and a reverse mapping index table was established to record the storage addresses of all data blocks corresponding to each cluster number, supporting fast retrieval based on pattern similarity.

[0029] When the AI ​​task scheduling system submits a training task, the task scheduling module receives a task description file, which includes parameters such as model type (LSTM neural network), training period (all day yesterday), input features (traffic flow, vehicle speed), target variable (congestion index for the next 15 minutes), and batch size (64 samples). The module analyzes its data dependencies, determining that the time range to be loaded is "from 00:00:00 to 23:59:59 yesterday," the equipment range is "all detection points in the central urban area" (a total of 12,000), and the feature type is "traffic flow and vehicle speed." Task priority is determined based on type: if it is a batch training task before the morning rush hour, it is assigned priority 3; if it is a real-time inference task after a sudden event, it is assigned priority 1 (the highest). This example is a batch training task, which enters the priority queue to await scheduling.

[0030] Upon receiving the scheduling instruction, the data loading module initiates a joint query planning process. First, it locates the 96 15-minute data blocks (4 blocks per hour) to be loaded using the basic time index. Second, it filters out 12,000 detection points belonging to the "Central Urban Area" device group using the device clustering index, generating a device ID list. Finally, combining the feature pattern index, if the task configuration file contains the condition "prioritize loading commuter mode road segments," it further filters out detection point data belonging to the "bi-peak commuter type" (cluster numbers 2, 5, 8, etc.). The query planning engine integrates the three types of index information to generate the optimal data retrieval path, transforming the original full table scan request into a precise file list and column pruning instructions, reading only the traffic flow and speed columns of the target device, avoiding I / O overhead from irrelevant columns.

[0031] Before the actual data loading, the intelligent prefetching engine initiates the prediction process. This engine incorporates a two-layer stacked Long Short-Term Memory (LSTM) network model, with each layer containing 64 memory units. The forget gate, input gate, and output gate all use sigmoid activation, and the cell state update uses the tanh function. The model input is the historical data request sequence for this AI task over the past 20 time steps (i.e., 5 hours), with each time step recording the requested data block ID and access frequency. The input sequence is mapped to a 64-dimensional vector through an embedding layer and then fed into the LSTM network. The output layer is a softmax classifier, outputting the probability distribution of each data block being accessed within the next 5 time steps (i.e., the next 75 minutes). The model is incrementally trained every 24 hours using new request data from the past 24 hours, employing mini-batch stochastic gradient descent (batch size=32), updating only the parameters of the last two layers to maintain stability. Based on the current prediction results, the system determines that the access probability of the "morning peak main road" data block in the first 15-minute window exceeds 0.9. Therefore, it drives the data loading module to load these high-probability data blocks from HDFS persistent storage to the local SSD cache in advance, and further preload them to the memory cache area of ​​the computing node to realize data pre-loading.

[0032] The data loading process is conducted in coordination with the cache management module. Each cached data item is a subset of data blocks with a 15-minute granularity, containing specified fields for a specified device. The module maintains two dynamic weight parameters for each cached item: access frequency weight W_f and time decay weight W_t. W_f is initially set to 1.0 and multiplied by a growth factor of 1.2 each time it is accessed (i.e., W_f ← W_f × 1.2), with a maximum limit of 10.0; W_t is initially set to 1.0 and multiplied by a decay factor of 0.99 every minute (i.e., W_t ← W_t × 0.99^Δt), ensuring that recent data has a higher weight. The cache eviction score S = W_f × W_t is calculated. The system scans the cache status every 10 seconds, calculates and sorts the S values ​​of all cached items, and evicts the 5% of data items with the lowest scores, freeing up memory space. The weighting coefficients of the eviction strategy are not fixed, but are dynamically adjusted by the online learning module based on the cache hit rate over the past minute: if the hit rate is below 85%, the time decay factor is slightly reduced to 0.98 to extend the data residence time; if the hit rate is above 95%, it is restored to 0.99 to improve the cache update speed.

[0033] The time-series data loaded into memory is fed into the AI ​​model for feature engineering, including sliding window statistics, Fourier transform frequency domain feature extraction, and spatial correlation modeling based on graph neural networks. During this process, the model feedback interface continuously collects performance monitoring data: data loading latency (average time from request issuance to data readiness), cache hit rate (the proportion of data directly retrieved from memory), and feature extraction time (processing time per unit sample). These metrics are summarized every 10 seconds to form a performance monitoring stream.

[0034] The model feedback interface inputs performance monitoring data to the index optimizer, which performs closed-loop optimization based on a reinforcement learning framework. The state space S of the reinforcement learning agent consists of three dimensions: current cache hit rate (0-100%), average data loading latency (milliseconds), and CPU utilization (percentage) of index update operations. The action space A contains three executable operations: adjusting the index update cycle (optionally rebuilding the feature pattern index every hour, every 2 hours, or every 4 hours), modifying the prefetch window size (optionally predicting in the next 3, 5, or 8 steps), and switching the caching strategy mode (conservative, balanced, or aggressive). The reward function R is designed as the reciprocal of the overall latency. in , , To normalize the coefficients and ensure consistency across dimensions, the agent employs the Double Q Network (DQN) algorithm for learning. The experience replay pool contains 10,000 transfer records, and the target network updates every 100 steps. After seven consecutive days of online learning, the agent autonomously decides to adopt a strategy of "updating the index every 2 hours, prefetching 5 steps, and balanced caching" during weekday mornings, and a strategy of "updating every 4 hours, prefetching 3 steps, and conservative caching" at night. This ensures the system maintains a cache hit rate above 91% under varying loads, with an average loading latency consistently around 85 milliseconds.

[0035] In actual operation, the system successfully supported the daily training tasks of the urban traffic prediction model. Compared with traditional solutions, due to the three-level index joint query, the response time for complex condition retrieval was reduced from an average of 3.2 seconds to 1.1 seconds; due to the intelligent prefetching mechanism, the prefetch hit rate reached 87.3%, and the model's waiting time for data was reduced by 76%; due to hybrid compression and compressed domain aggregation, I / O read efficiency was improved by 3.6 times, and the daily processing capacity of a single node reached 380 million records; the closed-loop optimization mechanism enabled the system to automatically adjust the prefetching strategy within 2 hours when traffic patterns changed drastically during holidays, with the cache hit rate only decreasing by 4 percentage points, demonstrating strong robustness. The end-to-end data supply latency remained stable at 118 milliseconds, meeting the business requirement of completing training before the morning rush hour.

[0036] Example 2 This embodiment focuses on an Industrial Internet of Things (IIoT) scenario, applied to a blast furnace operation status monitoring and predictive maintenance system in a large steel plant. Unlike the traffic flow prediction in Embodiment 1, the time-series data in this scenario has a higher sampling frequency (100 times per second), stronger physical coupling (strong correlation among multiple variables such as temperature, pressure, and flow), and more stringent real-time requirements (fault warnings must be completed within 50 milliseconds). Furthermore, the data semantics are complex, requiring support for "abnormal pattern backtracking" queries based on waveform similarity, which places higher demands on the indexing mechanism. Therefore, this embodiment introduces three key technical differences in its system architecture: First, data storage adopts a memory-first hybrid storage architecture; second, feature pattern index construction uses a convolutional autoencoder to capture local waveform features; third, the prefetch engine introduces a fusion prediction mechanism of physical mechanism models and LSTM, forming a "data-driven + knowledge-guided" dual-mode prefetching strategy.

[0037] The data access module receives 2000 sensor data streams from the blast furnace body, including thermocouple temperature (-50℃ to 1800℃), pressure transmitter (0-10MPa), and electromagnetic flowmeter (0-500m³ / h), with a uniform sampling frequency of 100Hz. Protocol parsing adopts the OPC UA standard. After data packet standardization, timestamps are aligned to UTC nanosecond levels. Missing values ​​are filled using cubic spline interpolation (limited to consecutive missing values ​​≤10 points). Outliers are verified through physical constraints: for example, if the inlet water temperature of a cooling wall is higher than the outlet water temperature, it is determined to be a sensor fault, the data is set to zero and marked. The standardized data packet structure adopts a compact floating-point array format, with each packet containing 10 key variables at 100 time points, and a total length of 40KB.

[0038] The data storage module employs a tiered storage strategy: data from the most recent 5 minutes is written to an in-memory database (based on RedisTimeSeries), data from the next hour is stored in columnar files on a local NVMe SSD, and historical data is archived to distributed object storage (MinIO). The in-memory database supports millisecond-level writes and sub-millisecond-level queries, providing low-latency guarantees for real-time inference. The data block splitting granularity is adjusted to 1 second, generating one data block per second, supporting precise location down to the sample point. Hybrid compression encoding is optimized for high-frequency data: after differential encoding, the incremental sequence is converted using ZigZag encoding, and then combined with Google's Cap'n Proto schema for compact serialization, achieving a compression ratio of 4.1:1. Compressed data supports extreme value detection (traversing the differential bitstream to find the maximum / minimum increment) and integral calculation (accumulating differential values ​​to restore the total amount) without decompression.

[0039] The feature pattern recognition process of the data indexing module has undergone substantial changes. Since local peaks, plateaus, and slopes in temperature fluctuation curves are crucial for fault diagnosis, fully connected autoencoders struggle to capture these local features. Therefore, this embodiment employs a one-dimensional convolutional autoencoder (1D-ConvAE): the encoder contains three convolutional layers (kernel sizes of 7, 5, and 3, stride of 2, and output channels of 32, 64, and 128), followed by a global average pooling layer, outputting a 128-dimensional feature vector; the decoder has a symmetric structure. The training samples are 1024-point (10.24-second) temperature sequences captured by a sliding window, and a physical consistency constraint term is added to the loss function. in Represents the first-order difference. =0.3 is the weighting coefficient, which forces the model to maintain its rate of change characteristics while reconstructing the original signal. The clustering stage employs a spectral clustering algorithm, constructing a similarity matrix based on the cosine similarity between feature vectors, which better separates non-convex pattern clusters. Ultimately, 68 feature pattern clusters are generated, including "normal steady state," "cooling water fluctuation," "thermocouple drift," and "sudden overheating," allowing maintenance personnel to retrieve historical similar events using example waveforms.

[0040] The intelligent prefetch engine of the data loading module adopts a dual-mode fusion architecture. The first path is a data-driven path, using an LSTM model to learn historical access sequences. The input is the access records of the past 100 1-second data blocks, and the output is the access probability of each block in the next 10 seconds. The second path is a knowledge-guided path, integrating a blast furnace thermodynamic mechanism model. This model calculates the rate of change of furnace heat load in real time based on the energy conservation equation. When the predicted heat load surge exceeds a threshold, it immediately triggers a prefetch command for sensor data in the relevant area. The two prediction results are fused by weighting: the final prefetch probability P = 0.7×P_LSTM + 0.3×P_physics, ensuring timely response even under sudden operating conditions. The prefetch decision is executed every 10 milliseconds, synchronized with the control cycle.

[0041] The cache management module optimizes the eviction policy for high-frequency data. Since the lifespan of a 1-second data block is extremely short, traditional time decay models are not applicable. This embodiment employs an "access density" model: cached items are grouped by time windows (every 5 seconds), and the average number of accesses per unit time within each group is calculated as the density index. During eviction, the entire group with the lowest density is removed first. Simultaneously, to ensure real-time inference, a dedicated reservation area is set up, forcibly storing all sensor data from the most recent 2 seconds, unaffected by the eviction policy.

[0042] The task scheduling module handles two types of tasks: real-time inference tasks (priority 1, response time limit 50 milliseconds) and offline diagnostic tasks (priority 4). For real-time tasks, the scheduler bypasses the batch merging mechanism and immediately executes a single data load; for diagnostic tasks, it merges query requests from the past hour and processes them in batches to reduce overhead. The performance metrics collected by the model feedback interface include a newly added "worst-case latency" (p99.9) as a state input for the reinforcement learning agent, ensuring that the system still meets real-time requirements under extreme loads.

[0043] In the closed-loop optimization mechanism, the action space of the reinforcement learning agent adds an "Enable / Disable Physical Model Guidance" option. Under normal operating conditions, the system learns that LSTM alone is sufficient to meet the requirements, so the physical model is disabled to save computational resources. When a significant increase in heat load fluctuation is detected, the agent automatically enables dual-mode prefetching, maintaining a prefetch hit rate above 82% during the disturbance period. Experiments show that in this embodiment, the data supply latency p99.9 is 48 milliseconds in the event of a sudden charge in the blast furnace, successfully supporting the fault warning model to output results within 50 milliseconds, which is a significant performance improvement over the general architecture of Embodiment 1 in this scenario.

[0044] Example 3 This embodiment is applied to time-series data management in the field of high-frequency trading in finance, processing Level-2 market data from the Shanghai and Shenzhen Stock Exchanges, including tick-by-tick transactions, five-level bid and ask orders, and order book changes, with a peak data rate of 2 million records per second. Compared with the previous two embodiments, the core difference in this scenario is: extremely high data update frequency, query mode mainly based on "tracing back the most recent N transactions", and a requirement for data consistency of "zero loss and strict order". Therefore, this embodiment makes three structural adjustments in system design: First, the data access module introduces a distributed message queue for traffic shaping; second, the data storage module adopts a log structure merged tree (LSM-Tree) instead of columnar storage; third, cache management uses a priority queue based on data freshness, rather than access frequency.

[0045] The data access module uses a Kafka cluster as a buffer layer to receive TCP streaming data from the exchange. After protocol parsing, the raw data is partitioned by exchange, security code, and timestamp, and written to a multi-replica topic. Standardization is performed on the consumer side: timestamps are uniformly converted to nanosecond-level UTC time, missing fields (such as order book depth less than five levels) are filled with default values, and abnormal prices (such as quotes outside the daily price limit) are marked and isolated. Standardized data packets are serialized in Avro format, containing message headers (timestamp, security code, message type) and variable-length fields.

[0046] The data storage module abandons Parquet columnar storage and adopts an LSM-Tree structure built with RocksDB, organizing data with a composite primary key of "securities code + timestamp". Data is divided into 100-millisecond micro-blocks by time window, with each micro-block serving as an SSTable file within the LSM-Tree. The index structure has been adjusted: the basic time index has degenerated into time partitioning (one partition per minute), and the device clustering index has been reconstructed into "industry sector clustering," grouping over 3000 stocks into 11 major categories according to GICS industry classification. The feature pattern index focuses on "trading activity patterns," using an autoencoder to extract features from the trading volume sequence every 100 milliseconds, clustering to generate patterns such as "quiet periods," "breakout periods," and "oscillation periods," to support backtesting queries for strategies that "find similar volatile market conditions."

[0047] The prefetching strategy of the data loading module has undergone a fundamental change. Since high-frequency trading models typically pull the latest data at fixed intervals (e.g., every 10 milliseconds), the access patterns are highly regular, limiting the predictive value of LSTM. This embodiment adopts a prefetching mechanism based on deterministic scheduling: the prefetching engine parses the sampling period T in the task configuration and starts data loading T / 2 seconds in advance, ensuring that the data is ready at the start of the computation period. For example, if the model executes every 10 milliseconds, the system automatically loads data from t to t+10 milliseconds at t-5 milliseconds, forming a pipelined supply.

[0048] The cache management module has been completely restructured. The memory cache is divided into three layers: L1 is a circular buffer that stores all market data from the most recent second, using an overwrite write mode; L2 is a hotspot cache that stores securities data accessed within the past minute, using an LRU strategy; and L3 is a prefetch buffer that stores data pre-loaded according to a scheduling plan. The eviction policy only applies to L2; L1 and L3 are automatically cleaned up by a time window. Cache hit rate calculation only applies to L2, as L1 data is guaranteed to be hit.

[0049] The task scheduling module is adapted to an event-driven architecture. Task triggering no longer relies on timers, but is instead driven by the arrival of market data. Whenever a new batch of data is written to storage, the system issues an event notification to the relevant AI model. The scheduler checks whether the model's dependent data is complete; if so, it immediately schedules the computation task, achieving data-driven real-time response.

[0050] The model feedback interface adds a "data latency jitter" metric, which is the standard deviation of the difference between the data arrival time and the timestamp. This metric is fed back to the index optimizer, and the reinforcement learning agent controls the latency jitter to within 200 microseconds by adjusting the Kafka consumer concurrency and RocksDB compaction strategy. In live-fire testing, this embodiment supported an AI system making 100,000 transaction decisions per second, with an end-to-end latency of 8 milliseconds and a zero data loss rate, verifying the applicability and robustness of this invention in extremely high-frequency scenarios.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A time-series AI data management system, characterized by, The system comprises: a data access module for receiving raw time series data streams from multiple data sources, performing protocol parsing and format standardization on the raw time series data streams, and outputting standardized time series data packets in a unified format; a data storage module for performing distributed persistent storage on the accessed standardized time series data packets, the data storage module adopting a columnar storage structure; a data index module for establishing a dynamically updated multi-level index system for the stored time series data, the multi-level index system comprising a basic time index, a device clustering index, and a feature pattern index; a data loading module for extracting a required time series data subset from the data storage module in response to a data request of a task type, and loading the time series data subset to a memory cache area of a computing node, the data loading module integrating an intelligent prefetching engine for predicting data regions to be accessed subsequently based on historical access patterns and current task context; a cache management module for performing life cycle management on the time series data cache in the memory, and adopting a hybrid elimination strategy based on access frequency and time effectiveness; a task scheduling module for receiving an execution instruction of an AI task, analyzing data dependency relationships thereof, and coordinating the collaborative work of the data loading module and the computing resource; a model feedback interface for receiving performance monitoring data generated by an AI model in a training or inference process, the model feedback interface feeding the performance monitoring data to the data index module and the data loading module for dynamically adjusting index update frequency and prefetching strategy.

2. The time-series Al data management system of claim 1, wherein, The intelligent prefetching engine internally comprises a long short-term memory network model, the long short-term memory network model taking a time sequence of historical data requests as input and outputting a probability distribution of data blocks possibly accessed in future multiple time steps, the data loading module loading high-probability data blocks from persistent storage to the memory cache area in advance according to the probability distribution.

3. The time-series Al data management system of claim 1, wherein, The model feedback interface inputs the performance monitoring data to an index optimizer, the index optimizer evaluates the influence of different index update strategies on overall system delay based on a reinforcement learning algorithm, and autonomously decides an optimal index reconstruction period and feature pattern clustering granularity, achieving dynamic balance between index maintenance cost and query performance.

4. The time series Al data management system of claim 1, wherein, When performing distributed persistent storage, the data storage module divides time series data into multiple data blocks according to fixed time windows, and independently compresses and encodes each data block, the compression algorithm adopting a hybrid compression strategy based on differential encoding and dictionary encoding, the hybrid compression strategy specifically comprising: performing differential encoding on a continuous measurement value sequence of the same device to obtain an incremental sequence; establishing a static dictionary for high-frequency values in the incremental sequence and using short codes to represent the high-frequency values; using variable-length encoding to represent abnormal incremental values not covered by the dictionary; and packing and storing the differential code stream and dictionary information, the compressed data blocks supporting aggregation operations such as summation, mean, or maximum value in the compression domain.

5. The time series Al data management system of claim 2, wherein, The long short-term memory network model comprises at least two hidden layers, each layer comprising multiple memory cells, the input sequence being data request records in past multiple time steps, and the output being access probabilities of data blocks in future multiple time steps, the model periodically performing incremental training using the latest data to update model parameters.

6. The time series Al data management system of claim 1, wherein, The task type includes an AI model training task or an inference task, the inference task is given a higher priority to ensure that its data request is responded within a preset time limit, and the AI model training task adopts a batch processing mode to combine data requests of adjacent time windows to reduce I / O overhead.

7. A timing AI data management and loading method, characterized in that, The method comprises the following steps: Receiving raw time series data streams from multiple data sources through a data access module, the raw time series data streams containing timestamps, device identifiers, measurement values and metadata fields, performing protocol analysis and format standardization processing on the raw time series data streams, and outputting standardized time series data packets in a unified format; Writing the standardized time series data packets into a distributed columnar storage system, dividing them into multiple data blocks according to fixed time windows and performing persistent storage, and establishing a multi-level index system based on time intervals and device dimensions; Using a machine learning model to identify feature patterns of the stored time series data, extracting low-dimensional implicit feature vectors from the time series data segments after normalization, clustering the implicit feature vectors to generate a plurality of feature pattern clusters, and labeling each time series data segment with a feature pattern cluster number; In response to a data reading request initiated by an AI model, analyzing the time range, device list and feature type conditions in the request, combining the multi-level index system to plan a joint query, and generating an optimal data retrieval path; During data loading, maintaining access frequency weights and time effectiveness weights for each cached data item, the access frequency weights being incremented with each access, the time effectiveness weights being exponentially decayed over time, sorting the data items in the memory cache area based on the weighted sum of the two, and preferentially eliminating the data item with the lowest weighted sum to achieve dynamic management of cache resources; Based on the collected performance monitoring data, dynamically adjusting the clustering parameters of the feature pattern index, the prediction window length of the pre-fetching model and the weight coefficients of the cache eviction strategy, forming a closed-loop optimization mechanism.

8. The timing AI data management and loading method of claim 7, wherein, The format standardization processing of the raw time series data stream further comprises: converting the local timestamp to UTC standard time and aligning it at a preset time granularity; when the number of consecutive missing timestamps does not exceed a first threshold, performing linear interpolation reconstruction based on the previous and next valid data points; when the number of consecutive missing timestamps exceeds the first threshold, marking the corresponding data segment as an untrusted area and writing a quality identifier in the metadata field.

9. The timing AI data management and loading method of claim 7, wherein, The process of extracting low-dimensional implicit feature vectors includes: using an autoencoder model to encode the normalized time series data segments, the number of input layer neurons of the autoencoder model being consistent with the sequence length, the dimension of the hidden layer being lower than that of the input layer, and the loss function being mean square error, and fixing the encoder parameters after training for online feature extraction.

10. The timing AI data management and loading method of claim 7, wherein, The initial value of the access frequency weight is a reference value, which is multiplied by a growth factor greater than 1 each time it is accessed; the initial value of the time effectiveness weight is a reference value, which is decreased by an exponential decay factor over time; the cache eviction score is determined by the product of the access frequency weight and the time effectiveness weight, and the system periodically performs eviction evaluation and eliminates a preset proportion of cache data items with the lowest score.

Images