Storage scheduling and analysis processing method for high-concurrency internet-of-things data

Abstract

The invention relates to the technical field of big data storage and processing, and discloses a storage scheduling and analysis processing method for high-concurrency Internet of Things data, and the method comprises the steps: firstly building a writing end discrete tension model based on the disorder degree and fluctuation amplitude of a data stream, and building a reading end aggregation tension model based on a query intention and access popularity; secondly, substituting the two-way tension value into a tension field model, dynamically selecting and instantiating a row skip list, a column prepolymerization array or a sparse index aggregation tree as a memory table structure; when data is written into a disk, frequency domain transformation coding is performed on periodic data, and a heterogeneous sorting character string table is generated; and finally, performing frequency domain calculation in combination with the Parseval theorem in the query process, and dynamically correcting the tension weight by using performance feedback data. Through structure variation and frequency domain compression during operation, the writing performance and the query efficiency are effectively balanced, and the storage cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data storage and processing, in particular to a storage scheduling and analysis processing method for high-concurrency IoT data. BACKGROUND

[0002] At present, time-series data generated in the industrial Internet of Things scene shows an explosive growth trend. Various sensors continuously report massive state monitoring, log recording and environmental parameters. These data streams have the characteristics of high-concurrency writing, large flow fluctuation and value density decay over time. The system needs to ensure that data is not lost under the writing throughput of millions, and also needs to meet the second-level response demand of business layer for historical data trend analysis, feature extraction and other complex queries, which poses a severe challenge to the architecture design of the underlying storage engine.

[0003] The existing time-series data management system usually adopts a storage architecture based on a log-structured merge tree to cope with the writing pressure. Data is first appended to the memory buffer, and when accumulated to a certain threshold, the system sorts and dumps the memory data into an immutable sorted string table (SSTable) on the disk. In order to support queries, the system will perform multi-level file merging operations in the background, supplemented by Bloom filters or inverted indexes to speed up retrieval. Some systems also introduce columnar storage format, which continuously stores the same type of indicators within the same time window, so as to use run-length encoding or Delta encoding for compression.

[0004] However, the above existing technology still has limitations in dealing with complex IoT scenarios. First, a single fixed memory organization structure cannot accommodate data streams with different physical characteristics. If a row-oriented structure is used, it can withstand random writing of high-entropy data, but it will cause read amplification when performing large-scale aggregation analysis. If a column-oriented structure is used, it is beneficial for statistics, but when dealing with unordered or high-frequency oscillation data, the pre-sorting and encoding overhead is too large, and the writing performance drops sharply. Second, the traditional storage mechanism only compresses data in the time domain dimension, ignoring the periodicity and frequency domain sparsity of physical signals. When performing energy analysis or variance calculation, the system still needs to completely decompress and restore the compressed blocks to the time domain point set, resulting in a large amount of invalid consumption of CPU and I / O resources. In addition, the existing scheduling strategy is based on static configuration and cannot perceive the drift of business query intention. When the device ages and the signal pattern changes, or the business focus shifts from real-time monitoring to historical review, the rigid parameter configuration cannot be adjusted adaptively, eventually leading to a gradual decline in system performance over time.

[0005] Therefore, the present application provides a storage scheduling and analysis processing method for high-concurrency IoT data to solve the deficiencies in the prior art. SUMMARY

[0006] To address the shortcomings of existing technologies, this invention provides a storage scheduling and analysis processing method for high-concurrency IoT data. It solves the problem that existing single fixed storage structures cannot adapt to the dynamic changes in the physical characteristics and query intentions of IoT data, resulting in a difficulty in balancing write throughput, storage cost, and analysis efficiency.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for storing, scheduling, and analyzing high-concurrency IoT data, comprising the following steps:

[0008] Step S1: Data access and write-end discrete tension modeling. During the process of receiving the time-series data stream uploaded by the IoT device, a time sliding window is established and the physical signal characteristics of the data stream are sampled. The disorder degree index and fluctuation amplitude index of the data within the time sliding window are calculated, and the disorder degree index and the fluctuation amplitude index are jointly mapped to the write-end discrete tension value.

[0009] Step S2: Query intent parsing and read-end aggregation tension modeling. Monitor the query requests received by the database query engine, parse the query requests to extract access metadata about the target time series, and generate read-end aggregation tension values ​​based on the query frequency of the target time series within the statistical period and the proportion of aggregation queries.

[0010] Step S3: Dynamic variation of memory table structure based on bidirectional tension field. Substitute the discrete tension value of the write end and the aggregate tension value of the read end as input parameters into the tension field model to determine the range of the resultant force state vector. Select and instantiate the physical data structure type of the memory table according to the range.

[0011] Step S4: Frequency domain-aware heterogeneous sorted string table persistence and compression. When the amount of data in the memory table reaches the write threshold, the data is persisted as a sorted string table file on the disk according to the physical data structure type of the memory table. For data blocks that are determined to have periodic characteristics, transform domain encoding from time domain to frequency domain is performed, and only the frequency domain coefficients are stored.

[0012] Step S5: Adaptive query routing and tension weight correction. When the query engine reads the sorted string table file, it selects the decoding path according to the file header marker, collects performance feedback data during the query execution process, and uses the performance feedback data to dynamically adjust the weight parameters used in step S2 to generate the read-end aggregate tension value.

[0013] By employing the above technical solution, this invention constructs a bidirectional mapping mechanism between the physical signal characteristics of the write end and the business query intent of the read end, namely a bidirectional tension field. Unlike traditional time-series databases that use a single fixed storage structure (such as LSM-Tree), this invention can dynamically determine the physical organization of memory data at runtime based on the entropy change characteristics of the data flow (discrete tension) and the analytical needs of the business layer (aggregate tension). This method performs structured shaping on the data during the write phase, ensuring write throughput for high-entropy random data, pre-computing and columnar reorganization for high-frequency aggregate query data, and compressing periodic data using frequency domain sparsity. Simultaneously, through a query feedback loop, the system can adaptively adjust the tension model parameters, ensuring that the storage strategy automatically evolves with changes in the state of IoT devices and shifts in business focus, thereby achieving a dynamic balance between write performance, storage cost, and query response speed under limited hardware resources.

[0014] Preferably, step S1, which maps the disorder level index and the volatility index together to the write-end discrete tension value, specifically includes: performing a first-order difference operation on the original data set within the time sliding window to obtain a difference sequence, eliminating static bias; statistically analyzing the probability distribution of the difference sequence in multiple equally spaced discrete intervals, and calculating the Shannon entropy based on the probability distribution as the disorder level index; calculating the normalized standard deviation of the difference sequence as the volatility index; normalizing the disorder level index using a logarithmic basis, and weighting it in conjunction with the volatility index. The calculation logic for the write-end discrete tension value is as follows: using the natural logarithm of the total number of discrete intervals used to construct the probability histogram as the denominator, and the negative of the sum of the products of the probabilities of each discrete interval and their natural logarithms as the numerator, the normalized entropy value is calculated, and then the normalized entropy value is multiplied by the sum of the normalized volatility.

[0015] By adopting the above technical solution, the physical randomness of the data stream is accurately quantified using information theory methods. This effectively distinguishes between different physical states such as sensor static and linear changes (low entropy) and fault vibration and environmental abrupt changes (high entropy), providing a physical-level quantitative basis for the selection of subsequent storage structures.

[0016] Preferably, the step S2 of generating the read-end aggregation tension value specifically includes: capturing read operation requests through a bypass monitor, parsing structured query statements to generate an abstract syntax tree; identifying the query intent as either an aggregation analysis intent focusing on statistical features or a detailed backtracking intent focusing on the original waveform within a sliding counting window with a time decay factor; calculating the ratio of the number of query requests for the aggregation analysis intent to the total number of query requests as the query structure proportion; performing logarithmic smoothing on the query rate per second of the target time series as the access popularity; adjusting the weight of the query structure proportion using an aggregation tendency weight coefficient, adjusting the weight of the access popularity using a popularity tendency weight coefficient, and linearly superimposing the two to calculate the read-end aggregation tension value.

[0017] By adopting the above technical solutions, abstract business query requirements are transformed into measurable numerical indicators, enabling the storage engine to perceive the upper-layer business's demand for pre-computation capabilities, thereby finding the optimal solution in the read-write conflict.

[0018] Preferably, in step S3, selecting and instantiating the physical data structure type of the memory table according to the range includes: when the write-end discrete tension value is higher than the discreteness threshold and the read-end aggregation tension value is lower than the aggregation threshold, selecting a row-based discrete skip list structure to store the original key-value pairs through a multi-level indexed linked list; when the write-end discrete tension value is not higher than the discreteness threshold and the read-end aggregation tension value is not lower than the aggregation threshold, selecting a column-based pre-aggregated array structure to maintain a real-time updated statistics register at the head node; and when the write-end discrete tension value is higher than the discreteness threshold and the read-end aggregation tension value is not lower than the aggregation threshold, selecting a sparse indexed aggregation tree structure.

[0019] By adopting the above technical solutions, heterogeneous memory data structures are matched for different tension ranges: row-based structures ensure high-concurrency random writes; column-based pre-aggregation structures utilize low-entropy characteristics to achieve real-time statistics at a lower cost; and sparse tree structures provide macro-statistical summaries while retaining abnormal original data, thus achieving refined utilization of storage space.

[0020] Preferably, the sparse index aggregation tree structure includes non-leaf nodes and leaf nodes: the non-leaf nodes store the aggregated statistical values ​​within the subtree's coverage time range; the leaf nodes store the residual data where the difference between the original data point and the statistical mean of the parent node exceeds the error limit, and for data points that do not exceed the error limit, an existence bitmap is recorded.

[0021] By adopting the above technical solution, while meeting the needs of high-frequency aggregation queries (directly reading non-leaf nodes), the storage overhead in high-entropy data scenarios is reduced by storing only the more obvious residual values, thus solving the problem of losing abnormal details in traditional downsampling.

[0022] Preferably, the transform-domain encoding from the time domain to the frequency domain in step S4 includes: identifying data blocks originating from the columnar pre-aggregated array structure; performing a discrete Fourier transform on the time-domain sequence within the data block to generate a complex frequency-domain coefficient sequence containing amplitude and phase; retaining the dominant frequency coefficient with the largest modulus according to the energy retention threshold and discarding high-frequency coefficients; splitting the retained dominant frequency coefficients into real part sequences and imaginary part sequences, performing floating-point compression on each, and generating a data body containing only the compressed frequency-domain coefficients.

[0023] By adopting the above technical solution, periodic signals can be truncated and compressed using their sparsity in the frequency domain, achieving a compression ratio far higher than that of general time-domain compression algorithms, thus reducing disk usage.

[0024] Preferably, the step S4 of persisting the data as a sorted string table file on the disk further includes: for data blocks originating from the row-based discrete skip list structure, generating a row storage format file using differential prefix encoding combined with a block compression algorithm; for data blocks originating from the sparse index aggregation tree structure, generating a hierarchical format file, wherein the hierarchical format file is physically divided into an index area storing the aggregated statistical values ​​and a data area storing the residual data.

[0025] By adopting the above technical solution, the physical layout of disk files directly corresponds to the characteristics of memory structure, avoiding the loss of structure flattening during serialization and supporting heterogeneous reading strategies during querying.

[0026] Preferably, step S5, which selects the decoding path based on the file header marker, includes: when reading data blocks encoded by the transform domain and processing variance or energy statistics queries, using Passevar's theorem, directly performing a sum of squares operation on the frequency domain coefficients without performing an inverse transform to restore the time domain data; when reading data blocks encoded by the transform domain and processing time point value retrieval, performing a local inverse transform only on the associated frequency domain components.

[0027] By adopting the above technical solution, the push-down of the computation operator is realized, which transforms the time-domain aggregation operation that originally required full decompression and calculation into direct frequency-domain calculation in the compressed state, thereby reducing CPU and I / O overhead.

[0028] Preferably, the performance feedback data collected during the query execution process in step S5 includes: monitoring the matching status between the query intent and the underlying storage structure; when the business layer initiates an aggregation query but the underlying storage structure is a row-based discrete skip list or raw time-domain data, it is determined to be an under-aggregation scenario and a first type of mismatch penalty is recorded; when the business layer initiates a raw value point query but the underlying storage structure is a frequency domain coefficient or sparse index aggregation tree structure, it is determined to be an over-aggregation scenario and a second type of mismatch penalty is recorded. Further, an error correction algorithm is used to dynamically adjust the weight parameters used to generate the read-end aggregation tension value: calculating the deviation between the actual query intent indication value and the current storage structure indication value, and combining the normalized resource consumption cost of the current query and the preset learning rate to calculate the weight correction amount; and using the weight correction amount to update the aggregation tendency weight coefficient.

[0029] By adopting the above technical solutions, an online learning mechanism based on mismatch cost was established, which enables the system to automatically calibrate the scheduling strategy according to the performance loss in actual operation, thus solving the problem that static thresholds cannot adapt to business model drift.

[0030] This invention provides a method for storing, scheduling, and analyzing high-concurrency IoT data. It offers the following advantages:

[0031] 1. This invention achieves dynamic variation of the in-memory table structure by constructing a bidirectional mapping model between write-side discrete tension and read-side aggregation tension. Unlike traditional time-series databases that rely on a single, fixed storage engine, this invention automatically adapts to row-based skip lists for high-entropy random data to ensure write throughput, to column-based pre-aggregated arrays for highly aggregated query data to reduce statistical overhead, and to sparse index aggregation trees for mixed scenarios. This mechanism breaks the limitations of general-purpose storage structures, enabling a dynamic balance between write performance and query response speed based on real-time changes in data physical characteristics and business needs, even with limited hardware resources.

[0032] 2. This invention utilizes frequency-domain-aware heterogeneous sorted string table technology to perform transform-domain encoding on data blocks with periodic characteristics and store only the dominant frequency coefficients. This not only leverages the sparsity of signals in the frequency domain to achieve a compression ratio higher than traditional time-domain algorithms, reducing disk space usage; but also, combined with Passevar's theorem, supports direct statistical calculations of energy and variance in the frequency-domain compressed state, realizing the pushdown of computational operators and avoiding the expensive inverse transform and full decompression process, thus improving the efficiency of analyzing and processing massive amounts of historical data.

[0033] 3. This invention establishes an adaptive weight correction mechanism based on performance feedback. By monitoring the matching status between query intent and the underlying storage structure, it quantifies the mismatch costs of under-aggregation and over-aggregation and dynamically adjusts the tension model parameters accordingly. This closed-loop feedback mechanism endows the system with online learning capabilities, enabling it to automatically adapt to the drift of IoT device states and the shift in business focus. This effectively avoids the performance degradation of static scheduling strategies over time and ensures the long-term operational stability of the system. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the overall process of the high-concurrency IoT data storage scheduling and analysis processing method of the present invention.

[0035] Figure 2 A schematic diagram illustrating the process of data access and write-end discrete tension modeling in this invention;

[0036] Figure 3 This is a flowchart illustrating the query intent parsing and read-end aggregation tension modeling of the present invention;

[0037] Figure 4 This is a schematic diagram of the dynamic variation process of the memory table structure based on the bidirectional tension field of the present invention.

[0038] Figure 5 This is a schematic diagram illustrating the process of frequency domain-aware heterogeneous sorted string table solidification and compression according to the present invention.

[0039] Figure 6 This is a schematic diagram of the adaptive query routing and tension weight correction process of the present invention. Detailed Implementation

[0040] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] See attached document Figure 1 This invention provides a method for storing, scheduling, and analyzing high-concurrency IoT data. The method provided by this invention may include:

[0042] First, step S1: Data access and write-end discrete tension modeling. During the process of receiving time-series data streams uploaded by IoT devices, the system first establishes a time sliding window for each time series. Within this sliding window, the system samples and calculates the physical signal characteristics of the data stream in real time. Specifically, the system calculates the probability distribution of the first-order difference sequence of the data sequence within the window, and calculates the information entropy value and volatility standard deviation based on this distribution. These two physical indicators are mapped to a quantized write-end discrete tension value. This write-end discrete tension value characterizes the degree of disorder and abrupt changes in the current input data stream; a higher value indicates that the data tends towards a random discrete distribution, while a lower value indicates that the data tends towards a stable or periodic distribution.

[0043] Secondly, step S2: Query intent parsing and read-end aggregation tension modeling. Parallel to the data writing process in step S1, the system runs a query log monitoring process. This process parses the query requests received by the database query engine in real time, extracting access metadata about specific time series. The system statistically analyzes the query frequency for that time series within a preset time period, as well as the specific type of query operation, particularly distinguishing between aggregation queries involving downsampling calculations and point queries that retrieve raw values. Based on the query frequency and the proportion of aggregation queries, the system generates a quantified read-end aggregation tension value. This read-end aggregation tension value is used to characterize the intensity of the business layer's demand for pre-computation and macro-statistical analysis of the data stream.

[0044] Next, step S3: Dynamic variation of the memory table structure based on the bidirectional tension field. The system inputs the write-end discrete tension value generated in step S1 and the read-end aggregated tension value generated in step S2 into the preset tension field model, and calculates the resultant force state vector under their action. According to the range of the resultant force state vector, the system dynamically determines the physical data structure type of the current memory table (MemTable) at runtime. If the resultant force state indicates high write-end discrete tension and low read-end aggregated tension, the memory table is instantiated as a row-based skip list structure to preserve the original data details; if the resultant force state indicates low write-end discrete tension and high read-end aggregated tension, the memory table is instantiated as a column-based pre-aggregated array structure, and a statistical summary is calculated in real time; if the resultant force state indicates both high tensions, the memory table is instantiated as a sparse indexed aggregated tree structure, which stores aggregated values ​​in non-leaf nodes and only stores abnormal original data exceeding the residual threshold in leaf nodes.

[0045] Subsequently, step S4: Frequency-domain-aware heterogeneous sorted string table persistence and compression. When the amount of data in the memory table reaches a preset write threshold, the system performs a disk flush operation, persisting the data in memory as a sorted string table (SSTable) file on the disk. The physical layout and encoding format of the generated SSTable file directly correspond to the structure type of the memory table at the time of disk flush. For data blocks originating from a columnar pre-aggregated array structure and determined to be low-entropy periodic signals, the system uses a frequency-domain truncation compression algorithm, utilizing discrete Fourier transform to convert time-domain data into frequency-domain coefficients, and storing only the main frequency component; for data blocks originating from a sparse indexed aggregated tree structure, the system generates a hybrid storage format containing a header aggregated index and body sparse data.

[0046] Finally, step S5: Adaptive Query Routing and Tension Weight Adjustment. When reading the SSTable file, the query engine selects the appropriate decoding and calculation path based on the structural markers in the file header. For data blocks using frequency domain compression, the query engine directly calculates based on the stored frequency domain coefficients when performing variance or energy statistics queries. Simultaneously, the system collects performance feedback data on query execution, including the hit status of the pre-aggregated index and query response time. The system uses an error correction algorithm to dynamically adjust the weight parameters used in step S2 to calculate read-end aggregation tension based on the feedback data, enabling the system's storage scheduling strategy to adaptively iterate in response to changes in business query patterns.

[0047] See attached document Figure 2 This invention includes a feature extraction module at the entry point of the data writing path, used to perform step S1: extraction of the spatiotemporal entropy change fingerprint and tension modeling at the writing end. The specific implementation process of this step is as follows:

[0048] First, for each connected IoT time-series data stream, the system maintains a fixed-length [database] in memory. A time-sliding window. This window uses a first-in-first-out queue mechanism to cache the latest arriving data in real time. There are 10 data sampling points. When a new data point arrives, the system does not directly write it into the underlying LSM-Tree structure, but instead triggers a signal feature analysis of the data sequence within that window. Let the original time series data set within the current sliding window be 10 ... ,in Indicates time The sampled values. To eliminate the interference of sensor baseline drift or static bias on the identification of fluctuation features, the system first processes the original sequence. Perform first-order difference operations to generate difference sequences. ,in This difference sequence retains only the rate of change characteristics of the signal, and can more accurately reflect the instantaneous fluctuations of the data.

[0049] Next, the system processes the difference sequence. The system performs probability distribution statistics to calculate its information entropy. First, it scans the difference sequence to obtain its value range. And divide the range of values ​​into There are 10 equally spaced discrete intervals (Bins), where These are preset system hyperparameters used to control the granularity of discretization. The system iterates through the difference sequence, statistically analyzing whether each interval falls within a given range. Number of data points within And calculate the probability of occurrence in each interval. This process constructs a probability histogram of the differencing data, which reflects the random distribution characteristics of the data fluctuations. Simultaneously, to distinguish between low-amplitude random fluctuations and high-amplitude violent oscillations, the system calculates the differencing sequence. The standard deviation is then normalized to obtain the normalized volatility. .

[0050] Subsequently, the system calculates the probability distribution based on the above calculations. and normalized volatility By combining the definitions of Shannon entropy in information theory and dispersion in statistics, a write-end discrete tension model is constructed. This model outputs a value between 0 and 1, representing the write-end discrete tension value. Discrete tension value at the writing end The calculation formula is defined as follows:

[0051] ;

[0052] in:

[0053] This represents the total number of discrete intervals used to construct the probability histogram, and serves as a logarithmic basis to normalize the information entropy value to a standard range.

[0054] This indicates that the difference sequence data falls into the first... The probability of each discrete interval, and satisfying ;

[0055] This represents the normalized standard deviation of the difference sequence within the sliding window, used for magnitude weighting of the pure distribution entropy.

[0056] The discrete tension value at the writing end is calculated using this formula. It can accurately quantify the physical storage requirements of the current data stream. When When the value approaches 0, it indicates that the distribution of the difference sequence is extremely concentrated (low entropy) and the fluctuation amplitude is small. Physically, this corresponds to the sensor being in a state of stillness, linear change, or strictly periodic fluctuation. This type of data contains less information and is suitable for encoding methods with high compression ratios. A high value indicates that the distribution of the difference sequence tends to be uniform (high entropy) or has a large fluctuation range. Physically, this corresponds to the sensor being in a state of fault vibration, sudden environmental changes, or nonlinear chaos. Such data contains a large amount of unpredictable random information, and a storage structure that can preserve the original accuracy must be used to avoid information loss. The value is then passed to the scheduling controller as the first input parameter to determine the variation of the memory table structure.

[0057] See attached document Figure 3 This invention deploys an independent query intent analysis module on the query engine side to perform step S2, query intent parsing and feedback tension modeling, in parallel. The specific implementation process of this step is as follows:

[0058] First, the system establishes a non-intrusive bypass monitor at the request entry point of the time-series database query engine. This monitor does not intercept or block normal business query processes; instead, it captures all read operation requests to the database in real time through traffic mirroring or log hooking mechanisms. For each captured request, the system parses its structured query statement (SQL) or API call parameters to generate a corresponding abstract syntax tree (AST) or execution plan tree. Based on the operator characteristics in the execution plan, the system extracts access metadata about a specific time series identifier (TimeSeriesID).

[0059] During metadata extraction, the system maintains a sliding counting window with a time decay factor in memory. Within this window, the system performs fine-grained classification and statistics on query behavior. Specifically, the system identifies and distinguishes two core operational intentions:

[0060] The first type is aggregation analysis intent, characterized by query statements containing downsampling aggregation operators such as SUM (summation), AVG (average), MAX / MIN (extreme values), and STDDEV (standard deviation), or time window grouping instructions such as GROUP BYtime(...). These queries typically do not focus on the specific values ​​of each raw data point, but rather on the statistical characteristics of the data over a period of time.

[0061] The second type is the detailed backtracking intent, characterized by query statements containing only SELECT*, SCAN, or GET operations, and without involving any form of numerical calculation or dimensionality compression. This type of query aims to obtain raw, unmodified waveform data.

[0062] Next, the system constructs a read-end aggregation tension model based on the aforementioned classification and statistical results. The core objective of this model is to quantify the business layer's demand for pre-computation features. To accurately reflect this demand, the system comprehensively considers two dimensions: the proportion of query structures and absolute access frequency. The system periodically (e.g., on a second-by-second basis) updates the read-end aggregation tension value for each time series using statistical values ​​within a counting window. .

[0063] Read end aggregation tension value The calculation formula is defined as follows:

[0064] ;

[0065] in:

[0066] This represents the total number of query requests identified as aggregation analysis intents for the target time series within the current statistical time window;

[0067] This represents the total number of all types of query requests for the target time series within the current statistical time window;

[0068] QueriesPerSecond (QS) represents the access frequency for the target time series at the current moment. This metric is logarithmically processed to smooth out numerical fluctuations in high-concurrency scenarios and prevent numerical explosions in a single dimension from masking the weight of the query type.

[0069] This represents a small positive smoothing term to prevent division by zero errors that may occur in the early stages of statistics when the total number of queries is zero.

[0070] This represents the aggregation tendency weighting coefficient, used to adjust the influence of query type structure on the final tension value. This coefficient reflects the system's sensitivity to computationally intensive loads.

[0071] This represents the heat index weighting coefficient, used to adjust the impact of query concurrency on the final tension value. This coefficient reflects the system's sensitivity to I / O-intensive loads.

[0072] Subsequently, the system will calculate the reading end polymerization tension value. The output is sent to the storage engine's scheduling layer. The value, as a dynamic feedback signal, directly guides the generation of the storage structure: when A higher value indicates that the business layer primarily uses this data for trend analysis or dashboard display. The system should consume additional computing resources on the write path to build a pre-aggregated index, thereby achieving millisecond-level query responses. A low value indicates that the business layer primarily uses this data for troubleshooting or auditing original records. The system should prioritize ensuring the complete writing and low-cost storage of the original data, avoiding unnecessary pre-calculation. In this way, the read-end aggregation tension model achieves reverse driving from business needs to the underlying storage layout.

[0073] See attached document Figure 4 This invention incorporates an adaptive scheduling controller at the core of the storage engine's write link to execute step S3, which involves dynamic mutation of the memory table structure based on a bidirectional tension field. This step is crucial for achieving a dynamic balance between storage resources and computational efficiency, and its specific implementation process is as follows:

[0074] First, the system defines a two-dimensional Cartesian coordinate system in memory, called the bidirectional tension field. The horizontal axis of this coordinate system corresponds to the discrete tension value at the write end calculated and output in step S1. The vertical axis corresponds to the reading end polymerization tension value calculated and output by step S2. The system pre-sets two key thresholds determined by statistical boundaries: the dispersion threshold. With aggregation degree threshold These two thresholds divide the tension field into different quadrant regions, each corresponding to a specific physical memory structure. When the system is ready to create a new mutable memory table, the scheduler obtains the latest... and The value is mapped to a state point in the tension field. Based on the region where the state point is located, it issues instructions to the memory allocator to instantiate a specific type of memory data structure.

[0075] Specifically, when the state point falls on and When the data stream falls into a region (i.e., a highly discrete, low-aggregate region), the system determines that the current data stream has a high degree of physical randomness and that the business layer is primarily concerned with the details of the original data. In this case, the system instantiates the memory table as a row-oriented skip list structure. In this structure, data is arranged in... Stored in key-value pair format, where For timestamps, These are the original measured values. Skip lists maintain the order of data through multi-level indexed linked lists, ensuring that even when data arrives in extremely disordered conditions, the data is preserved. The write complexity is reduced, and the original values ​​of all abnormal fluctuations are fully preserved without performing any lossy compression or pre-computation, thus ensuring the write throughput and original fidelity of the data.

[0076] When the state point falls and When the data flow is in a region with low dispersion and high aggregation (i.e., a region with stable physical characteristics and where the business layer primarily focuses on macroscopic statistical trends), the system determines that the current data flow's physical characteristics are stable. In this case, the system instantiates the memory table as a columnar pre-aggregation array structure. In this structure, data is no longer stored as linked list nodes but is directly appended to a contiguous memory array. More importantly, this structure maintains a set of real-time updated statistical registers (RollupRegisters) at the memory head node to store the SUM, COUNT, MAX, and MIN values ​​of the current data block. Whenever new data is written to the array, the system only needs to update the registers. The complexity of updating these registers is reduced. Because the data itself has low dispersion (low entropy), the system discards small noise fluctuations by default and only stores the baseline value and run length. This saves a lot of memory space while allowing aggregation queries to directly read the head node register and return the results without scanning the array body.

[0077] When the state point falls and When dealing with regions of high discreteness and high aggregation, the system determines that the current data stream faces the most complex scenario: it contains a large amount of random fluctuation information that must be retained, while also facing high aggregation query pressure. In this case, the system instantiates the in-memory table as a sparse index aggregation tree structure. This is an improved balanced tree structure (such as a B+ tree variant). In this structure, non-leaf nodes do not store the original data index, but instead store the aggregated statistics (pre-computed summaries) within the time range covered by their subtrees; leaf nodes employ a sparse storage strategy, storing only significant data points whose difference from the parent node's statistical mean (i.e., residual) exceeds a preset error limit, along with their original timestamps. For data points within the normal fluctuation range, leaf nodes only record their existence bitmap without storing specific values. This hybrid structure allows the query engine to directly utilize the summary pruning of non-leaf nodes when executing aggregation queries, while indexing the significant original values ​​of leaf nodes during anomaly backtracking, achieving a dynamic trade-off between read and write performance.

[0078] Finally, the system continuously monitors the trajectory changes of the tension field state points. Once the current memory table is full and transitions to an immutable state (ImmutableMemTable) awaiting disk flushing, the system immediately reassesses the structure type of the next-generation memory table based on the latest tension value. This mechanism ensures that the physical structure of the storage engine can smoothly switch between the three states over time, thus adaptively following changes in the operating state of IoT devices (such as transitioning from static standby to fault vibration) and shifts in business focus.

[0079] See attached document Figure 5 This invention incorporates a heterogeneous serialization module in the persistence layer of the storage engine to perform step S4, frequency-domain aware heterogeneous SSTable hardening and compression. This step maps data structures in a dynamic phase-change state in memory to static files on disk and introduces frequency-domain transformations for specific signal characteristics to achieve high compression ratios and computational pushdown. The specific implementation process of this step is as follows:

[0080] First, when the memory space occupied by the MemTable reaches the system's preset write threshold (e.g., 128MB), the system performs a freeze operation, converting the mutable MemTable into an immutable MemTable and adding it to the background disk flush queue. The serializer reads the metadata tags of the MemTable to identify its physical structure type (row-based skip list, column-based array, or sparse aggregate tree) determined in step S3. Based on the identification result, the system calls the corresponding encoder to generate a sorted string table (SSTable) file with a distinctly different physical layout. Each generated SSTable file contains a fixed-length header, which contains a specific structure type identifier (MagicNumber) to indicate the decoder type and computation operator that the query engine should load when subsequently reading this file block.

[0081] Next, for data blocks originating from columnar pre-aggregated array structures and marked as having low-entropy periodic characteristics, the system employs a frequency-domain truncation compression strategy. The system identifies that although this type of data continuously generates a large number of sampling points on the time axis, its waveform is mainly composed of the superposition of several fixed fundamental frequencies (e.g., vibration signals from rotating machinery or alternating current signals). To eliminate redundancy in the time-domain data, the system performs a Discrete Fourier Transform (DFT) on the discrete time-domain sequence within this time window, mapping it from the time-domain space to the frequency-domain space.

[0082] Let the time series data to be flushed to disk be... ,in This represents the total number of sampling points contained in the data block. The system calculates the corresponding frequency domain coefficient sequence through transformation. The core formula upon which this transformation process is based is defined as:

[0083] ;

[0084] in:

[0085] Indicates the time index in the time-domain sequence. The values ​​of the original sampling points;

[0086] This represents the total length of the time series window involved in the transformation, corresponding to the size of a data block.

[0087] This represents the index of the frequency component in the frequency domain, with values ​​ranging from 0 to... ,in It is the number of truncated and retained main frequency coefficients;

[0088] Represents the imaginary unit, satisfying ;

[0089] The base of the natural logarithm;

[0090] The transformed result is the first... Complex coefficients for each frequency component, which contain amplitude and phase information.

[0091] Subsequently, the system performs frequency domain truncation and encoding / storage operations. Based on signal processing theory, for low-entropy, inherently periodic physical signals, their energy is mainly concentrated in the low-frequency range. The system selects the preceding frequency domain truncation and encoding / storage operations according to a preset energy retention threshold (e.g., retaining 99% of the signal energy). The main frequency coefficient with the largest modulus value (i.e. to ),in For indexes greater than The system treats high-frequency coefficients as noise and discards them. The system will retain... Each complex coefficient is split into a real part sequence and an imaginary part sequence, and floating-point compression algorithms (such as Gorilla or Chimp algorithms) are applied to each sequence. The final SSTable body stores only these... The compressed frequency domain coefficients. Because of Passevar's theorem, the total energy of the time-domain signal is equal to the sum of the squares of the magnitudes of the frequency domain coefficients. Therefore, when the query engine performs subsequent variance, standard deviation, or energy integral queries, it can directly read and calculate the stored frequency domain coefficients to obtain accurate results, without needing to perform an inverse Fourier transform to restore the time-domain data.

[0092] Finally, for data blocks originating from the other two memory structures, the system employs corresponding heterogeneous encoding strategies. For row-based discrete skip lists, the system uses differential prefix encoding combined with a general block compression algorithm (such as ZSTD) to generate a standard row-based SSTable, where each data block retains complete... Yes, to support random access at any point in time. For sparse indexed aggregation trees, the system serializes them into a hierarchical SSTable structure. The file is physically divided into two independent regions: the index region at the tail stores the aggregate statistics (Sum, Min, Max) of non-leaf nodes, and the main data region stores the sparse residual values ​​of leaf nodes. This hierarchical physical layout allows the query engine to return results by loading only the index region at the tail of the file when processing aggregation queries, thus avoiding I / O reads of the main data region.

[0093] See attached document Figure 6 This invention deploys an intelligent routing and feedback controller at the interface between the query engine and the storage engine to execute step S5, adaptive query routing and tension weight correction. This step, through a closed-loop feedback mechanism, dynamically calibrates the tension model parameters from the preceding steps, ensuring that the system's storage strategy always aligns with actual business needs. The specific implementation process of this step is as follows:

[0094] First, upon receiving a query request, the query routing engine reads the header metadata of the target SSTable file. This metadata contains the structure type identifier written in step S4. Based on this identifier, the routing engine distributes the query request to the corresponding heterogeneous decoder.

[0095] For data blocks identified as frequency-domain compressed, if the query request is for energy statistics (such as variance, standard deviation, and root mean square calculation), the decoder does not perform an inverse Fourier transform (IDFT) to reconstruct the time-domain data. Instead, it directly uses Passevar's theorem to perform a sum of squares operation on the frequency-domain coefficients. This reduces the computational complexity from that in the time domain to... Reduced to the frequency domain (in The number of coefficients to be retained, and If the query request is a numerical retrieval at a specific time point, the decoder will only perform a local inverse transform on the frequency domain components associated with that time point.

[0096] For data blocks identified as sparse index aggregation trees, the routing engine prioritizes loading the aggregation index blocks at the end of the file. If the query is an aggregation operation (such as Sum), the engine directly returns the statistical values ​​in the index block, completely skipping I / O reads of the data body; if the query is a range scan, the engine uses the minimum-maximum index in the index block for Bloom filtering, and only reads the residual values ​​in the data body through sparse pointers when there is an intersection between the query range and the data range.

[0097] Next, the system runs a performance monitoring background thread to collect structure and intent matching data in real time during query execution. The system defines two types of mismatch penalties: the first type occurs in under-aggregation scenarios, where the business layer initiates an aggregation query, but the underlying storage structure is a row-based discrete skip list or raw time-domain data, causing the system to scan the entire dataset for on-the-spot calculations, resulting in high I / O latency; the second type occurs in over-aggregation scenarios, where the business layer initiates a raw point query, but the underlying storage structure is frequency domain compression or sparse tree, causing the system to perform inverse transformation or reconstruction operations, resulting in additional CPU overhead.

[0098] Subsequently, based on the aforementioned mismatch penalty data, the system dynamically corrects the weight coefficients in the read-end aggregation tension model defined in step S2 using an online gradient descent algorithm. This is particularly important for the aggregation type weight coefficients. The system incrementally updates the data by calculating the deviation between the actual query intent and the current storage structure. (Weight coefficient) The corrected formula is defined as:

[0099] ;

[0100] in:

[0101] This represents the aggregation type weight coefficient for the next time period after the update;

[0102] This represents the aggregation type weight coefficient for the current time period;

[0103] The learning rate is a preset small positive number (e.g., 0.01) used to control the step size of parameter adjustment and prevent numerical oscillations.

[0104] This indicates the total number of query samples collected during this feedback update period;

[0105] Indicates the first The actual intent indication value of a query sample; if the query is an aggregate query, then... If it is a detailed query, then ;

[0106] Indicates the first The physical structure indicator value of the data block accessed by the query; if the data block is a pre-aggregated structure (such as a columnar array or sparse tree), then... If it is a primitive storage structure (such as a row skip list), then ;

[0107] Indicates the first The normalized execution cost of a query is typically the normalized value of the query's response time or CPU / IO resource consumption.

[0108] This formula enables automated parameter tuning: when under-aggregation occurs (the query is aggregated)... Store as original Error term It is positive, and the cost is... At higher levels, the system will increase. The value makes the read-end aggregation tension calculated subsequently... Increased aggregation levels encourage the system to generate more pre-aggregated SSTables in the future; conversely, decreased aggregation levels occur when over-aggregation occurs (queries are for details). Stored as pre-aggregated Error term If it is negative, the system will decrease. This value prompts the system to revert to its original storage model. This mechanism ensures that the storage structure adapts and evolves with the business model.

Claims

1. A method for storing, scheduling, and analyzing high-concurrency IoT data, characterized in that, Includes the following steps: S1. Data access and write-end discrete tension modeling: During the process of receiving time-series data streams uploaded by IoT devices, a time sliding window is established and the physical signal characteristics of the data stream are sampled. The disorder degree index and fluctuation amplitude index of the data within the time sliding window are calculated, and the disorder degree index and the fluctuation amplitude index are jointly mapped to the write-end discrete tension value. S2. Query intent parsing and read-end aggregation tension modeling: Monitor the query requests received by the database query engine, parse the query requests to extract access metadata about the target time series, and generate read-end aggregation tension values ​​based on the query frequency of the target time series within the statistical period and the proportion of aggregation queries. S3. Dynamic variation of memory table structure based on bidirectional tension field: The discrete tension value at the write end and the aggregate tension value at the read end are used as input parameters and substituted into the tension field model to determine the range of the resultant force state vector. The physical data structure type of the memory table is selected and instantiated according to the range. S4. Frequency domain-aware heterogeneous sorted string table persistence and compression: When the amount of data in the memory table reaches the write threshold, the data is persisted as a sorted string table file on the disk according to the physical data structure type of the memory table. For data blocks that are determined to have periodic characteristics, a transform domain encoding from the time domain to the frequency domain is performed, and only the frequency domain coefficients are stored. S5. Adaptive query routing and tension weight correction: When reading the sorted string table file, the query engine selects the decoding path according to the file header marker, collects performance feedback data during the query execution process, and uses the performance feedback data to dynamically adjust the weight parameters used in step S2 to generate the read-end aggregate tension value.

2. The method for storing, scheduling, and analyzing high-concurrency IoT data according to claim 1, characterized in that, In step S1, jointly mapping the disorder level index and the fluctuation amplitude index to the write-end discrete tension value includes: A difference sequence is obtained by performing a first-order difference operation on the original data set within the time sliding window; The probability distribution of the difference sequence in multiple equally spaced discrete intervals is statistically analyzed, and the Shannon entropy is calculated based on the probability distribution as an indicator of the degree of disorder. The normalized standard deviation of the difference sequence is calculated as the volatility index; The disorder index is normalized using a logarithmic basis and weighted in conjunction with the fluctuation amplitude index to obtain the write-end discrete tension value, which characterizes the degree of random discrete distribution of the data stream.

3. The method for storing, scheduling, and analyzing high-concurrency IoT data according to claim 1, characterized in that, The generation of the read-end polymerization tension value in step S2 includes: The bypass monitor captures read operation requests and parses structured query statements to generate an abstract syntax tree. Within a sliding counting window with a time decay factor, the query intent is identified as either an aggregate analysis intent focused on statistical features or a detailed backtracking intent focused on the original waveform. The ratio of the number of query requests for the aggregated analytical intent to the total number of query requests is used as the query structure proportion. Log-smooth the query rate per second of the target time series and use it as the access popularity. The weight of the query structure proportion is adjusted using the aggregation tendency weight coefficient, and the weight of the access popularity is adjusted using the popularity tendency weight coefficient, so as to calculate the read end aggregation tension value.

4. The method for storing, scheduling, and analyzing high-concurrency IoT data according to claim 1, characterized in that, The physical data structure type for selecting and instantiating the memory table based on the range in step S3 includes: When the write end discrete tension value is higher than the discreteness threshold and the read end aggregate tension value is lower than the aggregateness threshold, a row-based discrete skip list structure is selected. The row-based discrete skip list structure stores the original key-value pairs through a multi-level indexed linked list. When the write end discrete tension value is not higher than the discreteness threshold and the read end aggregate tension value is not lower than the aggregateness threshold, a columnar pre-aggregation array structure is selected, and the columnar pre-aggregation array structure maintains a real-time updated statistics register in the head node; When the write end discrete tension value is higher than the discreteness threshold and the read end aggregate tension value is not lower than the aggregateness threshold, a sparse index aggregate tree structure is selected.

5. The method for storage, scheduling, analysis, and processing of high-concurrency IoT data according to claim 4, characterized in that, The sparse index aggregation tree structure includes non-leaf nodes and leaf nodes: The non-leaf nodes store aggregated statistics within the subtree's coverage time range; The leaf nodes store residual data where the difference between the original data point and the statistical mean of the parent node exceeds the error limit, and record an existence bitmap for data points that do not exceed the error limit.

6. The method for storage, scheduling, analysis, and processing of high-concurrency IoT data according to claim 4, characterized in that, The transform-domain coding from the time domain to the frequency domain performed in step S4 includes: Identify data blocks originating from the columnar pre-aggregated array structure; Perform a discrete Fourier transform on the time-domain sequence within the data block to generate a complex frequency-domain coefficient sequence containing amplitude and phase; Based on the energy retention threshold, retain the dominant frequency coefficient with the largest modulus and discard high-frequency coefficients; The retained main frequency coefficients are split into real part sequences and imaginary part sequences, and floating-point compression is performed on each sequence to generate a data body containing only the compressed frequency domain coefficients.

7. The method for storing, scheduling, and analyzing high-concurrency IoT data according to claim 4, characterized in that, The S4 step of persisting the data as a sorted string table file on disk also includes: For data blocks originating from the row-based discrete skip list structure, a row storage format file is generated using differential prefix coding combined with a block compression algorithm. For data blocks originating from the sparse index aggregation tree structure, a hierarchical format file is generated, which is physically divided into an index area storing the aggregated statistical values ​​and a data area storing residual data.

8. The method for storing, scheduling, and analyzing high-concurrency IoT data according to claim 6, characterized in that, The step S5, which involves selecting the decoding path based on the file header marker, includes: When reading data blocks encoded by the transform domain and processing variance or energy statistics queries, the sum of squares operation is directly performed on the frequency domain coefficients using Passevar's theorem, without performing an inverse transform to restore the time domain data. When reading the data block encoded by the transform domain and processing the time point value retrieval, only the associated frequency domain components are subjected to local inverse transform.

9. The method for storage, scheduling, analysis, and processing of high-concurrency IoT data according to claim 1, characterized in that, The performance feedback data collected during the query execution process in step S5 includes: Monitor the matching status between query intent and the underlying storage structure; When the business layer initiates an aggregation query but the underlying storage structure is a row-based discrete skip list or raw time-domain data, it is determined to be an under-aggregation scenario and the first type of mismatch penalty is recorded. When the business layer initiates a query for the original value point but the underlying storage structure is a frequency domain coefficient or a sparse index aggregation tree structure, it is determined to be an over-aggregation scenario and the second type of mismatch penalty is recorded.

10. The method for storing, scheduling, and analyzing high-concurrency IoT data according to claim 9, characterized in that, The weighting parameters used to generate the read-end aggregation tension value in step S5 include: For the aggregation tendency weighting coefficient, calculate the deviation between the actual query intent indication value and the current storage structure indication value; Get the normalized resource consumption cost of the current query; The weight correction amount is calculated based on the deviation, the normalized resource consumption cost, and the preset learning rate; The aggregation tendency weight coefficient is updated using the weight correction amount, such that the aggregation tendency weight coefficient is increased when the under-aggregation scenario occurs, and decreased when the over-aggregation scenario occurs.

Images