A time sequence data partition storage method for a high frequency acquisition system

Abstract

This invention discloses a time-series data partitioning and storage method for high-frequency acquisition systems, relating to the field of data storage management technology, and is used to solve the problem of dynamic imbalance in high-frequency write structures. This invention achieves rapid response and adaptive structural adjustment to sudden write loads in high-frequency acquisition systems by constructing a dynamic time-series data storage mechanism based on write pressure sensing and partition structure linkage. By constructing a sliding write rate sequence and a write pressure factor, it identifies write overload states, dynamically triggers the structural splitting operation of the main partition, generates sub-partitions with time boundaries and independent buffer paths, uses timestamp mapping and buffer utilization judgment to achieve load-balanced routing of write requests, and combines write activity and latency statistics to generate a structure merging evaluation factor. The partition structure is dynamically adjusted according to trend changes, enabling the write structure to match the system load, thereby improving the system's structural resilience and write efficiency in high-concurrency scenarios.

Description

Technical Field

[0001] This invention relates to the field of data storage management technology, and more specifically, to a method for partitioned storage of time-series data in a high-frequency acquisition system. Background Technology

[0002] With the rapid development of the Internet of Things, intelligent manufacturing, and industrial process control, high-frequency data acquisition systems for equipment are widely used in key scenarios such as condition monitoring, fault prediction, and production optimization. In such systems, sensor terminals continuously generate high-density time-series data at extremely short intervals (e.g., sub-second intervals), forming a typical high-frequency acquisition stream. To meet these high-speed write requirements, existing technologies generally adopt time-series databases as the underlying storage system and introduce logical partitioning strategies based on acquisition time to reduce the risk of single-table bloat and improve query efficiency.

[0003] The shortcomings of existing technologies: In real industrial operations, the data writing behavior of high-frequency acquisition systems exhibits significant unevenness, especially in scenarios such as sudden changes in equipment status, precursors to failures, or load disturbances. In these situations, the acquisition system can generate a surge in data records within a very short time window, causing a sudden increase in write load. Traditional fixed-period partitioning strategies lack the ability to dynamically perceive changes in write rate, resulting in all abnormally high-density data still being categorized into a single logical partition within the current period. This mismatch in partitioning structure leads to multiple problems: Firstly, write threads concentrate on competing for the buffer and lock resources of that partition, significantly increasing write latency; secondly, the index, metadata, and cache of that partition rapidly expand, affecting the read and write performance of other normal partitions, causing a severe disconnect between the partitioning strategy and data behavior, especially during periods of high load bursts, resulting in a significant degradation in system response performance. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, the following solution is proposed to address the problem of dynamic imbalance in the high-frequency writing structure described in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for partitioning and storing time-series data in a high-frequency acquisition system includes the following steps:

[0007] The system collects raw write requests from high-frequency data streams in real time, organizes the request data according to the write time, and constructs a write rate sequence within a time window.

[0008] Extract the slope of the write rate change and the rate of change of the queue length of the write buffer from the sliding write rate sequence, calculate the write pressure factor, and identify whether there is a write overload state.

[0009] Construct a partition switching intent signal based on changes in the write pressure factor, and determine whether the currently being written primary partition meets the conditions for triggering a structural split.

[0010] Collect the data organization parameters of the current primary partition within the set write cycle, divide the primary partition into sub-partitions, and determine the time range of each sub-partition based on the trend of write rate changes;

[0011] Establish a timestamp mapping table for sub-partitions, and route write requests to the corresponding sub-partition buffer area based on the time matching results and write load distribution.

[0012] Perform write activity monitoring and write latency statistics for each sub-partition, and generate partition structure merging evaluation factors;

[0013] Based on the changing trend of the partition structure merging evaluation factor, the sub-partition structure reclamation threshold and the index structure reconstruction triggering condition are dynamically adjusted to keep the sub-partition structure in harmony with the current write load.

[0014] In a preferred embodiment, raw write requests from a high-frequency data stream are acquired in real time, the request data is organized according to the write time, and a write rate sequence under a time window is constructed. Specific steps include:

[0015] Collect the timestamp and data length value of each write request;

[0016] Calculate the total number of write requests and the total amount of data within a fixed-length time window.

[0017] Based on the statistical results of continuous windows, a sliding write rate sequence is constructed to reflect changes in write frequency.

[0018] In a preferred embodiment, a sliding write rate sequence reflecting changes in write frequency is constructed based on the statistical results of a continuous window. Specific steps include:

[0019] By comparing the number of write requests in two adjacent time windows, the trend of the number of write requests in the current time period can be obtained.

[0020] The statistical results within a continuous time window are arranged in chronological order to form a sliding write rate sequence that reflects changes in the number of write requests.

[0021] In a preferred embodiment, the slope of the write rate change and the rate of change of the write buffer queue length are extracted from the sliding write rate sequence to calculate the write pressure factor and identify whether a write overload state exists. Specific steps include:

[0022] Calculate the slope of the change in write rate between adjacent time windows;

[0023] Calculate the rate of change of the buffer queue length within adjacent time windows;

[0024] The slope of change and the rate of change are normalized and then multiplied to generate the pressure factor.

[0025] When the write pressure factor exceeds the preset write pressure threshold, the current window is marked as write overloaded.

[0026] In a preferred embodiment, a partition switching intention signal is constructed based on changes in the write pressure factor, and it is determined whether the currently being written to the primary partition meets the conditions for triggering a structural split. Specific steps include:

[0027] Upon determining that a sudden increase in write load has occurred, a switching signal is generated to indicate that the current write path needs to be adjusted.

[0028] The switching signal is compared with the active period of the current write partition, the fluctuation range of the data write rate, and the index update density;

[0029] If the comparison result meets the triggering condition, then a splitting operation is performed to divide the currently written partition into sub-partitions.

[0030] In a preferred embodiment, the data organization parameters of the current primary partition within a set write cycle are collected, the primary partition is divided into sub-partitions, and the time range of each sub-partition is determined based on the trend of write rate changes, including the following steps:

[0031] Based on a complete write cycle before the split operation is triggered, the current write time period is divided into two or more sub-partitions at fixed time intervals.

[0032] Each sub-partition is assigned a non-overlapping time range identifier and an independent data write path for subsequent time mapping and buffer scheduling.

[0033] In a preferred embodiment, a timestamp mapping table for sub-partitions is established, and write requests are routed to the corresponding sub-partition buffer area based on the time matching results and write load distribution, including the following steps:

[0034] Based on the timestamp value of each write request, look up the matching time range identifier in the mapping table;

[0035] Based on the current buffer usage rate of the target sub-partition, determine whether the write condition is met;

[0036] When the conditions are met, the write request is redirected to the target subpartition and the write operation is performed.

[0037] In a preferred embodiment, write activity monitoring and write latency statistics are performed on each sub-partition to generate a partition structure merging evaluation factor, including the following steps:

[0038] Collect the cumulative number of writes completed by the sub-partition in the current write cycle, calculate the response time of each write request, and determine the average write latency of the sub-partition based on the statistical results of all response times.

[0039] The number of buffer refreshes completed by writing data to the buffer in a sub-partition within a period;

[0040] Based on the number of writes, average write latency, number of buffer refreshes, and preset behavioral characteristic judgment rules, a merged evaluation factor is generated to describe the stability of the write status of the sub-partition.

[0041] When the merge evaluation factor is lower than the calibrated merge reference value, and the differences in the number of writes, average write latency, and number of buffer refreshes between adjacent sub-partitions are within the preset tolerance range, the adjacent sub-partitions are determined to enter the structure merge process.

[0042] In a preferred embodiment, the sub-partition structure reclamation threshold and index structure reconstruction triggering conditions are dynamically adjusted based on the changing trend of the partition structure merging evaluation factor, so that the sub-partition structure remains coordinated with the current write load. This includes the following steps:

[0043] When the partition merge evaluation value is continuously lower than the preset stability threshold, and the difference in write activity between adjacent sub-partitions is less than the specified tolerance range, the index reconstruction operation is triggered.

[0044] After index reconstruction, the system is restored to a single primary partition structure.

[0045] In a preferred embodiment, the index reconstruction operation includes merging data blocks in sub-partitions, updating the time index structure, and releasing invalid path identifiers.

[0046] The technical effects and advantages of the time-series data partitioning storage method for high-frequency acquisition systems of the present invention are as follows:

[0047] This invention achieves real-time response and adaptive structural adjustment to burst write loads in high-frequency acquisition systems by constructing a dynamic storage mechanism for time-series data based on write pressure sensing and partition structure linkage. By collecting the timestamp and data length of each write request, a sliding write rate sequence is constructed. The write rate change slope and buffer queue length change rate are extracted to generate a write pressure factor and identify write overload states. Based on the evolution trend of the write pressure factor, a partition switching intention signal is triggered. Combined with the main partition's active period, write rate fluctuation amplitude, and index update density, it is determined whether to perform a structural split operation, dynamically generating sub-partition structures with time boundaries and independent write paths.

[0048] During data redirection, a timestamp mapping table is constructed and combined with the buffer utilization rate of the target sub-partition to achieve load balancing of write requests. By performing write activity and write latency statistics on the sub-partitions, a merging evaluation factor to describe the stability of the sub-partitions is constructed. Based on the trend changes of the merging evaluation factor, the structure reclamation threshold and index reconstruction conditions are dynamically adjusted to achieve continuous matching and resource coordination between the sub-partition structure and the write load. This method effectively solves the problems of rigid storage structure and unbalanced resource scheduling caused by strong write bursts and drastic load fluctuations in existing high-frequency acquisition systems, and significantly improves the structural flexibility, real-time response, and data write efficiency of the system in high-concurrency data write scenarios. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a time-series data partitioning and storage method for a high-frequency acquisition system according to the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In order to achieve the above objectives, Figure 1 A schematic diagram of a time-series data partitioning and storage method for a high-frequency acquisition system according to the present invention is provided, which specifically includes the following steps;

[0052] The system collects raw write requests from high-frequency data streams in real time, organizes the request data according to the write time, and constructs a write rate sequence within a time window.

[0053] Extract the slope of the write rate change and the rate of change of the queue length of the write buffer from the sliding write rate sequence, calculate the write pressure factor, and identify whether there is a write overload state.

[0054] Construct a partition switching intent signal based on changes in the write pressure factor, and determine whether the currently being written primary partition meets the conditions for triggering a structural split.

[0055] Collect the data organization parameters of the current primary partition within the set write cycle, divide the primary partition into sub-partitions, and determine the time range of each sub-partition based on the trend of write rate changes;

[0056] Establish a timestamp mapping table for sub-partitions, and route write requests to the corresponding sub-partition buffer area based on the time matching results and write load distribution.

[0057] Perform write activity monitoring and write latency statistics for each sub-partition, and generate partition structure merging evaluation factors;

[0058] Based on the changing trend of the partition structure merging evaluation factor, the sub-partition structure reclamation threshold and the index structure reconstruction triggering condition are dynamically adjusted to keep the sub-partition structure in harmony with the current write load.

[0059] Step 1: Real-time acquisition of raw write requests from high-frequency data streams, organization of the request data according to write time, and construction of a write rate sequence within a time window. The specific implementation is as follows:

[0060] The acquisition logic is deployed in the write control module to record each original write request, extract the timestamp and data length value corresponding to the write request. The timestamp is assigned by the high-frequency acquisition system's write queue receiving time, and the data length value is the number of data bytes corresponding to the write operation.

[0061] The continuously received write requests are organized in chronological order and divided into fixed-length time windows. The length of each time window is set according to the system performance requirements, such as 1 second, 5 seconds or other time periods agreed upon by the project. Within each time window, the total number of cumulative write requests and the sum of the data length values ​​corresponding to all write requests within that window are counted to obtain the total number of write requests and the total amount of data within that time window.

[0062] The statistical results from multiple time windows are arranged chronologically to construct a sliding write rate sequence reflecting changes in write frequency. The sliding write rate sequence is organized by time window, with each data point containing two fields: total number of write requests and total data volume, corresponding to write density and data pressure within the corresponding time period, respectively. The sliding write rate sequence serves as the input for subsequent steps to determine write pressure factors and trigger partition structure adjustments, enabling trend modeling and anomaly identification of the system's current write behavior.

[0063] Within each time window, the high-frequency acquisition system performs the following three operations:

[0064] The total number of all write requests within the time window, i.e. the number of write request entries successfully received and queued in the write queue within the window;

[0065] Sum the data length values ​​corresponding to all write requests within the window to obtain the total amount of data written within the time window;

[0066] The start time of the current time window, the total number of write requests, and the total amount of data written are encapsulated into a set of statistical records;

[0067] The statistical records of multiple consecutive windows are arranged sequentially in chronological order to form a sliding write rate sequence. This sequence is a time-series data structure, where each item represents the write behavior status within a fixed time window. It includes two dimensions: the number of write entries and the amount of data written. This structure is used to dynamically analyze the current system's write intensity trend over time. The number of write entries measures the level of data access activity per unit time, while the amount of data written reflects the data transfer load level per unit time.

[0068] It should be noted that the data length value is the size of the data carried in the write request, which is read from the data field specified in the request structure, and the unit is bytes; the time window length is a static parameter that can be configured by the user, representing the duration of each statistical period, used to separate time periods, such as every 1 second or 5 seconds, the system will reset the statistics within each duration; the write entry refers to the number of write requests received cumulatively within a single time window, which is recorded by the built-in counter of the system's write receiving thread.

[0069] Based on the statistical results of continuous windows, a sliding write rate sequence is constructed to reflect changes in write frequency. The specific implementation includes:

[0070] By statistically analyzing write requests, a fixed-length time window is set to divide the time axis into equal intervals. Each time window corresponds to a complete statistical period, which is used to summarize the write request behavior within that period. Within each time window, the total number of write requests is recorded. The total number of write requests refers to the total number of write requests successfully received and queued for processing between the start and end times of the window. The request counter in the high-frequency acquisition system accumulates in real time and archives the data at the end of the window.

[0071] The system processes two adjacent time windows sequentially. For each window, it reads the number of write requests counted for each and compares them to determine whether the number of write requests in the current time period has significantly increased, decreased, or remained stable. This comparison is based on a logical judgment of the difference in quantity and direction of change between the two values. Based on the comparison results, a data item representing the trend of write quantity changes is generated. This data item can be set as a status value with a trend indicator, such as "increase," "decrease," or "no significant change," or it can use a numerical label to represent the relative level of increase or decrease. This setting can be specified by the specific system policy configuration during the deployment phase.

[0072] The number of write requests and their changing trends within a series of time windows are arranged sequentially according to time order to form a complete sliding write rate sequence. This sliding write rate sequence consists of multiple data structure units, each of which includes: a time window number or time label (representing a statistical time period); the number of write requests within the current time window; and information on the changing trend of the number of write requests between the current time window and the previous time window.

[0073] The adjacent time window comparison operation classifies and judges based on the magnitude relationship between two statistical values. For example, a minimum change unit is set as the sensitivity. When the change in the number of write requests between two windows exceeds this unit, it is judged as a trend shift. The write frequency change trend is a trend indicator determined based on the above classification results. It is used to describe the direction and intensity of the change in data write activity within two consecutive statistical periods. It is a behavioral abstraction of the original value change. The sliding write rate sequence is a statistical sequence arranged in chronological order. It is used to describe the dynamic change process of write behavior within a continuous period of time. Each data item corresponds to a time window. The whole constitutes a data structure that reflects the evolution trajectory of write behavior.

[0074] Based on the previously constructed sliding write rate sequence and write buffer queue status information, it is possible to identify whether there is a write overload state. The purpose is to detect sudden changes in write behavior in a timely manner in the high-frequency acquisition system, thereby providing a basis for subsequent dynamic partition switching or structural adjustment.

[0075] Extract the slope of the write rate change and the rate of change of the write buffer queue length from the sliding write rate sequence, calculate the write pressure factor, and identify whether a write overload state exists. Specifically, the implementation is as follows:

[0076] The number of write requests in two adjacent time windows in the sliding write rate sequence is compared to obtain the trend of write rate change. In each time window, the number of write requests represents the total number of write requests actually received by the system in that window. Two adjacent windows are selected in chronological order, and the ratio between the difference in the number of write requests between the two windows and the time interval is calculated to describe the degree of change in write rate. The logical process is expressed as follows: when there is a significant increase or decrease in the number of write requests in two time windows, the system determines the direction and magnitude of the write rate change accordingly, forming the slope of the write rate change.

[0077] The system retrieves the queue length value at the end of each time window. The queue length value represents the number of data entries in the system's write buffer that have not yet been written. The change in queue length within adjacent time windows is used to reflect the data backlog trend of the system during that period. The system compares the queue length values ​​within two time windows to obtain the queue length change rate, the magnitude of which is used to indicate whether the system's current data processing capacity lags behind the data access rate.

[0078] The high-frequency acquisition system normalizes the two indicators mentioned above—the slope of the change in write rate and the rate of change in the queue length of the write buffer queue. Normalization means converting the current value of each indicator into a standardized numerical range based on the statistical results of long-term data in the system. Usually, the historical maximum and minimum values ​​are used as the calibration range to ensure that indicators of different dimensions have the same influence weight when performing comprehensive calculations.

[0079] The two normalized values ​​are combined to obtain the write pressure factor. This write pressure factor comprehensively reflects the instantaneous pressure level faced by the system in terms of both write intensity and data queuing within the current time window. The larger the value of the write pressure factor, the stronger the write impact and the more severe the data queuing. A write pressure threshold, called the preset write pressure threshold, can be set. This write pressure threshold is derived from the test data analysis results before the system goes live or during the initial operation phase. It represents the upper limit of write pressure that the system can tolerate under stable operation. When the actual value of the write pressure factor exceeds this threshold, the current time window will be marked as a write overload state. Once marked as a write overload state, the write behavior within this time window will trigger partition switching intent or other control mechanisms to alleviate the current write pressure and reconstruct the storage structure.

[0080] The specific process for obtaining the writing pressure factor through combined processing is as follows:

[0081] First, the normalized value of the slope of the write rate change, which represents the growth trend of the number of write requests in the current time window. The value ranges from zero to one, and the larger the value, the faster the write requests grow. Second, the normalized value of the rate of change of the write buffer queue length, which represents the degree of lag between processing capacity and write speed. The larger the value, the more serious the system backlog.

[0082] The two normalized values ​​are analyzed side by side. If both indicators are close to their normalization upper limit (i.e., the value is close to 1), the system judges that the write pressure is in an extreme state. At this time, the write pressure factor is marked as severe to trigger the instantaneous structural adjustment mechanism.

[0083] If only one of the two indicators is higher, such as a significant increase in write speed but no change in queue length, or vice versa, the system will mark the pressure factor as moderate, enter the observation state, and temporarily not trigger structural adjustments, only update the sliding window sequence;

[0084] If both indicators are at a low to medium level, the write pressure factor is marked as normal, indicating that the current write load is within a controllable range. For example, a severe pressure factor is when both normalized indicators are above 0.8; a moderate pressure factor is when one indicator is above 0.8 and the other is below 0.5; and a normal pressure factor is when both indicators are below 0.5.

[0085] For example, taking a 5-second time window as an example, a certain acquisition system records the following information in the first two windows: the number of write requests in the first window is 100, and in the second window it is 170. Based on the historical range, the system judges that the increase in write requests is significant, and after normalization, the normalized value of the rate change slope is 0.85. At the same time, the queue length of the write buffer in the first window is 20, and in the second window it is 45. After normalization, the queue length change rate is 0.90. According to the configured pressure factor combination rules, the system identifies that both indicators in the current time window are close to the normalized maximum value. The system marks the write pressure factor as severe and sends a partition switching suggestion to the upper-level scheduler, entering the structural response process.

[0086] It should be noted that the stress factor classification rules can be adjusted according to different business scenarios. For example, the severity level threshold can be set to 0.75 or 0.9.

[0087] After identifying that the write pressure factor has reached a preset threshold, a partition switching intention signal is further generated, and multiple influencing dimensions are combined to determine whether the primary partition currently being written to meets the structural splitting conditions, so as to improve the stability of data write processing and system throughput efficiency.

[0088] Based on the changes in the write pressure factor, a partition switching intention signal is constructed, and it is determined whether the currently being written to the primary partition meets the conditions for triggering a structural split. The implementation method is as follows:

[0089] When the write pressure factor, jointly generated by the sliding write rate sequence and buffer queue status information, is marked as critical, the system first generates a logical signal to trigger write path adjustments, called the partition switching intent signal. This signal indicates that the current primary partition may not be able to continue handling subsequent write requests under the current structural conditions, and it is necessary to evaluate whether to perform a structural split. The generation of the partition switching intent signal is automatically completed by the system pressure monitoring process. After each statistical cycle, the system determines whether the current window is in a write overload state and records the number of consecutive overloaded time windows. When the number of consecutive overloaded windows reaches a preset threshold, such as three consecutive windows being in an overloaded state, the state is written to the status judgment result table, and a partition switching intent signal is generated. This signal is transmitted to the partition for evaluation as a prerequisite for subsequent structural judgments. Upon receiving the partition switching intent signal, the system does not immediately perform structural change operations, but further performs multi-dimensional judgments on the current primary partition's operating status to avoid resource waste caused by frequent or unnecessary structural adjustments.

[0090] The first criterion is based on the active period of the primary partition. The active period refers to the total amount of time that the current primary partition has continuously received write requests from its creation to the current time. The system compares this value with the partition lifecycle assessment threshold. When the active period exceeds the set threshold, it indicates that the primary partition has reached its structural boundary and can enter the splittable assessment process.

[0091] The second criterion is the data write rate fluctuation range, which represents the degree of difference between the maximum and minimum number of write requests in the primary partition over several recent time windows. The system calculates the difference between the maximum and minimum number of write requests in the primary partition's five most recent windows based on a sliding write rate sequence, and then normalizes the result according to historical fluctuation levels to determine if the current fluctuation range is abnormal. If the fluctuation range exceeds the historical reference standard, it indicates that the current write behavior is unstable and there may be a risk of short-term write surges, requiring structural adjustments.

[0092] The third criterion is index update density. This metric represents the frequency of index structure updates within the primary partition per unit of time. Each time a data block is written to disk, the system needs to update the data index structure. If, within a certain time period, the number of index structure updates occurring in the primary partition is significantly higher than the system's expected update frequency, it indicates that a large number of data blocks have accumulated within the primary partition, increasing structural complexity and decreasing retrieval efficiency. The index update density is then compared with the index load threshold set by partition management to determine whether a structural split is triggered.

[0093] The comparison results of the above three indicators are logically judged. If any two of the following conditions are met simultaneously: the active period of the primary partition is greater than the life cycle assessment threshold; the data write rate fluctuation exceeds the preset stable range; and the index update density is greater than the index load threshold, then it is considered that the current write partition no longer has the ability to continuously maintain a monolithic structure, meeting the conditions for structural splitting. Subsequently, a primary partition structural splitting operation is performed, dividing the current primary partition into two or more sub-partitions according to the time window structure. Each sub-partition is assigned an independent time range and buffer structure, and write requests will be rerouted to the new write path based on the timestamp and the sub-partition mapping table.

[0094] Taking a real-time industrial data acquisition system as an example, a time window is set to 5 seconds, allowing the main partition to continuously receive write requests for a maximum of 10 minutes, with an index update density not exceeding 5 updates per second. If, during a certain operating cycle, the system records that the write pressure factor is marked as severe for three consecutive time windows, the system then determines that the current main partition has been running for more than 12 minutes, the fluctuation range of the number of write requests in the last five time windows exceeds the normal value by more than 50%, and the index update per unit time has reached 7 times per second. If all the above three judgment dimensions are met, the system immediately performs a main partition structure split, dividing it into 4 sub-partitions, corresponding to the next 20 seconds, 20 seconds, 10 seconds, and the remaining time for write operations, respectively. At this time, the sub-partition structure will take over the write request receiving task of the main partition, avoiding abnormal system response or data backlog due to excessive load on a single partition structure.

[0095] It should be noted that the partition switching intent signal is used to identify the state where the current primary partition structure may need to be adjusted; the active period refers to the actual running time of the primary partition continuously receiving written data since its creation; the data write rate fluctuation range represents the range of changes in the number of write requests in adjacent time windows, reflecting the stability of the data flow; the index update density is the frequency of index updates within the partition per unit time, reflecting the complexity of the storage structure and the frequency of writes; the structure splitting operation is when the system splits the current write partition into multiple new partitions with independent time ranges and cache spaces, and reconstructs the time mapping and write path relationship.

[0096] The structural response mechanism to write pressure fluctuations requires the system to sample the time span and data organization status of the current main partition after triggering the main partition split operation, and to perform partition reconstruction based on the rate change trend of the current write cycle to generate multiple sub-partition structures to optimize subsequent data reception and cache scheduling efficiency.

[0097] Specifically, the data organization parameters of the current primary partition within a set write cycle are collected, the primary partition is divided into sub-partitions, and the time range of each sub-partition is determined based on the trend of write rate changes. The implementation steps are as follows:

[0098] Locate the current primary partition's write time period, which is the time interval covered from the establishment of the primary partition to the current split trigger time. This time interval generally covers several time windows. Each sliding window has recorded data organization parameters such as the number of write requests, data volume, and index update density. These data organization parameters are used to describe the write activity, structural complexity, and timing density within the primary partition. The sampling method can be periodic timed pull or real-time collection based on the split signal trigger.

[0099] After collecting the above parameters, the system uses the trigger time of the split operation as a reference point and traces back to the previous complete write cycle, such as a predefined time length of 10 seconds, 30 seconds, or 60 seconds, as the basic time interval required for this split operation. This basic time interval will be split by the system according to the set time interval, usually into two or more consecutive, non-overlapping time periods. Each time period corresponds to a new sub-partition. The time range identifier of each sub-partition is defined by a timestamp comparison method, that is, the start timestamp and the end timestamp together form a unique time range key to ensure that the write request can be accurately matched in subsequent operations.

[0100] While partitioning into sub-partitions, the time span of each sub-partition is dynamically adjusted based on the write rate trend before the split. If the write rate in a certain time period is significantly higher than in other time periods (e.g., a significant increase in the average number of write requests per second), a shorter time range is allocated to that sub-partition to avoid single-point load concentration. Conversely, during periods of relatively stable write activity, a relatively longer sub-partition span can be set to reduce redundant indexes and structural resource waste. Each newly created sub-partition is assigned an independent write path, corresponding to a specific data buffer area and logical storage channel for subsequent data reception and caching. Write path configuration includes buffer initialization, memory space mapping, and pre-write index table creation, ensuring that data write tasks have independent execution capabilities at the sub-partition level. The path information and time range identifiers of all sub-partitions are jointly recorded in the sub-partition write mapping table.

[0101] For example, taking an industrial IoT device as an example, its acquisition frequency is hundreds of write requests per second. The system sets the maximum running cycle of each main partition to 30 seconds and the sliding window length to 5 seconds. When the write pressure factor of three consecutive sliding windows is detected to be abnormal, the main partition split operation is executed.

[0102] The write rate change trend refers to the fluctuation pattern of the number of write requests per unit time within a certain write cycle, which determines the sub-partition granularity design. The sub-partition write mapping table is a mapping table that records the time interval of the sub-partition and its corresponding write path, and is used for request location and path scheduling.

[0103] Establish a timestamp mapping table for sub-partitions, and route write requests to the corresponding sub-partition buffer areas based on time matching results and write load distribution, including the following steps:

[0104] First, while splitting the primary partition structure, each newly created sub-partition is assigned a unique time range identifier. This time range identifier consists of a start timestamp and an end timestamp, representing the data writing period that the sub-partition can accept. For example, if a primary partition is divided into three sub-partitions from time T0 to time T30, the following time range identifiers may be formed: Sub-partition 1 (T0 to T10), Sub-partition 2 (T10 to T20), and Sub-partition 3 (T20 to T30). These time range identifiers and their corresponding sub-partition buffer area addresses are registered together in a timestamp mapping table to establish a time-path matching relationship.

[0105] When an actual write request arrives, the system first extracts the timestamp field from the write request. This timestamp is usually assigned by the write task initiator or the acquisition device and reflects the actual time the data was generated. After extracting the timestamp, the system uses this timestamp as a keyword to sequentially search for the corresponding time range identifier in the timestamp mapping table. The matching principle is to determine whether the timestamp value falls between the start and end timestamps of a certain sub-partition. If it belongs to a certain interval, the target sub-partition of the write request is initially determined.

[0106] To prevent a subpartition from becoming a load bottleneck due to frequent matching under high load conditions, the system also needs to determine the current buffer usage of the target subpartition. The buffer usage rate is a dynamic parameter, usually obtained by calculating the proportion of the amount of data written in the current buffer to the upper limit of the buffer capacity. When the utilization rate does not reach the preset write saturation threshold (such as 80% or set according to the specific system situation), the system determines that the sub-partition has the ability to accept new write requests. Once the target sub-partition is successfully matched and the buffer utilization rate is in an acceptable state, the system immediately performs the routing operation of the write request. Specifically, the data payload in the write request is redirected to the buffer channel bound to the target sub-partition, and the corresponding write instruction set is called to complete the data writing action. At the same time, the system can also record the write behavior in the sub-partition's index structure and write history table after the write is completed to ensure the integrity and consistency of subsequent data access and backtracking. In some special cases, such as when the target sub-partition buffer is close to saturation or the write density rises rapidly in a short period of time, a temporary adjustment can be made based on the load balancing strategy, that is, the write request is routed to the adjacent sub-partition buffer with lower load. In this case, the system needs to mark the non-strict time matching operation so that it can be compensated in subsequent index retrieval or structure merging to ensure that the global consistency of the time series is not destroyed.

[0107] Taking a practical application as an example, in a smart industrial edge node, high-frequency sensors collect and upload monitoring information at a rate of hundreds of data write requests per second. Within a certain period, the system divides the collection time (e.g., 10 seconds) into three sub-partitions, each corresponding to a write time range and buffer channel. When the system receives a request with a timestamp of 3.27 seconds, it first matches the timestamp to be within sub-partition 1 (T0 to T4); then it detects that the buffer utilization rate of this sub-partition is only 55%, lower than the set 75% threshold, confirming that the write is acceptable; finally, the request is written to the buffer of sub-partition 1, and the behavior is recorded in the index structure.

[0108] The above steps ensure that write requests are reasonably matched in time and balanced and controllable in load, thereby effectively avoiding system bottlenecks caused by data accumulation, path conflicts, or structural inconsistencies in traditional high-frequency write scenarios. By combining the design of the timestamp mapping table with the dynamic write path scheduling mechanism, this invention can enhance the high-frequency acquisition system's ability to cope with write peaks while ensuring timing consistency, and improve the reliability and efficiency of overall time-series data processing.

[0109] Perform write activity monitoring and write latency statistics on each sub-partition, and generate partition structure merging evaluation factors, including the following steps:

[0110] Specifically, the system first continuously records every write operation completed by the sub-partition within each write cycle (e.g., 1 minute, 5 seconds, etc., configured by the system). This includes the time to receive the write request and the actual completion time. The system calculates the time from when the request enters the buffer queue to when the data is successfully written, using each write request as the object. This time is recorded as the response time of the request. All response times are temporarily stored in the statistical cache. After the cycle ends, the system performs statistical processing on the batch of response times to calculate the average write latency of the current sub-partition. This parameter reflects the smoothness of the sub-partition's write link and the level of load impact. It also needs to collect the number of times the write buffer of the sub-partition performs data refresh operations within the current cycle. Data refresh refers to the process of writing the temporary data in the buffer to the underlying storage or structured index in batches. It is usually triggered after the buffer occupancy reaches a certain proportion. The number of buffer refreshes can indirectly reflect the write density and data accumulation of the sub-partition within the cycle. If a partition refreshes frequently, it usually indicates that its write activity is relatively active, and vice versa.

[0111] The three statistical parameters obtained—subpartition write count, average write latency, and buffer refresh count—are used as basic feature inputs, and combined with pre-defined behavioral feature judgment rules to generate a merged evaluation factor. The judgment rules can be configured by the system administrator according to different business scenarios. For example, in industrial scenarios, low write frequency (e.g., less than 10 times within a period), high average write latency (e.g., exceeding a set threshold), and very few buffer flush operations (e.g., only 1 time) can jointly constitute the judgment conditions for "low-activity" sub-partitions. The system matches these rules with statistical values ​​and outputs an evaluation factor to describe the stability of the current write state of the sub-partition, usually expressed as a score or grade. The lower the score, the closer the sub-partition is to a mergeable state. After completing the evaluation of all sub-partitions, the system will compare the write behavior characteristic parameters between adjacent sub-partitions. The parameters include write frequency, average write latency, and buffer flush frequency. The comparison method is to calculate the absolute difference between the corresponding parameters of each pair of sub-partitions and determine whether these differences are all within the preset tolerance range. For example, the difference in write frequency does not exceed 5 times, the difference in latency does not exceed 0.2 seconds, and the difference in flush frequency does not exceed 1 time. The tolerance range can be set based on training and calibration of historical running samples or directly given by engineering rules.

[0112] When it is detected that a pair of adjacent sub-partitions not only have their own merge evaluation factors lower than the merge reference value calibrated by the system (this reference value can represent the threshold for determining the minimum activity of the sub-partition), but also that the differences between the three types of behavioral characteristic parameters are all within the tolerance range, the system determines that the group of sub-partitions meets the structural merge conditions, and performs the merge scheduling based on the determination result, and starts the structural merge process of the sub-partitions. The structural merge includes operations such as unifying the time range identifier, merging the contents of the buffer queue, updating the timestamp mapping table and index path, etc., to ensure that the merged new sub-partition still has complete traceability and write consistency.

[0113] For example, a high-frequency acquisition system monitors sub-partitions A and B within a certain write cycle. It finds that A completes 8 writes with an average latency of 0.5 seconds and 1 refresh; B completes 7 writes with an average latency of 0.52 seconds and 1 refresh. The system's preset tolerance ranges are within 5 writes, within 0.1 seconds of average latency, and within 1 refresh. After calculation, it is found that the differences between the two are within the tolerance range, and their respective merge evaluation factors are both lower than the reference value of 0.3. Based on this, the system triggers the merge process, merging A and B into a new sub-partition C, unifying its write path and time identifier, and reconstructing the sub-partition mapping table to ensure that subsequent write operations can be correctly routed.

[0114] The above steps achieve dynamic adaptive adjustment of the sub-partition structure through multi-dimensional parameter monitoring and rule determination. This not only improves storage space utilization and reduces index structure complexity, but also enhances the system's elastic control capability in the face of high-frequency write environments, avoiding problems such as performance degradation or low data retrieval efficiency caused by the long-term existence of low-activity sub-partitions.

[0115] Based on the changing trend of the partition structure merging evaluation factor, dynamically adjust the sub-partition structure reclamation threshold and the index structure reconstruction triggering condition to keep the sub-partition structure in harmony with the current write load. This includes the following steps:

[0116] The system continuously tracks and calculates the partition structure merge evaluation factor for each subpartition during each write cycle. This factor reflects the write activity and stability of the subpartition in the current cycle. The calculation is based on statistical values ​​such as the number of writes, average write latency, and buffer flushes. The system maintains a periodic sliding window to monitor the changing trend of the evaluation factor for each subpartition over multiple cycles. If the system detects that the merge evaluation factor of one or more adjacent subpartitions is lower than the system-set stability threshold for several consecutive cycles (indicating that the subpartition is continuously in a low-activity, stable write state), then this trend is marked as a structural stability trend.

[0117] Compare the write behavior characteristics parameters between adjacent sub-partitions, including the number of writes, write latency, and buffer refresh count. If the comparison results show that the differences in these parameters are all less than the tolerance range set by the system, it means that these sub-partitions have shown highly consistent load characteristics and write behavior in actual operation. At this time, they are judged to be in a state of feature convergence.

[0118] When the system simultaneously detects that both the structural stability trend and the characteristic convergence state are satisfied, it considers that the current sub-partition structure has lost the necessity to maintain partition granularity and will trigger an index structure reconstruction operation. This operation first merges the time range identifiers of the target sub-partitions and uniformly organizes their routing paths, storage block mapping relationships, and timestamp mapping tables in the underlying index structure. Simultaneously, the system clears the independent buffers of the merged sub-partitions and uniformly routes subsequent write requests to the buffer path corresponding to the primary partition. Throughout the process, the system must ensure the atomicity of the merging process to prevent index corruption or data loss during concurrent writes or queries. The index reconstruction operation includes merging data blocks in the sub-partitions, updating the time index structure, and releasing invalid path identifiers.

[0119] After the index structure is reconstructed and all sub-partitions are successfully merged, the system automatically restores the current primary partition structure, forming a new write start state. If the write pressure factor tends to increase again in the future, the splitting process can be restarted according to the aforementioned mechanism to achieve dynamic and iterative adjustments to the structure.

[0120] For example, after monitoring sub-partitions X and Y for five consecutive write cycles, a system found that X and Y each had fewer than five write operations per cycle, average write latency less than 0.3 seconds, and only one buffer refresh per cycle. The evaluation factor remained below the preset stability threshold of 0.25 throughout the five cycles. Furthermore, the write operations of X and Y differed by no more than two, the latency by no more than 0.05 seconds, and the refresh count was identical, all falling within the system's tolerance range. Based on this, it was determined that X and Y were in a stable and converging state, and an index reconstruction was immediately triggered. The two were merged into a new primary partition Z, and the buffer resources corresponding to X and Y were released. Simultaneously, the time range mapping and index path table of Z were updated to ensure that subsequent write operations uniformly access the Z path.

[0121] It should be noted that in this embodiment, "system" refers to a high-frequency acquisition system. The threshold information in this embodiment was set in advance by professionals and will not be explained in detail here. In this embodiment, some parameters may have the same English letters, but they have different meanings when used and will not be explained one by one here.

[0122] This invention constructs a dynamic time-series data storage mechanism based on write pressure sensing and partition structure linkage, realizing real-time response and adaptive structural adjustment to sudden write loads in high-frequency acquisition systems. By collecting the timestamp and data length of each write request, a sliding write rate sequence is constructed, and the write rate change slope and buffer queue length change rate are extracted to generate a write pressure factor and identify write overload states. Based on the evolution trend of the write pressure factor, a partition switching intention signal is triggered, and the main partition active cycle, write rate fluctuation amplitude, and index update density are combined to determine whether to perform a structure splitting operation, dynamically generating a sub-partition structure with time boundaries and independent write paths.

[0123] During data redirection, a timestamp mapping table is constructed and combined with the buffer utilization rate of the target sub-partition to achieve load balancing of write requests. By performing write activity and write latency statistics on the sub-partitions, a merging evaluation factor to describe the stability of the sub-partitions is constructed. Based on the trend changes of the merging evaluation factor, the structure reclamation threshold and index reconstruction conditions are dynamically adjusted to achieve continuous matching and resource coordination between the sub-partition structure and the write load. This method effectively solves the problems of rigid storage structure and unbalanced resource scheduling caused by strong write bursts and drastic load fluctuations in existing high-frequency acquisition systems, and significantly improves the structural flexibility, real-time response, and data write efficiency of the system in high-concurrency data write scenarios.

[0124] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0125] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0126] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0128] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0129] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for partitioning and storing time-series data in a high-frequency acquisition system, characterized in that: Includes the following steps: The system collects raw write requests from high-frequency data streams in real time, organizes the request data according to the write time, and constructs a write rate sequence within a time window. Extract the slope of the write rate change and the rate of change of the queue length of the write buffer from the sliding write rate sequence, calculate the write pressure factor, and identify whether there is a write overload state. Construct a partition switching intent signal based on changes in the write pressure factor, and determine whether the currently being written primary partition meets the conditions for triggering a structural split. Collect the data organization parameters of the current primary partition within the set write cycle, divide the primary partition into sub-partitions, and determine the time range of each sub-partition based on the trend of write rate changes; Establish a timestamp mapping table for sub-partitions, and route write requests to the corresponding sub-partition buffer area based on the time matching results and write load distribution. Perform write activity monitoring and write latency statistics for each sub-partition, and generate partition structure merging evaluation factors; Based on the changing trend of the partition structure merging evaluation factor, the sub-partition structure reclamation threshold and the index structure reconstruction triggering condition are dynamically adjusted to keep the sub-partition structure in harmony with the current write load. Based on the changes in the write pressure factor, a partition switching intention signal is constructed, and it is determined whether the currently being written to the primary partition meets the conditions for triggering a structural split. The specific steps include: Upon determining that a sudden increase in write load has occurred, a switching signal is generated to indicate that the current write path needs to be adjusted. The switching signal is compared with the active period of the current write partition, the fluctuation range of the data write rate, and the index update density; If the comparison result meets the triggering condition, then a splitting operation is performed to divide the currently written partition into sub-partitions.

2. The method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 1, characterized in that: The process involves real-time acquisition of raw write requests from high-frequency data streams, organization of the request data according to write time, and construction of a write rate sequence within a time window. Specific steps include: Collect the timestamp and data length value of each write request; Calculate the total number of write requests and the total amount of data within a fixed-length time window. Based on the statistical results of continuous windows, a sliding write rate sequence is constructed to reflect changes in write frequency. A time window is a fixed-length period of time that is updated over time and is used to track write requests.

3. The method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 2, characterized in that: Based on the statistical results of continuous windows, a sliding write rate sequence reflecting changes in write frequency is constructed. The specific steps include: By comparing the number of write requests in two adjacent time windows, the trend of the number of write requests in the current time period can be obtained. The statistical results within a continuous time window are arranged in chronological order to form a sliding write rate sequence that reflects changes in the number of write requests.

4. The method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 3, characterized in that: Extract the slope of the write rate change and the rate of change of the write buffer queue length from the sliding write rate sequence, calculate the write pressure factor, and identify whether a write overload state exists. Specific steps include: Calculate the slope of the change in write rate between adjacent time windows; Calculate the rate of change of the buffer queue length within adjacent time windows; The slope of change and the rate of change are normalized and then multiplied to generate the pressure factor. When the write pressure factor exceeds the preset write pressure threshold, the current window is marked as write overloaded.

5. A method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 4, characterized in that: Collect the data organization parameters of the current primary partition within a set write cycle, divide the primary partition into sub-partitions, and determine the time range of each sub-partition based on the trend of write rate changes. This includes the following steps: Based on a complete write cycle before the split operation is triggered, the current write time period is divided into two or more sub-partitions at fixed time intervals. Each sub-partition is assigned a non-overlapping time range identifier and an independent data write path for subsequent time mapping and buffer scheduling.

6. The method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 5, characterized in that: Establish a timestamp mapping table for sub-partitions, and route write requests to the corresponding sub-partition buffer areas based on time matching results and write load distribution, including the following steps: Based on the timestamp value of each write request, find the matching time range identifier in the timestamp mapping table; The timestamp mapping table is used to record the mapping relationship between the corresponding time period of the sub-partition and the timestamp range; Based on the current buffer usage rate of the target sub-partition, determine whether the write condition is met; When the conditions are met, the write request is redirected to the target subpartition and the write operation is performed.

7. A method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 6, characterized in that: Perform write activity monitoring and write latency statistics on each sub-partition, and generate partition structure merging evaluation factors, including the following steps: Collect the cumulative number of writes completed by the sub-partition in the current write cycle, calculate the response time of each write request, and determine the average write latency of the sub-partition based on the statistical results of all response times. The number of buffer refreshes completed by writing data to the buffer in a sub-partition within a period; Based on the number of writes, average write latency, number of buffer refreshes, and preset behavioral characteristic judgment rules, a merged evaluation factor is generated to describe the stability of the write status of the sub-partition. When the merge evaluation factor is lower than the calibrated merge reference value, and the differences in the number of writes, average write latency, and number of buffer refreshes between adjacent sub-partitions are within the preset tolerance range, the adjacent sub-partitions are determined to enter the structure merge process. Here, "group" refers to a group of sub-partitions that are currently selected for comparison, are adjacent to each other, and simultaneously meet the merge conditions.

8. A method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 7, characterized in that: Based on the changing trend of the partition structure merging evaluation factor, dynamically adjust the sub-partition structure reclamation threshold and the index structure reconstruction triggering condition to keep the sub-partition structure in harmony with the current write load. This includes the following steps: When the partition merge evaluation value is continuously lower than the preset stability threshold, and the difference in write activity between adjacent sub-partitions is less than the specified tolerance range, the index reconstruction operation is triggered. After index reconstruction is performed, the data from each sub-partition is merged into the primary partition, and the structural state is restored.

9. A method for partitioning and storing time-series data in a high-frequency acquisition system according to claim 8, characterized in that: Index reconstruction operations include merging data blocks in sub-partitions, updating time-based index structures, and releasing invalid path identifiers.

Images