An Incremental Adaptive Downsampling Method for Time-Series Data

Abstract

This application provides an incremental adaptive downsampling method for time-series data during write operations, belonging to the field of data processing technology. Specifically, it includes: triggering bucket partitioning determination when writing raw time-series data; maintaining the current open bucket and extracting the P5 and P95 quantile values ​​of the data within the bucket; calculating the fluctuation amplitude based on the quantile range; determining the bucket boundary using a dual-dimensional condition of a time upper limit and a fluctuation threshold; generating and storing a summary result containing the first value, P5 value, P95 value, last value, and their corresponding timestamps when the bucket closes; and dynamically updating the fluctuation threshold using an exponentially weighted moving average method based on the fluctuation amplitude of the closed bucket. This invention moves downsampling from the query side to the write side, allowing direct reading of the summary result during queries, avoiding repeated scanning of the original data. Furthermore, the threshold can automatically evolve according to data characteristics, effectively resisting transient spikes and requiring minimal deployment.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an adaptive downsampling method for write-time incremental data. Background Technology

[0002] Currently, in scenarios such as high-performance computing clusters, cloud platform monitoring systems, and data center resource monitoring systems, metrics such as CPU utilization, memory usage, and network traffic are typically collected continuously at intervals of 5 to 60 seconds and written to the database in the form of time-series data.

[0003] Existing downsampling schemes can be mainly divided into three categories. The first category is query-side downsampling schemes, represented by the LTTB algorithm. These read the entire original data and perform downsampling during a query. The drawback is that the entire data needs to be rescanned for each query, and the query overhead increases significantly with the range. The second category is write-time pre-aggregation schemes, which generate aggregation results in advance according to a fixed time window. The drawback is that bucket partitioning is rigid, key parameters rely on manual configuration, different metrics require separate parameter tuning, and the same metric is prone to parameter failure at different times. The third category is fixed-threshold adaptive schemes, which adjust the sampling ratio or strategy by setting a preset threshold. The drawback is that the threshold is still manually set, failing to form a closed-loop mechanism for the threshold to automatically evolve with data fluctuations. Furthermore, calculating fluctuation amplitude based on maximum and minimum values ​​is susceptible to instantaneous spikes, and some schemes rely on GPU / FPGA, Spark clusters, or specific databases, which is not conducive to low-cost deployment.

[0004] It is evident that there is an urgent need for an adaptive downsampling method for time-series data with high sampling efficiency, adaptability, and stability during write operations. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an adaptive downsampling method for write-time incremental data, which at least partially solves some of the problems existing in the prior art.

[0006] This invention provides a write-time incremental adaptive downsampling method for time-series data, comprising: Step 1: Obtain the raw time-series data corresponding to the CUP monitoring indicators, wherein the raw time-series data includes at least a sequence identifier, a timestamp, and a numerical value; Step 2: Locate the corresponding current open bucket based on the sequence identifier. If the current open bucket does not exist, use the current original time series data as the starting data of the new bucket, set the bucket start time, record the first value, initialize the buffer, and set the count in the bucket to 1. Step 3: If the current open bucket already exists, include the current original time series data into the buffer of the current open bucket, update the count in the bucket, sort the data in the buffer by value, extract the P5 quantile value and corresponding timestamp, the P95 quantile value and corresponding timestamp of the current bucket, and calculate the fluctuation amplitude of the current bucket, wherein the fluctuation amplitude is the difference between the P95 quantile value and the P5 quantile value. Step 4: Perform a two-dimensional bucket boundary determination on the current open bucket: if the difference between the current timestamp and the bucket start time is greater than the maximum bucket duration threshold, then the current open bucket is determined to be closed; or, if the difference between the current timestamp and the bucket start time is greater than the minimum cumulative duration threshold and the fluctuation amplitude is greater than the current fluctuation threshold, then the current open bucket is determined to be closed. Step 5: When it is determined that the current open bucket is closed, generate the bucket summary result of the closed bucket and write it into the downsampling result storage. The bucket summary result includes the bucket start time, bucket end time, first value, P5 quantile, timestamp corresponding to the P5 quantile, P95 quantile, timestamp corresponding to the P95 quantile, and last value. Step 6: Update the current fluctuation threshold according to the fluctuation range of the closed bucket using an exponentially weighted moving average method; Step 7: Use the current original time series data that triggered the closure as the starting data for the next open bucket, and return to Step 2.

[0007] According to a specific implementation of an embodiment of the present invention, step 4 specifically includes: Time dimension determination: when When the current open bucket is determined to be closed, wherein, This is the current timestamp. The start time of the bucket. The maximum bucket duration threshold; Fluctuation dimension determination: when and When the current open bucket is determined to be closed, wherein, The minimum cumulative duration threshold, This represents the current fluctuation range of the bucket. This represents the current fluctuation threshold. The time dimension determination and the fluctuation dimension determination are related by OR logic. If either condition is met, the current open bucket is determined to be closed.

[0008] According to a specific implementation of an embodiment of the present invention, the bucket summary result further includes a sequence identifier, which is used to distinguish different monitoring indicator sequences, and each sequence independently maintains the corresponding open bucket state and fluctuation threshold.

[0009] According to a specific implementation of an embodiment of the present invention, the P5 quantile value is the quantile value located at the 5th percentile after the data in the buffer is sorted numerically, and the P95 quantile value is the quantile value located at the 95th percentile after the data in the buffer is sorted numerically.

[0010] According to a specific implementation of the present invention, the expression for updating the current fluctuation threshold is: ; in, The updated fluctuation threshold, The fluctuation threshold before the update. This represents the fluctuation range of the current closed bucket. For smoothing coefficients, ; The updated fluctuation threshold is set to be no less than the preset lower threshold. .

[0011] According to a specific implementation of an embodiment of the present invention, the state variable of the currently open bucket includes: bucket start time. First value Buffer Counting inside the bucket Each item in the buffer is The binary tuple, wherein the current fluctuation threshold is assigned an initial value when the sequence first enters the processing flow. .

[0012] According to a specific implementation of an embodiment of the present invention, after step 6, the method further includes a query step: When a trend query request within a specified time range is received, the original time series data within the corresponding time range is not scanned. Instead, the corresponding bucket summary result is read directly from the downsampling result storage based on the sequence identifier and the bucket start time range. During front-end rendering, the first value, P5 quantile, P95 quantile, and last value in each bucket summary result are sorted according to their respective timestamps and then connected sequentially to restore the trend curve.

[0013] The write-time incremental adaptive downsampling scheme for time-series data in this embodiment of the invention includes: Step 1, obtaining the original time-series data corresponding to the CUP monitoring index, wherein the original time-series data includes at least a sequence identifier, a timestamp, and a value; Step 2, locating the corresponding current open bucket according to the sequence identifier; if the current open bucket does not exist, using the current original time-series data as the starting data of the new bucket, setting the bucket start time, recording the first value, initializing the buffer, and setting the count in the bucket to 1; Step 3, if the current open bucket already exists, incorporating the current original time-series data into the buffer of the current open bucket, updating the count in the bucket, sorting the data in the buffer by value, extracting the P5 quantile and corresponding timestamp, the P95 quantile and corresponding timestamp of the current bucket, and calculating the fluctuation amplitude of the current bucket, wherein the fluctuation amplitude is the difference between the P95 quantile and the P5 quantile; Step 4: Perform a two-dimensional bucket boundary determination on the current open bucket: if the difference between the current timestamp and the bucket start time is greater than the maximum bucket duration threshold, then the current open bucket is determined to be closed; or, if the difference between the current timestamp and the bucket start time is greater than the minimum cumulative duration threshold and the fluctuation amplitude is greater than the current fluctuation threshold, then the current open bucket is determined to be closed. Step 5: When the current open bucket is determined to be closed, generate the bucket summary result of the closed bucket and write it into the downsampling result storage, wherein the bucket summary result includes the bucket start time, bucket end time, first value, P5 quantile, timestamp corresponding to the P5 quantile, P95 quantile, timestamp corresponding to the P95 quantile, and last value. Step 6: Update the current fluctuation threshold according to the fluctuation amplitude of the closed bucket using an exponentially weighted moving average method. Step 7: Use the current original time series data that triggered the closure as the starting data of the next open bucket, and return to Step 2.

[0014] The beneficial effects of the embodiments of the present invention are as follows: 1) The query path is lightweight, resulting in a significant improvement in query performance.

[0015] This invention moves downsampling calculation from the query side to the write side. During querying, only the generated bucket summary results need to be read directly, without scanning the full original data. In the embodiment, under the condition of querying the entire 7-day period, the query time of this invention is about 1.4ms, while the query-side LTTB is about 52-56ms, and the fixed window aggregation is about 7-9.5ms, showing a significant improvement in query performance, and maintaining low time consumption even after the query range is expanded.

[0016] 2) Threshold adaptive evolution, eliminating the need for manual parameter tuning across indicators.

[0017] This invention employs an EWMA mechanism to dynamically update the fluctuation threshold based on the actual fluctuation amplitude of the closed-loop data. The threshold can automatically evolve with the fluctuation characteristics of the data itself. In the embodiments, under exactly the same parameter settings, the threshold converges to approximately 46.80 in the CPU scenario and to approximately 6.70 in the memory scenario, indicating that the threshold can automatically adapt to indicators with different dimensions and fluctuation patterns, avoiding the tedious operation of presetting thresholds for each indicator separately.

[0018] 3) Strong resistance to transient burr interference and stable barrel boundary determination.

[0019] This invention uses the P95-P5 quantile range instead of the traditional maximum minus minimum value to characterize the fluctuation within the bucket. A single abnormal spike will only affect a very small number of positions after sorting and will not directly amplify the range estimate of the whole bucket. It has stronger stability in monitoring time series data containing spikes and spikes and can effectively avoid misjudgment of bucket boundaries caused by instantaneous anomalies.

[0020] 4) The trend fidelity capability is superior to existing solutions.

[0021] With the same output point budget, this invention demonstrates better trend preservation capabilities compared to query-side LTTB and fixed-window aggregation. In the embodiments, under CPU conditions, this invention achieves an NRMSE of 0.2887, superior to LTTB's 0.5476 and fixed-window aggregation's 0.4082; under memory conditions, this invention achieves an NRMSE of 0.5785, superior to LTTB's 1.4624 and fixed-window aggregation's 1.4853.

[0022] 5) No need to readjust parameters when the query range is expanded, and it has good adaptive stability.

[0023] The output points of this invention naturally expand with the query range. When the query range expands from 1 day to 7 days, no parameters need to be reset, and the error remains stable. In contrast, existing solutions show a significant increase in error if the original parameters are used. In the embodiment, under CPU conditions, the NRMSE of this invention remains at approximately 0.2865, while LTTB degenerates to 0.7429, and fixed window aggregation degenerates to 0.6441.

[0024] 6) Low deployment requirements and good project implementation.

[0025] This invention relies only on the basic capabilities of a general data storage system and the application-layer state machine logic, without depending on dedicated GPU / FPGA hardware, distributed computing clusters, or specific database products. It can be deployed at low cost on a conventional software stack and has wide applicability. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating an adaptive downsampling method for write-time increments of time-series data provided in an embodiment of the present invention; Figure 2 A system framework diagram of a write-time incremental adaptive downsampling method for time-series data provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the specific implementation process of an adaptive downsampling method for write-time increments of time-series data provided in an embodiment of the present invention. Detailed Implementation

[0028] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0030] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this invention, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0031] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0033] In scenarios such as high-performance computing clusters, cloud platform monitoring systems, and data center resource monitoring systems, metrics such as CPU utilization, memory usage, and network traffic are typically collected continuously at intervals of 5 to 60 seconds and written to the database as time-series data. The front-end monitoring panel needs to display trend curves in real time according to the user-specified time range. As the retention period increases, the amount of raw time-series data expands rapidly. Directly plotting the raw data on the front end can lead to slow queries, heavy front-end rendering burden, and unresponsive interactions. Therefore, downsampling is generally required in engineering practice.

[0034] Regarding downsampling of time-series data, currently available solutions can be broadly categorized into five areas: query-side downsampling algorithms, write-time pre-aggregation solutions, prediction model downsampling, distributed big data processing solutions, and adaptive threshold or dynamic sampling solutions.

[0035] Current technologies have at least the following five drawbacks: 1) The query-side approach repeatedly scans the original data, and the query overhead increases significantly as the query range expands. Non-write incremental generation of downsampled data is used. Query-side downsampling schemes essentially require rereading the full amount of raw data within a specified time range when a user queries, before performing sorting, point selection, and return. For scenarios like monitoring systems that involve high-frequency scaling and frequent replays, the larger the query range, the higher the scanning and computation costs, and the more unstable the query latency.

[0036] 2) The bucket partitioning method is rigid, key parameters rely on manual configuration, and it is difficult to reuse them across indicators. Most existing write-time pre-aggregation schemes use fixed time windows, while fixed threshold schemes rely on manually preset thresholds. For metrics such as CPU, memory, and network traffic, which have significantly different dimensions and fluctuation patterns, separate parameter tuning is often required. Furthermore, the original parameters for the same metric can easily become invalid at different times.

[0037] 3) Insufficient adaptability to changes in data format and query scope. Existing solutions typically perform well only under a certain typical fluctuation pattern or a fixed query range. When the query range expands from 1 day to 7 days or even longer, the query-side solution often needs to reset the number of output points or buckets; when the data moves from a stable period to a period of sharp fluctuation, fixed window or fixed threshold solutions struggle to adjust the bucket density in a timely manner.

[0038] 4) Fluctuation metrics based on rawmin / max are easily affected by transient spikes. If the maximum and minimum values ​​within the bucket are used directly to calculate the fluctuation amplitude, a single outlier can significantly amplify the fluctuation estimate within the bucket, causing misjudgment of the bucket boundary and further interfering with subsequent threshold updates, resulting in poor system stability in monitoring scenarios with spikes and glitches.

[0039] 5) Some existing solutions are heavily reliant on infrastructure and have deep technology stack bindings. Some solutions rely on dedicated hardware such as GPUs / FPGAs, while others are tied to specific time-series databases, message middleware, or Spark clusters, which is not conducive to low-cost deployment in general databases and conventional application service stacks.

[0040] This invention provides a write-time incremental adaptive downsampling method for time-series data. The method can be applied to the CPU monitoring index data downsampling process in scenarios such as high-performance computing clusters, cloud platform monitoring systems, and data center resource monitoring systems.

[0041] See Figure 1 This is a flowchart illustrating an adaptive downsampling method for write-time increments of time-series data provided in an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method mainly includes the following steps: Step 1: Obtain the raw time-series data corresponding to the CUP monitoring indicators, wherein the raw time-series data includes at least a sequence identifier, a timestamp, and a numerical value; In practice, the data acquisition agent can periodically collect target monitoring indicators, generate raw data records containing series_id, timestamp ts, and value, and write them into the time-series raw data storage.

[0042] For example, the data acquisition agent collects CPU usage and memory usage data every 5 seconds and organizes each sampled data into a triplet of series_id, ts, and value, which is then written to metrics_raw. The series_id is used to distinguish different monitoring sequences (including cpu_usage and memory_usage), ts is used to identify the sampling time point, and value is used to identify the monitoring value corresponding to that time point.

[0043] In this embodiment, the CPU utilization sequence and memory usage sequence are entered into the subsequent processing flow as two independent monitoring sequences. The system maintains the corresponding open bucket status and fluctuation threshold for different series_id.

[0044] Step 2: Locate the corresponding current open bucket based on the sequence identifier. If the current open bucket does not exist, use the current original time series data as the starting data of the new bucket, set the bucket start time, record the first value, initialize the buffer, and set the count in the bucket to 1. In practice, the bucket status maintenance module reads the current open bucket status corresponding to the series_id. If there is no open bucket at present, the data is used as the starting data of the new bucket, the bucket start time T0 is set, the first value first_val is recorded, the buffer[] is initialized, and the bucket count is set to 1.

[0045] Where T0 represents the starting timestamp of the current open bucket; first_val represents the value of the first data in the current bucket; buffer[] represents the data point buffer in the current bucket, with each item being (value, timestamp); and count represents the cumulative number of data in the current bucket. These state variables are used to support subsequent quantile feature extraction, boundary determination, and closed bucket result generation. When a sequence first enters the processing flow, the system preferably assigns an initial value F_initial to the current fluctuation threshold as the starting threshold for subsequent two-dimensional bucket boundary determination.

[0046] Step 3: If the current open bucket already exists, include the current original time series data into the buffer of the current open bucket, update the count in the bucket, sort the data in the buffer by value, extract the P5 quantile value and corresponding timestamp, the P95 quantile value and corresponding timestamp of the current bucket, and calculate the fluctuation amplitude of the current bucket, wherein the fluctuation amplitude is the difference between the P95 quantile value and the P5 quantile value. In practice, if an open bucket already exists, the new data (Ti, Vi) is added to the current bucket buffer, and the sample count is incremented by 1, so that the current bucket continues to accumulate monitoring data within the same period of time.

[0047] Then, the data in the current bucket buffer is sorted numerically, and the P5 quantile value P5_val and corresponding timestamp P5_ts, as well as the P95 quantile value P95_val and corresponding timestamp P95_ts, are extracted for the current bucket. The fluctuation range of the current bucket is then calculated. ; In this step, the present invention uses the P5 / P95 quantile values ​​instead of rawmin / max to characterize the fluctuation within the bucket. The reason for this is that a single abnormal spike will only affect a very small number of positions after sorting, and will not directly amplify the range of the entire bucket like the difference between the maximum and minimum values. Therefore, it is more suitable for resisting transient spike interference in monitoring-type time-series data.

[0048] Step 4: Perform a two-dimensional bucket boundary determination on the current open bucket: if the difference between the current timestamp and the bucket start time is greater than the maximum bucket duration threshold, then the current open bucket is determined to be closed; or, if the difference between the current timestamp and the bucket start time is greater than the minimum cumulative duration threshold and the fluctuation amplitude is greater than the current fluctuation threshold, then the current open bucket is determined to be closed. In practice, the system can simultaneously perform time-dimensional and fluctuation-dimensional judgments on the current bucket: Time dimension determination: If If the current bucket has reached its maximum allowed duration, it should be closed. Fluctuation dimension determination: If and If the current bucket has shown significant fluctuations after the minimum cumulative duration, it should be closed.

[0049] If any of the above conditions are met, the current bucket should be closed; if neither condition is met, the current bucket continues to receive subsequent data.

[0050] in, Indicates the maximum allowed duration for a single bucket, used as a safety valve in the time dimension; This indicates the minimum cumulative duration for which fluctuation judgments are allowed to be executed, used to prevent the bucket from closing too early after it has been opened; This represents the current fluctuation threshold, used to determine whether the fluctuation within the bucket has reached a level where the bucket should be cut. Through this dual-dimensional determination, the bucket boundary is simultaneously controlled by both time constraints and fluctuation constraints.

[0051] Step 5: When it is determined that the current open bucket is closed, generate the bucket summary result of the closed bucket and write it into the downsampling result storage. The bucket summary result includes the bucket start time, bucket end time, first value, P5 quantile, timestamp corresponding to the P5 quantile, P95 quantile, timestamp corresponding to the P95 quantile, and last value. In practice, when the current bucket meets the closure condition, the system organizes the bucket's start time, end time, first value, P5 quantile and timestamp, P95 quantile and timestamp, and last value into an 8-field bucket summary result and writes it into the downsampling result storage.

[0052] The preferred 8-field bucket summary result includes: bucket_start, bucket_end, first_val, P5_val, P5_ts, P95_val, P95_ts, and last_val. In multi-sequence scenarios, series_id can be further included as a sequence identifier field.

[0053] Step 6: Update the current fluctuation threshold according to the fluctuation range of the closed bucket using an exponentially weighted moving average method; In practice, after the closed bucket is written with the result, the threshold adaptive module updates the current fluctuation threshold according to the actual fluctuation range of the bucket (bucket_range) using an exponentially weighted moving average method. ; And preferably set a lower threshold: ; Through this step, the threshold is no longer set statically and manually all at once, but is continuously updated based on the fluctuation characteristics of the most recently closed bucket. When the data fluctuates sharply, F is gradually increased; when the data remains stable, F is gradually decreased; when the data changes from stable to sharply fluctuating, or from sharply fluctuating to stable, the lag effect of EWMA will automatically adjust the bucket width during the transition period.

[0054] in, The smoothing coefficient satisfies ; The threshold before the update; The updated threshold; This is the lower limit of the threshold.

[0055] Step 7: Use the current original time series data that triggered the closure as the starting data for the next open bucket, and return to Step 2.

[0056] In practice, after the threshold update is completed, the bucket state is reinitialized with the current data as the starting data for the next open bucket, so that the system can enter the next round of write judgment and bucket boundary control.

[0057] Furthermore, when the front-end requests the trend curve of a monitoring sequence within a specified time range, the query service directly reads the downsampled results from metrics_summary by the range of series_id + bucket_start, instead of scanning the full original data within the corresponding time range. During rendering, the front-end rendering module sorts the four value points in each bucket by timestamp and connects them sequentially to reconstruct the main trend within the bucket.

[0058] Overall, it forms a continuous closed-loop processing chain of "writing a piece of data, judging the bucket boundary once, updating the threshold once after closing, and directly reading the result during querying".

[0059] The write-time incremental adaptive downsampling method for time-series data provided in this embodiment moves the downsampling calculation from the query side to the write side, so that bucket partitioning and summary result generation are triggered every time raw data is written. During the query phase, only the generated bucket summary results need to be read directly without scanning the entire original data. Therefore, as the query range expands, the query time of this invention remains basically unchanged, while the query time of the query-side LTTB increases linearly with the amount of data. By using the EWMA mechanism to dynamically update the fluctuation threshold according to the actual fluctuation amplitude of the closed bucket, the fluctuation threshold can continuously evolve with the data's own fluctuation characteristics. Therefore, under the same parameter settings, the thresholds for CPU scenarios and memory scenarios can automatically converge to different values, without needing to preset thresholds for indicators with different dimensions and fluctuation patterns. By using the P95-P5 quantile range instead of the maximum minus the minimum value to characterize the fluctuation within the bucket, a single abnormal spike only affects a very small number of positions after sorting and does not directly amplify the entire bucket range estimation. Therefore, in monitoring time-series data containing spikes and spikes, bucket boundary determination is less susceptible to instantaneous interference, resulting in stronger system stability. By simultaneously retaining four key values—the first value, the P5 quantile, the P95 quantile, and the last value—along with their corresponding timestamps in the bucket summary results, the front-end rendering can sequentially connect these values ​​to reconstruct the main trends within the bucket. Therefore, under the same output point budget, this invention has stronger trend fidelity compared to schemes that only retain the average value or a single extreme value, and its NRMSE index is superior to LTTB and fixed window aggregation. By decoupling the time threshold and fluctuation threshold in the bucket boundary determination conditions from the specific query range, the bucket summary results are fixedly stored once generated. When the query range expands, the number of output points naturally expands with the data coverage without resetting parameters. Therefore, when the query range expands from 1 day to 7 days, the error of this invention remains stable, while existing schemes that require preset output point numbers or window numbers experience a significant increase in error if the original parameters are used. By relying only on the basic capabilities of a general data storage system and the application layer state machine logic to achieve incremental downsampling during writes, it does not require dedicated GPU / FPGA hardware, distributed computing clusters, or specific database products. Therefore, it can be deployed at low cost on a conventional software stack and is not bound by specific infrastructure or technology stacks.

[0060] The method of the present invention will be further described below with reference to several specific embodiments: Example 1: Consider a high-performance computing cluster monitoring system that continuously collects CPU utilization and memory usage data from each node and displays trend curves for a specified time range in a web monitoring panel. To verify the method of this invention's ability to adapt parameters uniformly across different data formats, its trend fidelity, and its ability to reduce query-side parameter tuning, a comprehensive experiment was conducted using two typical scenarios: CPU utilization and memory usage. The data scale and scenario characteristics are shown in Table 1.

[0061] Table 1

[0062] The parameters are set uniformly as shown in Table 2.

[0063] Table 2

[0064] See Figure 3 The implementation process of the write-time incremental adaptive downsampling method in this embodiment mainly includes the following steps: Step E1: Periodically collect monitoring indicators and write them into the raw data storage.

[0065] The data acquisition agent collects CPU usage and memory usage data every 5 seconds and writes each sampled data into a triplet of series_id, ts, and value into metrics_raw. The series_id is used to distinguish different monitoring sequences (including cpu_usage and memory_usage), ts is used to identify the sampling time point, and value is used to identify the monitoring value corresponding to that time point.

[0066] In this embodiment, the CPU utilization sequence and memory usage sequence are entered into the subsequent processing flow as two independent monitoring sequences. The system maintains the corresponding open bucket status and fluctuation threshold for different series_id.

[0067] Step E2: Locate the current open bucket for the written data and complete the initial state establishment.

[0068] After a piece of raw data is written, the bucket state maintenance module reads the current open bucket state corresponding to the series_id. If there is no open bucket for the sequence, the data is used as the starting data of the new bucket, the bucket start time T0=ts is set, the first value first_val=value is recorded, the buffer[] is initialized, and the sample count in the bucket is set to 1.

[0069] Furthermore, when a sequence enters the processing flow for the first time, the system preferably initializes the current fluctuation threshold to F_initial=50.0 so that fluctuation determination can be performed after the minimum cumulative duration condition is met.

[0070] Step E3: Incorporate the new data into the current bucket and extract the key quantile features within the bucket.

[0071] If the sequence already has an open bucket, the newly written data (value, ts) is added to the current bucket buffer, and the data in the buffer is sorted by value. P5_val, P5_ts, P95_val, and P95_ts are extracted, and then the fluctuation range of the current bucket is calculated. ; In this embodiment, P95_val-P5_val is used to characterize the fluctuation within the bucket instead of directly using rawmin / max to calculate the range, in order to reduce the interference of individual abnormal peaks or short-term spikes on the determination of the bucket boundary.

[0072] Step E4: Perform bucket boundary determination based on time and fluctuation conditions.

[0073] The system performs both time-dimensional and volatility-dimensional assessments on the current bucket. If If the current bucket has reached its maximum allowed duration, it should be closed; if and If the current bucket has shown significant fluctuations after the minimum cumulative duration, it should be closed.

[0074] In this embodiment, Tmax = 1800 seconds and Tmin = 60 seconds. By setting Tmax, the bucket can be prevented from extending indefinitely in long-term stable scenarios; by setting Tmin, it can be prevented from being accidentally cut off due to local disturbances immediately after the bucket is opened; by setting the current fluctuation threshold F, the bucket width can be dynamically changed according to the degree of data fluctuation.

[0075] Step E5: Close the current bucket, write the bucket summary result, and update the fluctuation threshold.

[0076] When the current bucket meets the closure condition, the system organizes the bucket_start, bucket_end, first_val, P5_val, P5_ts, P95_val, P95_ts, and last_val of the bucket into an 8-field bucket summary result, and writes it together with the series_id into metrics_summary.

[0077] After the result is written, the threshold adaptive module performs an EWMA update based on the fluctuation range of the current closed bucket: ; In this embodiment, , Through the above update method, the system can continuously adjust the threshold based on the fluctuation of the most recently closed bucket, so that the determination of the boundary of subsequent buckets is more in line with the current data state.

[0078] Step E6: Open the next open bucket with the current data and directly read the bucket summary results during the query phase.

[0079] After the threshold update is completed, the bucket state is reinitialized using the current data that triggered the closing of the bucket as the starting data for the next opening of the bucket, allowing the system to enter the next round of write processing. When the front end initiates a query request, the query service no longer scans the full amount of raw data within the corresponding time range, but directly reads the bucket summary results already generated in metrics_summary according to the range of series_id + bucket_start.

[0080] Furthermore, the front end connects the curves in chronological order of the key value points corresponding to bucket_start, P5_ts, P95_ts, and bucket_end to restore the main trend changes within that time range.

[0081] Using the unified parameters shown in Tables 1 and 2, and following the implementation process of the present invention in Example 1, write-time incremental adaptive downsampling processing was performed in both CPU utilization and memory usage scenarios, and the implementation results are shown in Table 3.

[0082] Table 3

[0083] The implementation results show that, under the same parameters, both CPU and memory scenarios can stably complete the write downsampling processing of the present invention. The threshold can automatically converge according to different fluctuation scales, indicating that the present invention has good uniform parameter adaptation capability on different monitoring sequences.

[0084] Example 2: Comparative Experiment 1: Fair Budget Comparison under a 1-Day Query Window To verify the trend fidelity and query performance of this invention under a 1-day query window, this comparative experiment uses the algorithm parameters in Tables 1 and 2, i.e., Tmax, Tmin, alpha, F_initial, and F_min remain unchanged, but the front-end query time range is limited to 1 day. For external comparison methods, to ensure fairness, the query-side LTTB method and the fixed-window aggregation method use the same or as close as possible to the output point count as the method of this invention as comparison constraints. Query-side LTTB and fixed-window aggregation are not part of the implementation process of this invention; they are only used as reference objects for verifying the technical effect. The results of the comparative experiment are shown in Table 4.

[0085] Table 4

[0086] The comparative experimental results show that, within a 1-day query window, when external methods compare the output points with or as close as possible to those of this invention, this invention not only demonstrates better trend fidelity but also lower query time compared to fixed-window aggregation and query-side LTTB. This indicates that moving the downsampling process forward to the writing stage and directly reading the bucket summary results during the query stage helps reduce query overhead while maintaining trend representation capabilities.

[0087] Within a one-day query window, when the output point count and bucket count budgets are aligned with the method of this invention, the NRMSE of this invention is 0.2887 in the CPU scenario, lower than 0.5476 for query-side LTTB and 0.4082 for fixed-window aggregation; in the memory scenario, the NRMSE of this invention is 0.5785, lower than 1.4624 for LTTB and 1.4853 for fixed-window aggregation. Meanwhile, in terms of query time, the method of this invention is approximately 1.423ms in the CPU scenario and approximately 1.522ms in the memory scenario, lower than 2.219ms and 2.244ms for fixed-window aggregation, and also significantly lower than 8.511ms and 9.982ms for query-side LTTB. This indicates that under the same output budget, the method of this invention not only has better trend fidelity but also a lighter computational path during the query phase.

[0088] Example 3: Comparative Experiment 2: Comparison of Fixed Parameters Across Ranges During a 7-Day Full-Process Query To verify the stability of this invention when the query range is expanded, this comparative experiment continues to use the algorithm parameters in Tables 1 and 2, i.e., Tmax, Tmin, alpha, F_initial, and F_min remain unchanged, but the front-end query time range is expanded to a full 7-day query period. For external comparison methods, since the output points of query-side LTTB and fixed-window aggregation cannot adaptively change with the query range as in this invention, this comparative experiment still uses the output points or corresponding window configuration determined in Example 3 when comparing the 1-day query window, to examine the performance degradation of each method after the query range is expanded. It should be noted that the processing chain of query-side LTTB and fixed-window aggregation is not part of the implementation process of this invention; only its results are used as the object of comparative analysis.

[0089] The comparative experiment results show that when the query range is expanded from 1 day to 7 days, the present invention can still maintain low error and high trend fidelity while keeping the algorithm parameters unchanged; while the external method, because the number of output points or window configuration cannot be adapted to the query range, can only use the output budget of 1 day query in Example 3, which leads to a significant increase in error, serious imbalance in trend fidelity and increased query time.

[0090] Specifically, in the CPU scenario, the NRMSE of query-side LTTB increased from 0.5476 for 1-day queries to 0.7429 for 7-day queries, while the correlation coefficient plummeted from 0.4268 to 0.0751; the NRMSE of fixed-window aggregation increased from 0.4082 to 0.6441, while the correlation coefficient decreased from 0.6634 to 0.1476; whereas the NRMSE of the method of this invention remained at approximately 0.2865, and the correlation coefficient remained at approximately 0.6971. In the memory scenario, if the query-side LTTB and fixed-window aggregation continue to use the 1-day query parameters, their output point density decreases significantly, and the trend expression becomes noticeably coarser.

[0091] Regarding query latency, in CPU scenarios, the query-side LTTB increases to approximately 52.105ms, and fixed-window aggregation requires approximately 7.179ms, while the method of this invention remains at approximately 1.423ms. In memory scenarios, the query-side LTTB increases to approximately 56.278ms, and fixed-window aggregation requires approximately 9.516ms, while the method of this invention remains at approximately 1.522ms. This is mainly due to the fact that this invention directly reads the 8-field bucket summary results generated during write operation, and the number of output points naturally expands with the query range, without needing to re-determine the number of points or buckets during the query phase. This result demonstrates that when the query range expands, this invention does not need to reset the query-side parameters, and can still maintain low error and low query latency, exhibiting good adaptive performance.

[0092] In summary, this embodiment verifies that: 1) The method of the present invention can automatically adapt to monitoring indicators with different data formats under unified parameters; 2) Under fair budget conditions, this invention has better trend fidelity than query-side LTTB and fixed-window aggregation; 3) When the query range is expanded, the present invention can maintain stable results without resetting the query side parameters; 4) This invention has significant technical advantages in reducing query-side parameter tuning, maintaining trend fidelity, and implementing incremental processing during writes.

[0093] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof.

[0094] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A write-time incremental adaptive downsampling method for time-series data, characterized in that, include: Step 1: Obtain the raw time-series data corresponding to the CUP monitoring indicators, wherein the raw time-series data includes at least a sequence identifier, a timestamp, and a numerical value; Step 2: Locate the corresponding current open bucket based on the sequence identifier. If the current open bucket does not exist, use the current original time series data as the starting data of the new bucket, set the bucket start time, record the first value, initialize the buffer, and set the count in the bucket to 1. Step 3: If the current open bucket already exists, include the current original time series data into the buffer of the current open bucket, update the count in the bucket, sort the data in the buffer by value, extract the P5 quantile value and corresponding timestamp, the P95 quantile value and corresponding timestamp of the current bucket, and calculate the fluctuation amplitude of the current bucket, wherein the fluctuation amplitude is the difference between the P95 quantile value and the P5 quantile value. Step 4: Perform a two-dimensional bucket boundary determination on the current open bucket: if the difference between the current timestamp and the bucket start time is greater than the maximum bucket duration threshold, then the current open bucket is determined to be closed; or, if the difference between the current timestamp and the bucket start time is greater than the minimum cumulative duration threshold and the fluctuation amplitude is greater than the current fluctuation threshold, then the current open bucket is determined to be closed. Step 5: When it is determined that the current open bucket is closed, generate the bucket summary result of the closed bucket and write it into the downsampling result storage. The bucket summary result includes the bucket start time, bucket end time, first value, P5 quantile, timestamp corresponding to the P5 quantile, P95 quantile, timestamp corresponding to the P95 quantile, and last value. Step 6: Update the current fluctuation threshold according to the fluctuation range of the closed bucket using an exponentially weighted moving average method; Step 7: Use the current original time series data that triggered the closure as the starting data for the next open bucket, and return to Step 2.

2. The method according to claim 1, characterized in that, Step 4 specifically includes: Time dimension determination: when When it is determined that the currently open bucket is closed, wherein, This is the current timestamp. The start time of the bucket. The maximum bucket duration threshold; Fluctuation dimension determination: when and When it is determined that the currently open bucket is closed, wherein, The minimum cumulative duration threshold, This represents the current fluctuation range of the bucket. This represents the current fluctuation threshold. The time dimension determination and the fluctuation dimension determination are related by OR logic. If either condition is met, the current open bucket is determined to be closed.

3. The method according to claim 1, characterized in that, The bucket summary result also includes a sequence identifier, which is used to distinguish different monitoring indicator sequences. Each sequence independently maintains its corresponding open bucket status and fluctuation threshold.

4. The method according to claim 1, characterized in that, The P5 quantile is the quantile value at the 5th percentile after the data in the buffer is sorted numerically, and the P95 quantile is the quantile value at the 95th percentile after the data in the buffer is sorted numerically.

5. The method according to claim 1, characterized in that, The expression for updating the current fluctuation threshold is: in, The updated fluctuation threshold, The fluctuation threshold before the update. This represents the fluctuation range of the current closed bucket. For smoothing coefficients, ; The updated fluctuation threshold is set to be no less than the preset lower threshold. .

6. The method according to claim 1, characterized in that, The current state variables of the open bucket include: bucket start time. First value Buffer Counting inside the bucket Each item in the buffer is The binary tuple, wherein the current fluctuation threshold is assigned an initial value when the sequence first enters the processing flow. .

7. The method according to claim 1, characterized in that, Following step 6, the method further includes a query step: When a trend query request within a specified time range is received, the original time series data within the corresponding time range is not scanned. Instead, the corresponding bucket summary result is read directly from the downsampling result storage based on the sequence identifier and the bucket start time range. During front-end rendering, the first value, P5 quantile, P95 quantile, and last value in each bucket summary result are sorted according to their respective timestamps and then connected sequentially to restore the trend curve.

Images