Time sequence data adaptive processing method and system, electronic equipment and medium
By employing a data processing mechanism that separates static and dynamic data in high-frequency pulse scenarios, data is written to a memory buffer in real time and processed hierarchically, thus solving the problems of database latency and storage costs and ensuring the integrity and stability of key features.
Patent Information
- Application Number
- CN202610205932.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-15
AI Technical Summary
In high-frequency pulse scenarios, existing time-series databases experience a surge in latency during high-throughput writes due to memory table flushing or file merging operations, threatening the stability of real-time control. At the same time, massive data storage is costly and lossy compression can easily erase key features.
By responding to the experimental trigger signal, the system enters discharge mode to pause persistent storage writing. Real-time data is directly written to the memory buffer and then processed in a hierarchical manner in defragmentation mode. Transient and steady-state intervals are identified, and lossless or lossy compression strategies are used to process the data.
It achieves time decoupling between ultra-fast writing and data processing, eliminates latency peaks, reduces storage costs, retains key features, and meets microsecond-level protection requirements.
Smart Images
Figure CN122045199A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial control systems and database storage technology, and in particular to a time-series data adaptive processing method, system, electronic device, and medium. Background Technology
[0002] In large scientific facilities such as nuclear fusion-assisted heating and particle accelerators, the system operation exhibits typical pulse or cannon characteristics. That is, the system discharges high power in an extremely short period of time, generating massive amounts of high-frequency sampling data, and then enters a relatively long interval for cooling and recharging.
[0003] Currently, mainstream time-series databases are primarily designed for continuous and stable internet traffic, and their underlying structure often employs a log-merged tree structure. During high-throughput writes during discharge periods, the database backend may uncontrollably trigger disk flushing or file merging operations. These operations preempt computing and disk I / O resources, causing a sudden spike in write latency, severely threatening the stability of the real-time closed-loop control of the power protection system. Furthermore, to capture fault characteristics such as micro-arcing or overvoltage, the system must employ ultra-high sampling rates. Using lossless compression would require enormous storage space and be prohibitively expensive. Using general-purpose lossy compression, because the algorithm treats all data equally, it easily erases minute pre-fault characteristics during compression, rendering physical experimental data unusable for analysis.
[0004] Therefore, how to eliminate write jitter in high-frequency pulse scenarios while resolving the contradiction between the cost of massive data storage and the preservation of key features has become an urgent technical challenge. Summary of the Invention
[0005] The main objective of this invention is to provide a time-series data adaptive processing method, system, electronic device, and medium, which aims to eliminate write jitter in high-frequency pulse scenarios while resolving the contradiction between the cost of massive data storage and the preservation of key features.
[0006] To achieve the above objectives, this invention proposes an adaptive processing method for time-series data, comprising the following steps: In response to receiving the start trigger signal of this experiment, it enters the discharge mode; in the discharge mode, writing operations to the persistent storage device are suspended, and the real-time collected time-series data is directly written to the pre-allocated memory buffer in binary stream form. In response to receiving the end trigger signal of this experiment, the system enters the sorting mode; in the sorting mode, the timing data in the memory buffer is read and hierarchical processing is performed. The hierarchical processing includes: calculating the rate of change index of the time series data; comparing the rate of change index with a judgment threshold to identify the transient and steady-state intervals in the time series data; The data in the transient range is processed using lossless compression or original precision storage, while the data in the steady-state range is processed using lossy compression, and the processed data is written to the persistent storage device.
[0007] Preferably, a dynamic calibration step is included before the start trigger signal is received: Acquire no-load reference data collected by the sensor during the preparation period; Calculate the noise statistical characteristics of the unloaded reference data; The determination threshold for this experiment is generated based on the aforementioned noise statistical characteristics.
[0008] Preferably, calculating the noise statistical characteristics of the unloaded reference data includes calculating the baseline mean of the unloaded reference data. and noise standard deviation ; The step of generating the judgment threshold for this experiment based on the noise statistical characteristics includes calculating the judgment threshold according to the following formula. :
[0009] in, The threshold is a preset sensitivity coefficient; the judgment threshold is used to measure the allowable range of signal fluctuations or the range of change rate.
[0010] Preferably, the step of directly writing the real-time acquired time-series data into a pre-allocated memory buffer in binary stream form includes: Bypassing the file system page cache, the timing data is written to a circular buffer via direct memory access or memory copy. The time-series data is not serialized or indexed when written to the memory buffer.
[0011] Preferably, calculating the rate of change index of the time series data includes calculating the first derivative of the time series data within the time window. ; The process of identifying transient and steady-state intervals in the time-series data includes: If the first derivative If the absolute value is greater than the determination threshold, then the corresponding time window is marked as the transient interval; If the first derivative If the absolute value is less than or equal to the determination threshold, then the corresponding time window is marked as the steady-state interval.
[0012] Preferably, the processing of the data in the transient interval by lossless compression or original precision storage includes compressing the data in the transient interval using differential coding combined with the Zstandard algorithm; The lossy compression method used to process the data in the steady-state range includes using the Chimp algorithm or dead-zone compression algorithm to compress the data in the steady-state range.
[0013] This application also discloses a time-series data adaptive processing system, including: The state awareness and scheduling module is used to monitor the trigger signal of the experiment, and switch to the discharge mode when it receives the start trigger signal, and switch to the tidying mode when it receives the end trigger signal. A dual-mode write engine is used to pause write operations to persistent storage devices in the discharge mode and directly write real-time acquired time-series data into a pre-allocated memory buffer in binary stream form. The feature classification processing module is used to read time-series data in the memory buffer in the sorting mode, calculate the rate of change index of the time-series data, and compare the rate of change index with a judgment threshold to identify the transient and steady-state intervals in the time-series data. The storage management module is used to process the data in the transient range using lossless compression or original precision storage, process the data in the steady-state range using lossy compression, and write the processed data to the persistent storage device.
[0014] Preferably, the system further includes a dynamic calibration module, which is used to acquire the idle reference data collected by the sensor during the preparation period before receiving the start trigger signal, calculate the noise statistical characteristics of the idle reference data, and generate the judgment threshold for this experiment based on the noise statistical characteristics.
[0015] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method described in any of the preceding claims.
[0016] This application also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the method described in any of the preceding claims.
[0017] The above technical solution has the following advantages: This invention, in response to an experimental trigger signal, pauses persistent storage writing in discharge mode and writes directly to memory in binary stream form. In compaction mode, it performs tiered data processing and disk write, achieving decoupling of ultra-fast writing and data processing on the time axis. This dynamic-static separation mechanism completely eliminates latency peaks caused by internal database operations during discharge, meeting microsecond-level protection requirements. Simultaneously, by calculating the rate of change index to identify transient and steady-state intervals, it uses lossless processing to preserve physical details for critical transient fault characteristics and lossy compression to significantly reduce data volume for stable steady-state data. This significantly reduces hardware storage costs while ensuring the integrity of high-value data. Attached Figure Description
[0018] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein: Figure 1 A flowchart of a time-series data adaptive processing method provided in an embodiment of the present invention.
[0019] Figure 2 This is a schematic diagram of the structure of the time-series data adaptive processing system provided in an embodiment of the present invention.
[0020] Figure 3 This is a schematic diagram illustrating the hierarchical compression principle based on waveform features, provided in an embodiment of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0022] Example 1 This embodiment provides an adaptive processing method for time-series data in pulsed experimental environments. This method is primarily applied to the control and monitoring systems of large scientific facilities such as nuclear fusion auxiliary heating devices and particle accelerators. These systems exhibit typical pulsed characteristics, namely, high-power discharges occurring within an extremely short timeframe, specifically 10 to 100 seconds, generating massive amounts of high-frequency sampling data at MHz levels, followed by a relatively long interval for cooling and recharging, typically 10 to 20 minutes. Addressing the write jitter issues caused by background operations during pulsed discharges in existing time-series databases, and the contradiction between storage costs and data value preservation at high sampling rates, this embodiment employs a dynamic-static separation data processing mechanism. Utilizing the intermittent characteristics of pulsed experiments, it decouples rapid writing and data processing along the time axis, and combines physical waveform characteristics to achieve lossless storage of critical data and high-pressure storage of stable data in a hierarchical manner.
[0023] like Figure 1 As shown, the method includes the following specific steps: The first stage is the preparatory phase, which involves noise fingerprint self-learning and dynamic threshold anchoring. Before the system receives the start trigger signal 201 for this experiment, there is typically a device charging or preparatory period. This embodiment utilizes this window period to perform dynamic parameter calibration to eliminate the impact of sensor 200 baseline drift on feature extraction. Specifically, the system acquires idle reference data collected by sensor 200 during the preparatory phase. For example, the system automatically extracts the idle signal from the last segment of the preparatory phase as a reference sample; this time length is... For example, it can be set to 500ms. Next, the noise statistics of the unloaded reference data are calculated. This process includes calculating the baseline mean of the reference sample in real time. and noise standard deviation This establishes the current environmental noise fingerprint. Subsequently, a judgment threshold for this experiment is generated based on the noise statistical characteristics. The system then applies a preset sensitivity coefficient. Dynamically calculate the threshold for determining the current discharge cycle The calculation formula is:
[0024] Among them, sensitivity coefficient This is used to adjust the system's sensitivity to signal fluctuations and can be set to a value between 3 and 6. The calculated decision threshold... It is updated in real time to the system's configuration register to measure the range of signal fluctuations or rate of change that are allowed afterward. This replaces the traditional fixed threshold and enables it to automatically adapt to the differences in the electromagnetic environment of different experimental devices, thus solving the problem of sensor zero-point drift caused by equipment aging or temperature changes.
[0025] The second stage is the adaptive writing step of time-series data in discharge mode. Upon receiving the start trigger signal 201 for this experiment, such as the SHOT_START signal, the system immediately enters discharge mode. In discharge mode, the system performs strict resource scheduling, suspends write operations to persistent storage device 107, locks the disk write thread, and binds the CPU affinity to the acquisition core to ensure maximum acquisition performance. During this period, the system directly writes the real-time acquired time-series data into a pre-allocated memory buffer 106 in binary stream form. To ensure nanosecond-level write determinism, this step employs kernel bypass technology. Specifically, a large-capacity circular memory buffer is pre-allocated at startup, and memory preheating is performed to avoid runtime page faults. Subsequently, bypassing the file system page cache, the time-series data is written to this memory buffer via direct memory access or memory copy. The memory buffer can be a circular buffer, a double-buffered queue, or a multi-level buffered linked list. When the time-series data is written to the memory buffer 106, it does not undergo serialization, index building, or any compression processing, but flows directly as a binary stream of double-precision floating-point numbers or single-precision floating-point numbers. This design postpones input / output operations and computational loads to the intermittent period, completely eliminating latency peaks caused by internal database operations during discharge, thus meeting microsecond-level protection requirements.
[0026] The third stage involves feature extraction and hierarchical processing in the sorting mode. Upon receiving the end-of-experiment trigger signal 201, such as the SHOT_END signal, the system enters sorting mode. In sorting mode, the system wakes up the background processing thread, reads the time-series data from memory buffer 106, and performs hierarchical processing. The core of hierarchical processing lies in identifying the physical semantics of the data. The system first calculates the rate of change index of the time-series data, specifically including the time window... Calculate the first derivative from time series data. And the second derivative. Then, combined with... Figure 3 The waveform feature grading principle shown compares the rate of change index with the judgment threshold generated in the first stage to identify transient and steady-state intervals in the time series data. The specific judgment logic is as follows: if the first derivative... The absolute value is greater than the judgment threshold. Then, the corresponding time window is marked as the transient interval, i.e., the region of interest. This type of region typically corresponds to critical physical processes such as the rising edge, falling edge, or fault spikes of a signal. If the first derivative... The absolute value is less than or equal to the judgment threshold. If the time window is defined as such, then it is marked as the steady-state interval. This type of region typically corresponds to the flat-top segment or zero-level segment of the signal.
[0027] The fourth stage involves adaptive hierarchical compression and storage. Based on the labeling results from the third stage, the system employs a heterogeneous compression strategy for data in different intervals. Data in transient intervals is processed using lossless compression or storage with original precision. For example, the system may compress transient data or directly store raw floating-point numbers, ensuring 100% restoration of physical details and guaranteeing no loss of data precision during faults. Data in steady-state intervals is processed using lossy compression. For example, the system may use lossy compression algorithms (such as Chimp, dead-zone compression, rotating door compression, or Sprintz algorithms) to compress steady-state data, retaining only the average value or trend line. This approach significantly reduces data volume while preserving basic trend information, achieving intelligent space optimization. Finally, the processed data is written to persistent storage device 107, such as a solid-state drive array or object storage system. Before writing, the system can also rearrange out-of-order data based on hardware timestamps, generating a sorted string table file for disk storage.
[0028] Example 2 This embodiment provides a time-series data adaptive processing system 100 for pulse experimental environments. The system is built based on the method described in Embodiment 1 and can be deployed in edge computing nodes or high-performance servers to solve the data writing bottleneck and storage efficiency problems in high-frequency pulse scenarios.
[0029] like Figure 2 As shown, the system mainly includes a state awareness and scheduling module 101, a dual-mode writing engine 102, a feature hierarchical processing module 103, a storage management module 104, and a dynamic calibration module 105.
[0030] The state awareness and scheduling module 101, acting as the system's control center, monitors the experimental trigger signals 201 and switches the system's operating mode based on the signal type. Specifically, this module is connected in real-time to the hardware synchronization trigger circuit of the central control system. When it receives a start trigger signal 201, such as the SHOT_START signal, the state awareness and scheduling module 101 broadcasts a discharge mode command to the system and forcibly suspends all non-real-time tasks, locking computing resources to the data acquisition process. When it receives an end trigger signal 201, such as the SHOT_END signal, the module broadcasts a cleanup mode command to the system, releases the resource lock, and triggers background processing threads.
[0031] The dual-mode write engine 102 is the system's data receiving front-end, designed with two independent logical channels: a fast channel and a processing channel. In discharge mode, the dual-mode write engine 102 enables the fast channel. This channel is configured to suspend write operations to the persistent storage device 107 and directly write real-time acquired time-series data into a pre-allocated memory buffer 106 as a binary stream. To achieve extremely low latency, the fast channel employs kernel bypass technology, pre-allocating a large-capacity memory ring buffer and performing memory warm-up at startup to avoid runtime page faults. Subsequently, bypassing the operating system's file system page cache, data is directly injected into the memory ring buffer using direct memory access technology or efficient memory copy instructions. At this time, the time-series data does not undergo any serialization, indexing, or compression encoding when written to the memory buffer 106, ensuring nanosecond-level write response speeds.
[0032] The feature grading processing module 103 is used to perform semantic analysis on the data backed up in memory in a cleanup mode. This module reads time-series data from the memory buffer 106 and performs grading processing. Specific operations include calculating the rate of change index of the time-series data, i.e., the time window... Calculate the first derivative from the data within. Subsequently, the module compares the calculated rate of change index with the judgment threshold provided by the dynamic calibration module 105 point by point or window by window to identify transient and steady-state intervals in the time series data. If the absolute value of the rate of change index exceeds the judgment threshold, the time window is marked as a transient interval; otherwise, it is marked as a steady-state interval.
[0033] The storage management module 104 is responsible for the final compression and disk write of the data. Based on the interval markings output by the feature classification processing module 103, this module calls a heterogeneous compression algorithm library. For data marked as transient intervals, the storage management module 104 processes it using lossless compression or original precision storage methods, such as differential coding combined with the Zstandard algorithm, to ensure the integrity of fault characteristics. For data marked as steady-state intervals, this module processes it using lossy compression methods, such as the Gorilla algorithm, Chimp algorithm, or dead-zone compression algorithm, retaining only the statistical characteristics or trend lines of the data. The processed data is encapsulated in a sorted string table format and finally written to the persistent storage device 107, such as a solid-state drive array.
[0034] The dynamic calibration module 105 is used to perform environmental noise self-learning before receiving the start trigger signal 201. This module acquires the no-load reference data collected by the sensor 200 during the preparation period and calculates its noise statistics, including the baseline mean. and noise standard deviation Based on these statistical characteristics, the dynamic calibration module 105 uses the formula
[0035] Generate the judgment threshold for this experiment. Among them, This is a preset sensitivity coefficient. The generated decision threshold is then injected into the feature classification processing module 103 for subsequent interval determination.
[0036] Example 3 This embodiment provides an electronic device, which may be a high-performance industrial computer, a server, or an embedded edge controller. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor.
[0037] The memory is used to store instructions and data, and may include high-speed random access memory or non-volatile memory, such as at least one disk storage device, flash memory device, or other solid-state storage device. In the pulse experiment scenario, a contiguous physical memory region of sufficient capacity is reserved in the memory to construct the circular buffer in the aforementioned embodiments and to support memory warm-up to prevent page faults, thereby supporting high-volume data caching during discharge.
[0038] The processor can be a general-purpose processor, including a central processing unit (CPU) and a network processor; it can also be a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor is connected to the memory to execute computer programs stored in the memory.
[0039] When the processor executes the computer program, it implements the adaptive processing method for timing data in a pulse experimental environment as described in Embodiment 1. Specifically, it includes: in response to receiving the start trigger signal 201 of the experiment, entering a discharge mode, pausing write operations to the persistent storage device 107, and directly writing the real-time acquired timing data into the pre-allocated memory buffer 106 in binary stream form; in response to receiving the end trigger signal 201 of the experiment, entering a sorting mode, reading the timing data in the memory buffer 106, and performing hierarchical processing; calculating the rate of change index of the timing data and comparing it with a judgment threshold to identify transient and steady-state intervals; performing lossless compression or original precision storage on the transient interval data, and lossy compression on the steady-state interval data; finally, writing the processed data into the persistent storage device 107. In addition, the processor also performs a dynamic calibration step, generating a judgment threshold based on the noise statistical characteristics of the idle reference data.
[0040] Example 4 This embodiment provides a computer-readable storage medium, which may be a read-only memory, random access memory, optical disc read-only memory, magnetic tape, floppy disk, or optical data storage device, etc. A computer program is stored on it, which, when executed by a processor, implements the method described in Embodiment 1.
[0041] The computer program contains instruction modules for performing the following operations: the first module is used to listen to the trigger signal 201 and manage system mode switching; the second module is used to perform kernel bypass writing in discharge mode, storing data into the ring buffer at high speed; the third module is used to analyze the data change rate and divide the data interval based on the dynamic threshold in the defragmentation mode; the fourth module is used to select a lossless or lossy compression algorithm according to the interval type and complete the data write to disk; the fifth module is used to analyze background noise and dynamically calculate the judgment threshold during the experimental preparation period.
[0042] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the statement "comprising a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0043] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. An adaptive processing method for time-series data, characterized in that, Includes the following steps: In response to receiving the start trigger signal of this experiment, it enters the discharge mode; in the discharge mode, writing operations to the persistent storage device are suspended, and the real-time collected time-series data is directly written to the pre-allocated memory buffer in binary stream form. Upon receiving the end-of-experiment trigger signal, it enters the sorting mode; In the sorting mode, the timing data in the memory buffer is read and hierarchical processing is performed; The hierarchical processing includes: calculating the rate of change index of the time series data; comparing the rate of change index with a judgment threshold to identify the transient and steady-state intervals in the time series data; The data in the transient range is processed using lossless compression or original precision storage, while the data in the steady-state range is processed using lossy compression, and the processed data is written to the persistent storage device.
2. The method according to claim 1, characterized in that, Before receiving the start trigger signal, a dynamic calibration step is also included: Acquire no-load reference data collected by the sensor during the preparation period; Calculate the noise statistical characteristics of the unloaded reference data; The determination threshold for this experiment is generated based on the aforementioned noise statistical characteristics.
3. The method according to claim 2, characterized in that, The calculation of the noise statistical characteristics of the unloaded reference data includes calculating the baseline mean of the unloaded reference data. and noise standard deviation ; The step of generating the judgment threshold for this experiment based on the noise statistical characteristics includes calculating the judgment threshold according to the following formula. : in, The threshold is a preset sensitivity coefficient; the judgment threshold is used to measure the allowable range of signal fluctuations or the range of change rate.
4. The method according to claim 1, characterized in that, The step of directly writing the real-time acquired time-series data into a pre-allocated memory buffer in binary stream format includes: Bypassing the file system page cache, the timing data is written to a circular buffer via direct memory access or memory copy. The time-series data is not serialized or indexed when written to the memory buffer.
5. The method according to claim 1, characterized in that, The calculation of the rate of change index of the time series data includes calculating the first derivative of the time series data within the time window. ; The process of identifying transient and steady-state intervals in the time-series data includes: If the first derivative If the absolute value is greater than the determination threshold, then the corresponding time window is marked as the transient interval; If the first derivative If the absolute value is less than or equal to the determination threshold, then the corresponding time window is marked as the steady-state interval.
6. The method according to claim 1, characterized in that, The process of processing the data in the transient interval using lossless compression or original precision storage includes compressing the data in the transient interval using differential coding combined with the Zstandard algorithm; The lossy compression method used to process the data in the steady-state range includes using the Chimp algorithm or dead-zone compression algorithm to compress the data in the steady-state range.
7. A time-series data adaptive processing system, characterized in that, include: The state awareness and scheduling module is used to monitor the trigger signal of the experiment, and switch to the discharge mode when it receives the start trigger signal, and switch to the tidying mode when it receives the end trigger signal. A dual-mode write engine is used to pause write operations to persistent storage devices in the discharge mode and directly write real-time acquired time-series data into a pre-allocated memory buffer in binary stream form. The feature classification processing module is used to read time-series data in the memory buffer in the sorting mode, calculate the rate of change index of the time-series data, and compare the rate of change index with a judgment threshold to identify the transient and steady-state intervals in the time-series data. The storage management module is used to process the data in the transient range using lossless compression or original precision storage, process the data in the steady-state range using lossy compression, and write the processed data to the persistent storage device.
8. The system according to claim 7, characterized in that, The system also includes a dynamic calibration module, which is used to acquire the idle reference data collected by the sensor during the preparation period before receiving the start trigger signal, calculate the noise statistical characteristics of the idle reference data, and generate the judgment threshold for this experiment based on the noise statistical characteristics.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.