An Embedded Time-Series Data Storage Method Based on Multi-Level Storage and Multi-Level DMA Acceleration

Abstract

An embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration is disclosed. This method includes: writing the acquired raw time-series data points into a circular buffer of tightly coupled memory for temporary storage; the main direct memory access controller (MDMA) performing zero-copy data transfer of the time-series data in the circular buffer; compressing the transferred time-series data in on-chip high-speed static random access memory (SRAM) using hardware acceleration instructions to obtain compressed data blocks and constructing metadata corresponding to the compressed data blocks; constructing and updating a three-level hybrid index based on the compressed data blocks and their corresponding metadata; and storing the compressed data blocks, metadata, and the three-level hybrid index into flash memory via a standard MDMA controller to complete the time-series data storage. This method solves the problems of poor real-time performance, wasted computing resources, low retrieval efficiency, and rapid wear of storage media in low-resource embedded devices for massive time-series data storage.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and specifically to an embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration. Background Technology

[0002] With the rapid development of applications such as the Internet of Things (IoT), industrial automation, and smart devices, edge storage and processing of time-series data are becoming increasingly important in resource-constrained embedded devices. However, embedded devices such as microcontrollers are typically limited by extremely small memory capacity and limited processor computing power, causing traditional time-series database solutions based on distributed or conventional storage architectures to expose the following significant technical shortcomings when directly migrated and deployed: 1. Low utilization of heterogeneous storage resources and severe media wear: Traditional time-series databases fail to fully integrate the multi-level storage characteristics of the microcontroller's underlying layer, resulting in frequent inefficient data interaction between memory and external Flash. This not only has a severe write amplification effect but also accelerates the physical lifespan decay of the Flash media.

[0003] Second, general compression algorithms have high overhead and lack random access capabilities: Existing time-series data compression schemes often do not fully utilize the hardware computing instructions of embedded processors for acceleration, resulting in a prominent CPU computing power bottleneck when writing at high frequency; moreover, traditional streaming compression and decompression are mostly full operations, which cannot perform precise random access when facing specific time window queries, resulting in extremely low decompression efficiency.

[0004] III. Index Bloating and Slow Query Performance Due to Massive Historical Data: With limited memory, existing index structures struggle to simultaneously handle high-speed writing of real-time data and multi-dimensional tag retrieval of massive historical data. As data volume accumulates, the memory index expands rapidly, and the lack of efficient mechanisms for separating hot and cold data and merging incremental indexes makes it difficult to meet the millisecond-level real-time requirements of multi-dimensional combined queries in industrial settings.

[0005] IV. Data Transfer and Statistical Computation Consume Extremely High CPU Resources: In high-concurrency sensor data access scenarios, traditional solutions rely heavily on the direct involvement of the CPU for data transfer and pre-aggregation calculations. This blocking processing flow consumes a significant amount of CPU clock cycles, preventing the system from handling other real-time control tasks and severely limiting overall concurrent processing capabilities.

[0006] The invention patent with publication number CN110737723A proposes a method for storing and retrieving time-series data for embedded devices. This method employs a hybrid index structure, storing the index of recent data in volatile memory to ensure efficient writing and real-time querying, while storing the index of historical data in non-volatile memory to save memory space. Furthermore, it uses a difference-based compression algorithm to compress the time-series data, reducing storage space usage. This method solves the problem that a single index structure cannot simultaneously handle massive historical data storage and efficient querying in memory-constrained environments, achieving the effects of saving memory usage and improving query efficiency. However, its drawback lies in the fact that its data compression, index updates, and data transfer between memory and storage still heavily rely on CPU serial processing, failing to introduce a DMA hardware acceleration mechanism to free up CPU computing power. Summary of the Invention

[0007] In order to overcome the shortcomings of the existing technology, the present invention aims to provide an embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration, which solves the problems of poor real-time performance, waste of computing resources, low retrieval efficiency and rapid wear of storage media in low-resource embedded devices for massive time-series data storage.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration is applied to an ARM architecture including tightly coupled memory, on-chip high-speed static random access memory, flash memory, main direct memory access controller, standard direct memory access controller, and microprocessor, comprising the following steps: Step 1: Write the collected raw time series data points into a tightly coupled memory circular buffer to temporarily store the time series data; Step 2: The main direct memory access controller performs zero-copy data transfer of the timing data in the circular buffer and generates a hardware interrupt notification; Step 3: The microprocessor core responds to the hardware interrupt notification, uses hardware acceleration instructions in the on-chip high-speed static random access memory to compress the transferred timing data, obtains compressed data blocks, and constructs metadata corresponding to the compressed data blocks. Step 4: Based on the compressed data blocks and their corresponding metadata, complete the construction and update of the three-level hybrid index, and store the compressed data blocks, metadata, and three-level hybrid index in flash memory through the standard direct memory access controller to complete the persistent storage of time-series data; the three-level hybrid index includes the L0 level memory index located in the on-chip high-speed static random access memory, the L1 level incremental index located in flash memory, and the L2 level basic index.

[0009] Furthermore, in step 1, the circular buffer is managed using head pointers and tail pointers, and the memory address and length of the circular buffer both follow the fixed byte alignment constraints of the processor bus architecture.

[0010] Furthermore, in step 2, the primary direct memory access controller's zero-copy data transfer of timing data within the circular buffer specifically includes: When the amount of time-series data in the circular buffer reaches the preset block size, the microprocessor configures the transmission parameters of the main direct memory access controller and controls the main direct memory access controller to initiate a data transmission to achieve zero-copy data transfer of time-series data. The transmission parameters of the main direct memory access controller include the source address and the destination address. The source address is set to the current head pointer position of the circular buffer, and the destination address is set to the data temporary storage area of ​​the on-chip high-speed static random access memory.

[0011] Furthermore, in step 3, the transferred timing data is compressed using hardware acceleration instructions in the on-chip high-speed static random access memory to obtain compressed data blocks, specifically including: Incremental-to-increment dual-track differential coding is used to compress the timestamp sequence of the transported time-series data, and XOR operation is used to compress the measurement value sequence of the transported time-series data. During the compression process, the hardware instruction set in the ARM architecture is used to accelerate variable-length coding operations, and checkpoints are inserted into the time-series data points at fixed intervals. After compression, the corresponding compressed data block is obtained.

[0012] Furthermore, the hardware instruction set in the ARM architecture includes CLZ leading zero counting instructions and bitwise operation instructions.

[0013] Furthermore, in step 3, constructing the metadata corresponding to the compressed data block specifically includes: The Kahan summation algorithm is used to accumulate the additivity index within the compressed data block, and the Welford online algorithm is used to perform a single pass through the time series data within the compressed data block to calculate metadata containing statistical values; the statistical values ​​include mean, variance, maximum value, minimum value, sum, and standard deviation.

[0014] Furthermore, in step 4, the construction and updating of the three-level hybrid index based on the compressed data blocks and their corresponding metadata specifically includes: Key features of the compressed data block are extracted, including start timestamp, end timestamp, and sequence identifier; these key features are then converted into bit-mapping relationships using a mapping function. Then it is stored in the on-chip high-speed static random access memory to form an L0 level memory index; the L0 level memory index adopts a coarse-grained bitmap structure, in which each bit corresponds to a preset time span. By assigning the corresponding bit position that matches the target compressed data block to a value of 1, the status label of the target compressed data block is realized. When the size of the time-series data covered by the L0-level index reaches the preset memory buffer threshold of the on-chip high-speed static random access memory, the system triggers an index flush operation: the metadata of all compressed data blocks within the time period is serialized to form an L1-level incremental index data packet; the L1-level incremental index data packet is persistently stored in the incremental index partition in the flash memory using an append write method to form an L1-level incremental index; at the same time as the append write, a lightweight mount table is maintained to record the logical order of each L1-level incremental index block in the flash memory; The system background process monitors the number of L1-level incremental indexes in flash memory in real time. When the number of L1-level incremental indexes reaches a preset merging threshold, it triggers multi-way merge sort to sort the timestamps of the L1-level incremental indexes. Multiple L1-level incremental indexes are read into memory, reordered and deduplicated according to their timestamps, and incrementally merged with the existing L2-level base indexes in memory to form an updated L2-level base index. After merging, the node information of the L2-level base indexes is updated, and the merged L1-level incremental index area is marked as erasable, thus realizing the construction and updating of a three-level hybrid index structure.

[0015] Furthermore, in step 4, storing the compressed data blocks, metadata, and three-level hybrid index into flash memory via a standard direct memory access controller specifically includes: Based on the logical address of the flash memory encapsulated in the compressed data block, the data is recorded in the L1 level incremental index of the flash memory in an append-only manner; The compressed data blocks and their corresponding metadata are moved to the flash memory buffer via a standard direct memory access controller, and the flash memory buffer is aligned with the page size of the flash memory. Flash memory initiates write operations to compressed data blocks, metadata, and a three-level hybrid index to complete the persistent storage of time-series data.

[0016] Furthermore, after completing the persistent storage of time-series data, the process also includes merging and optimizing the space storage of the L1-level incremental index and L2-level basic index in flash memory, specifically including: As time-series data is continuously written, the L1-level incremental index will gradually grow. When the size of the L1-level incremental index reaches a preset threshold or the system is in an idle period, a low-priority background task is triggered. This low-priority background task reads the current L2-level basic index and all L1-level incremental indexes from the flash memory into the on-chip high-speed static random access memory (SRAM). In the SRAM, it performs a multi-way merge sort on the L2-level basic indexes and then uses a basic plus incremental merge algorithm to merge the sorted L2-level basic indexes and all L1-level incremental indexes into a unified index structure. During the merging process, the timestamp of the time-series data is used as the primary index key, and the sequence ID of the time-series data is used as the secondary index key. The sorted L2-level basic indexes and all L1-level incremental indexes are then compared and overlapped. The merged unified index structure is written sequentially to a new location in the flash memory, the system's root pointer is updated to point to the unified index structure, and the merged L2-level basic indexes and all L1-level incremental indexes are reclaimed and cleaned up to optimize flash memory storage space.

[0017] Furthermore, after completing the persistent storage of time-series data, if it is necessary to retrieve and query time-series data, the following operations should be performed: When a query request is received, the query engine parses the time range of the query; Coarse-grained filtering is performed using the L0 level memory index resident in the on-chip high-speed static random access memory to identify candidate data blocks that match the query time range; When the query request is an aggregation query request, candidate data blocks are read, and the statistical values ​​in the metadata corresponding to the candidate data blocks are merged and calculated to obtain a global aggregation result corresponding to the query time range; the overall aggregation result is encapsulated according to a preset return format and returned directly to complete the response to the aggregation query request; When the query request is a detailed query request for raw time series data points, the L1-level incremental index and L2-level basic index of the flash memory are used to locate the candidate data block containing the target raw time series data point. The candidate data block containing the target raw time series data point is read, and the query engine locates the upstream checkpoint closest to the query request timestamp based on the timestamp of the query request. Based on the nearest upstream checkpoint, a data coverage window corresponding to the query time is defined, and the time series data within the data coverage window is sequentially incrementally decompressed. The raw time series data points that meet the query time conditions are parsed and extracted from the decompressed time series data, and then packaged and returned according to the preset return format to complete the detailed query of the raw data points.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention solves the problems of poor time determinism, data jitter, and packet loss that easily occur when writing time-series data points transmitted by high-frequency sensors into ordinary memory under high load operation of embedded systems by writing the collected raw time-series data points into a tightly coupled memory ring buffer. It achieves a stable and extremely low access latency effect and has the advantages of high real-time data acquisition and strong concurrent processing capability.

[0019] 2. This invention performs zero-copy data transfer of time-series data in the circular buffer through the main direct memory access controller, which solves the problem of high CPU utilization caused by frequent processor participation in data movement in traditional storage solutions. It achieves a deep asynchronous decoupling effect between data transmission and processor calculation, and has the advantages of low overall system power consumption and sufficient available processor resources.

[0020] 3. This invention solves the problems of low compression efficiency and high risk of memory overflow in embedded devices by using hardware acceleration instructions in the on-chip high-speed static random access memory to compress the transferred time-series data and construct metadata. It achieves extremely high data throughput and data compression ratio, and has the advantages of significantly saving storage space and reducing I / O pressure.

[0021] 4. This invention solves the conflict between "small memory" and "large data volume index" in embedded systems by constructing a three-level hybrid index architecture that includes L0-level memory index, L1-level incremental index and L2-level basic index, and achieves multi-level filtering and hierarchical retrieval effects. In addition, the L0-level coarse-grained bitmap index located in the on-chip high-speed static random access memory realizes millisecond-level query "pruning", which greatly improves the query response speed and has the advantages of low retrieval latency and small memory usage.

[0022] 5. This invention solves the problems of flash memory write amplification and shortened lifespan caused by frequent modifications to the global index by persistently storing L1-level incremental indexes and L2-level basic indexes in flash memory and using a merging algorithm of basic and incremental. By converting scattered writes into sequential append writes and periodically performing multi-way merge sorting, a smooth write load distribution is obtained, which has the advantages of significantly extending the lifespan of flash memory hardware and improving the long-term reliability of the system.

[0023] In summary, this invention addresses the technical challenges faced by low-resource embedded devices when processing massive amounts of time-series data, including memory constraints, high processor load, and severe flash write amplification. It proposes a hardware-software co-operational storage and indexing optimization scheme: high-speed, lossless data access is achieved through the combination of tightly coupled memory and direct memory access; efficient compression is performed in on-chip high-speed static random access memory using processor hardware acceleration instructions, reducing storage space usage from the source; a three-level hybrid index architecture and a basic plus incremental merging algorithm significantly extend the lifespan of the flash memory while ensuring retrieval efficiency; and through deep hardware coupling and logical hierarchy optimization across the entire chain, this invention achieves industrial-grade high-performance time-series database functionality with extremely low hardware overhead. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the physical layout of the multi-level storage architecture of the present invention.

[0025] Figure 2 This is a schematic diagram of the system initialization process of the present invention.

[0026] Figure 3 This is a flowchart of the embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to the present invention.

[0027] Figure 4 This is a schematic diagram of the execution flow of the three-level storage of time-series data in this invention.

[0028] Figure 5 This is a schematic diagram of the time-series data compression process of the present invention.

[0029] Figure 6 This is a schematic diagram of the execution flow of time-series data query in this invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments: See Figure 1This invention adopts a top-down four-layer architecture, including a user interface layer, a core function layer, a backend maintenance layer, and a hardware support layer. The user interface layer provides unified interfaces for system initialization, data writing, data querying, and data management. The core function layer is the logical core of the time-series data storage in this invention, including time-series data storage, compression, and querying, and is responsible for data persistence, compression encoding, and query execution. The backend maintenance layer is responsible for completing background tasks such as index segment merging, garbage collection (GC), and wear leveling scheduling without interfering with foreground requests. The hardware support layer encapsulates tightly coupled memory (TCM), on-chip high-speed static random access memory (SRAM), flash memory, multi-level DMA controllers (including main direct memory access controller (MDMA) and standard direct memory access controller (DMA), and microcontrollers (e.g., STM32H7 series MCUs), providing a unified hardware abstraction for upper-layer modules. During system operation, the data flow ultimately terminates and executes on the multi-level storage media and DMA channels of the hardware support layer, forming an end-to-end hardware and software collaborative processing link.

[0031] like Figure 2 As shown, the system initialization process is mainly divided into two stages: core logic configuration and hardware transmission link establishment. After the system starts, it first enters the parameter verification stage to verify the system configuration and storage boundaries. Then, it synchronously executes the initialization of the tightly coupled memory ring buffer and the initialization of the heat manager, and builds the data access link and access heat statistics table in the on-chip high-speed static random access memory. Next, it completes the write-ahead log initialization and executes the write-ahead log replay recovery logic as needed to ensure power-off consistency. Finally, after completing the index management initialization to establish the three-level index logic framework, the system performs hardware initialization on the main direct memory access controller and the standard direct memory access controller, thereby configuring the full-link hardware accelerated transmission channel. After all hardware channels and logical handles are ready, the system officially enters the time-series data acquisition and storage ready state.

[0032] An embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration is applied to an ARM architecture including tightly coupled memory, on-chip high-speed static random access memory, flash memory, main direct memory access controller, standard direct memory access controller, and microprocessor. The tightly coupled memory's circular buffer provides nanosecond-level data temporary storage, achieving zero-wait writes. The on-chip high-speed static random access memory serves as a computation acceleration layer, temporarily storing data and caching indexes. Flash memory acts as a persistence layer, storing data and write-ahead logs. The writing of raw time-series data points follows a layered flow strategy, sequentially passing through tightly coupled memory and on-chip high-speed static random access memory. The data is then transferred to flash memory, forming a three-tier storage architecture. Combined with batch write and alignment mechanisms, the write amplification effect of flash memory is significantly reduced. The main direct memory access controller is used to perform zero-copy data transfer between tightly coupled memory and on-chip high-speed static random access memory without processor intervention, and manages multi-channel data concurrency scheduling. The standard direct memory access controller is responsible for data interaction between the on-chip high-speed static random access memory and the flash memory controller interface, eliminating blocking caused by slow peripheral I / O. The microcontroller is used to execute hardware-accelerated compression algorithms, build and maintain the logical state of the three-tier hybrid index, and coordinate timing synchronization between various hardware components.

[0033] See Figure 3 and Figure 4 The method includes the following steps: Step 1: Write the collected raw time series data points into a circular buffer of tightly coupled memory to temporarily store the time series data; in this embodiment, the raw time series data points are collected by a sensor or analog-to-digital converter module at a fixed frequency.

[0034] In step 1, the circular buffer is managed using head and tail pointers, and its memory address and length both adhere to the fixed byte alignment constraints of the processor bus architecture. This circular buffer, as the first-level high-speed buffer for data entering the system, ensures that data can be temporarily stored without loss when it surges in, preparing for subsequent batch processing.

[0035] Step 2: The main direct memory access controller performs zero-copy data transfer of the timing data in the circular buffer and generates a hardware interrupt notification; In step 2, the primary direct memory access controller performs zero-copy data transfer of the timing data within the circular buffer, specifically including: When the accumulated amount of time-series data in the circular buffer reaches the preset block size, the microprocessor configures the transfer parameters of the main direct memory access controller (MDH) and controls the MDH to initiate a data transfer to achieve zero-copy data transfer of the time-series data. The MDH transfer parameters include a source address and a destination address. The source address is set to the current head pointer position of the circular buffer, and the destination address is set to the data temporary storage area of ​​the on-chip high-speed static random access memory (SRAM). The preset block size is any specific configuration value among 4KB, 8KB, and 64KB. The setting of this preset block size threshold is mainly based on the following two considerations: Balancing bus transfer efficiency with interrupt overhead (reducing fixed overhead): The register configuration, arbitration scheduling, and hardware interrupt response after transfer completion of the main direct memory access controller all incur fixed time overhead. If the threshold is set too small, hardware interrupts will be triggered frequently, causing the microprocessor to enter a period of intensive context switching; conversely, using a larger block size (such as in the 64KB large buffer scenario tested in this system) can fully leverage the advantages of pipeline parallelism, maximize the amortization of the fixed overhead of a single transfer, and improve bus data transfer performance by approximately 10.5%.

[0036] Physical Boundaries of Cache Capacity (Overflow Prevention): Although large block transfers are beneficial to throughput, the total capacity of tightly coupled memory in embedded microcontrollers is extremely limited, and the on-chip high-speed static random access memory must simultaneously accommodate subsequent Gorilla compressed buffers and write-ahead log storage. Therefore, a strict upper limit must be set for the block size threshold (generally not recommended to exceed 64KB) to serve as a benchmark for backpressure flow control, preventing high-concurrency data from instantly overwhelming memory and causing overflow, thus ensuring the absolute stability of the system in resource-constrained environments.

[0037] Step 3: The microprocessor core responds to the hardware interrupt notification, uses hardware acceleration instructions in the on-chip high-speed static random access memory to compress the transferred timing data, obtains compressed data blocks, and constructs metadata corresponding to the compressed data blocks. In step 3, the transferred timing data is compressed using hardware acceleration instructions in the on-chip high-speed static random access memory to obtain compressed data blocks, specifically including: See Figure 5 The process employs Delta-of-Delta dual-track differential encoding to compress the timestamp sequence of the transferred time-series data, and XOR operations to compress the measurement value sequence of the transferred time-series data. During compression, the hardware instruction set in the ARM architecture is used to accelerate variable-length encoding operations, and checkpoints are inserted into the time-series data points at fixed intervals. After compression, the corresponding compressed data block is obtained. The hardware instruction set in the ARM architecture includes CLZ leading zero counting instructions and bitwise operation instructions.

[0038] In this embodiment, when compressing the timestamp sequence of the transported time series data using Delta-of-Delta dual-track differential coding, the first-order difference between adjacent timestamps is calculated first, and then the second-order difference between the first-order differences is calculated. Since most of the second-order difference values ​​are close to zero, only a very small number of bits are needed to complete the compression and storage.

[0039] When compressing the measurement sequence of transported time-series data using the XOR operation, the current measurement value in the sequence is XORed with the previous measurement value to obtain a sequence containing a large number of consecutive leading zeros and trailing zeros, which effectively reduces the number of bits of data storage and achieves a high-efficiency compression effect.

[0040] When using the hardware instruction set in the ARM architecture to accelerate variable-length encoding operations, the CLZ leading zero counting instruction is used to obtain the number of leading zeros in the timing difference between adjacent timing data to achieve efficient on-chip compression encoding. At the same time, bitwise operation instructions are used to quickly complete the alignment calculation of the on-chip high-speed static random access memory storage address to ensure that the compressed data is aligned with the buffer page, providing basic support for the batch regular storage of compressed timing data.

[0041] In addition, this embodiment introduces a checkpoint insertion mechanism, which inserts checkpoints into the time-series data points at fixed intervals, that is, inserts one checkpoint every N original data points. The checkpoint fully stores the original timestamp and measurement value of that point. During decompression, the data is replayed sequentially starting from the nearest checkpoint, thereby maintaining a high compression ratio while providing random access capability.

[0042] In step 3, constructing the metadata corresponding to the compressed data block specifically includes: The Kahan summation algorithm is used to accumulate the additivity indices within the compressed data block, and the Welford online algorithm is used to perform a single pass through the data within the compressed data block to calculate metadata containing statistical values. The statistical values ​​include mean, variance, maximum value, minimum value, sum, and standard deviation. The additivity indices within the compressed data block include time-series measurements with cumulative physical meaning, such as cumulative flow of equipment, power consumption, production count, continuous running time, and number of failures.

[0043] Step 4: Based on the compressed data blocks and their corresponding metadata, complete the construction and update of the three-level hybrid index, and store the compressed data blocks, metadata, and three-level hybrid index in flash memory through the standard direct memory access controller to complete the persistent storage of time-series data; the three-level hybrid index includes the L0 level memory index located in the on-chip high-speed static random access memory, the L1 level incremental index located in flash memory, and the L2 level basic index.

[0044] In step 4, the construction and updating of the three-level hybrid index based on the compressed data blocks and their corresponding metadata specifically includes: Key features of the compressed data block are extracted, including start timestamp, end timestamp, and sequence identifier; these key features are then converted into bit-mapping relationships using a mapping function. Then it is stored in the on-chip high-speed static random access memory to form an L0 level memory index; the L0 level memory index adopts a coarse-grained bitmap structure, in which each bit corresponds to a preset time span. By assigning the corresponding bit position that matches the target compressed data block to a value of 1, the status label of the target compressed data block is realized. In this embodiment, the time span refers to dividing the globally continuous time axis into discrete time window granularities with fixed periods. Under a fixed duration window, the time span can be set to 5 minutes, 30 minutes, or 1 hour. Taking a time span of 5 minutes as an example, a bitmap consisting of a 32-bit unsigned integer resident in the on-chip high-speed static random access memory is used as an example. Its 32 bits from low to high correspond to a time window of 5 minutes within a 160-minute period. When the system receives a compressed data block whose timestamp falls within the 10th to 15th minute (i.e., corresponding to Bit 2), Bit 2 is assigned the value 1 through bit operations.

[0045] The L0-level memory index in this embodiment is used to quickly filter invalid query requests and accelerate the retrieval of recent hot data. When executing a query, the system first scans the bitmap with extremely low latency using hardware instructions (such as CLZ / CTZ) of the microcontroller. If the corresponding bit is 0, it is directly determined that there is no target data in that interval, thereby blocking invalid flash memory reads. If it is 1, the corresponding data block is directly located in high-speed memory, meeting the microsecond-level query response requirements for "latest data" in scenarios such as industrial real-time monitoring screens. When the size of the time-series data covered by the L0-level index reaches the preset memory buffer threshold of the on-chip high-speed static random access memory, the system triggers an index flush operation: the metadata of all compressed data blocks within the time period is serialized to form an L1-level incremental index data packet; the L1-level incremental index data packet is persistently stored in the incremental index partition in the flash memory using an append write method to form an L1-level incremental index; at the same time as the append write, a lightweight mount table is maintained to record the logical order of each L1-level incremental index block in the flash memory; The L1 incremental index is used to alleviate the "write amplification" and CPU power consumption caused by frequent reconstruction of the full index when high-concurrency data is written to disk. It plays a buffer role between the memory index and the global basic index. When the system processes mid-term historical data queries, it only needs to traverse the lightweight mount table to locate the relatively new data that has not yet been merged. While ensuring the ultimate write throughput, it also takes into account the real-time queryability of the data. This implementation presets the on-chip high-speed static random access memory (SRAM) memory buffer threshold to any specific configuration value such as 16KB, 32KB, and 64KB. The basis and reason for setting this threshold are mainly based on the following two considerations: First, the physical capacity of the on-chip high-speed SRAM is extremely limited. The system needs to strictly divide the memory space between data cache, index resident, and program execution stack. If the threshold is set too high, it is easy to cause memory overflow, leading to system crash. Second, it is for optimal alignment with the granularity of flash physical pages. When the accumulated index data packets reach an integer multiple of the flash physical page size, the flush is triggered. This not only maximizes the efficiency of a single transfer, but also completely avoids internal fragmentation and additional erase and write losses caused by write misalignment, and reduces write amplification.

[0046] The system background process monitors the number of L1-level incremental indexes in flash memory in real time. When the number of L1-level incremental indexes reaches a preset merging threshold, it triggers multi-way merge sort to sort the timestamps of the L1-level incremental indexes. Multiple L1-level incremental indexes are read into memory, reordered and deduplicated according to timestamp order, and incrementally merged with the original L2-level base indexes in memory to form an updated L2-level base index. After merging, the node information of the L2-level base index is updated, and the merged L1-level incremental index area is marked as erasable, so as to realize the construction and updating of the three-level hybrid index structure. The L2-level base index is used to support global accurate retrieval and multi-dimensional complex analysis of large-scale historical time-series data. As the final full index form persistently stored in flash memory, it completely eliminates the redundancy and fragmentation problems caused by L1-level append writes through background multi-way merging, constructing a compact index structure with strict time order. When dealing with long-cycle historical backtracking spanning months and years, multi-dimensional tag joint filtering, and large-span aggregation calculations, the L2-level base index can provide stable and low-latency physical addressing and positioning capabilities, which is the core foundation for achieving long-term query performance of time-series databases.

[0047] The merging threshold in this embodiment is based on the physical characteristics of the flash memory and the query latency requirements of the embedded device, as follows: a) Indicator 1 (Quantity Threshold): The number of L1-level incremental index segments reaches 8; b) Indicator 2 (Capacity Threshold): The total storage space occupied by the L1 incremental index reaches 4 times the physical erase block size of the flash memory; c) Triggering logic: Merging can be initiated when either a) or b) is met. This merging threshold has been experimentally determined to maintain high write throughput while ensuring that the depth of the L2-level base index remains at logarithmic time complexity. The ideal range.

[0048] In step 4, storing the compressed data blocks, metadata, and three-level hybrid index into flash memory via a standard direct memory access controller specifically includes: Based on the logical address of the flash memory encapsulated in the compressed data block, the data is recorded in the L1 level incremental index of the flash memory in an append-only manner; The compressed data blocks and their corresponding metadata are moved to the flash memory buffer via a standard direct memory access controller, and the flash memory buffer is aligned with the page size of the flash memory. Initiate write operations to the flash memory for compressed data blocks, metadata, and a three-level hybrid index to complete the persistent storage of time-series data.

[0049] After completing the persistent storage of time-series data, the process also includes merging and optimizing the space storage of the L1-level incremental index and L2-level basic index in flash memory, specifically including: As time-series data is continuously written, the L1-level incremental index will gradually grow. When the size of the L1-level incremental index reaches a preset threshold or the system is in an idle period, a low-priority background task is triggered. This low-priority background task reads the current L2-level base index and all L1-level incremental indexes from the flash memory into the on-chip high-speed static random access memory (SRAM). In the SRAM, it performs multi-way merge sort on the L2-level base index and uses a base-plus-delta merge algorithm to merge the sorted L2-level base index and all L1-level incremental indexes into a unified index structure. During the merge process, the timestamp of the time-series data is used as the primary key, and the sequence ID of the time-series data is used as the secondary key. The sorted L2-level base index and all L1-level incremental indexes are then compared and overlapped. This embodiment uses a base-plus-delta (Base+Delta) merge algorithm. When the L1-level incremental index grows to a threshold, it is logically merged with the L2-level base index in memory and then written back to disk, avoiding the overhead of a full rebuild and effectively controlling index bloat.

[0050] The merged unified index structure is written sequentially to a new location in the flash memory, the system's root pointer is updated to point to the unified index structure, and the merged L2-level basic indexes and all L1-level incremental indexes are reclaimed and cleaned up to optimize flash memory storage space.

[0051] See Figure 6After the time-series data has been persistently stored, if you need to retrieve or query the time-series data, perform the following operations: When a query request is received, the query engine parses the time range of the query; Coarse-grained filtering is performed using the L0 level memory index resident in the on-chip high-speed static random access memory to identify candidate data blocks that match the query time range; When the query request is an aggregation query request, candidate data blocks are read, and the statistical values ​​in the metadata corresponding to the candidate data blocks are merged and calculated to obtain a global aggregation result corresponding to the query time range. The overall aggregation result is encapsulated according to a preset return format and returned directly to complete the response to the aggregation query request. The global aggregation result includes the global sum (SUM), global maximum value (MAX), global minimum value (MIN), global mean (AVG), total number of data points (COUNT), and global variance and standard deviation (VAR / STD) within the target time range.

[0052] When the query request is a detailed query request for raw time series data points, the L1-level incremental index and L2-level basic index of the flash memory are used to locate the candidate data block containing the target raw time series data point. The candidate data block containing the target raw time series data point is read, and the query engine locates the upstream checkpoint closest to the query request timestamp based on the timestamp of the query request. Based on the nearest upstream checkpoint, a data coverage window corresponding to the query time is defined, and the time series data within the data coverage window is sequentially incrementally decompressed. The raw time series data points that meet the query time conditions are parsed and extracted from the decompressed time series data, and then packaged and returned according to the preset return format to complete the detailed query of the raw data points.

[0053] Without interfering with the high-frequency data collection at the front end, the backend maintenance layer achieves self-recycling of system resources and self-balancing of performance through the following three core automatic execution modules: Hot and cold partition monitoring: Real-time scanning of the sequence access heat table in the on-chip high-speed static random access memory, accurately locating "cold data" that has not been updated for a long time and triggering degradation processing to release tightly coupled memory resources in a timely manner.

[0054] Level 3 index merging: Starts during system idle intervals, uses a multi-way merge sort algorithm to logically merge the fragmented L1 incremental index in flash memory with the L2 basic index, and re-persistes it using dual-zone atomic switching technology, effectively eliminating the risk of linear index expansion and ensuring stable and low-latency retrieval of historical data.

[0055] Flash wear leveling and garbage collection: Real-time tracking of write and erase counts for each physical sector, prioritizing the allocation of merged, well-organized data to sectors with low write / erase rates, and utilizing a batch sequential write mechanism to ensure strict physical page alignment. This mechanism eliminates fragmentation caused by frequent small packet writes at the underlying level, significantly reducing write amplification and ensuring the full lifecycle reliability of low-resource embedded devices under industrial-grade high concurrency.

[0056] The application effects of this invention will be described in detail below with reference to simulation experiments: To objectively and quantitatively evaluate the comprehensive performance and engineering practical value of the embedded time-series data storage method (named EmbedTSDB) based on multi-level storage and multi-level DMA acceleration of this invention, comprehensive simulation tests were conducted on EmbedTSDB based on this invention under a standard embedded hardware platform and software environment, and a comparative analysis was performed with FlashDB, a similar open-source solution in the industry. The tests aimed to verify the actual performance of this invention in key indicators such as write throughput, query response latency, storage space efficiency, CPU resource consumption, and Flash lifespan optimization.

[0057] I. Test Environment and Benchmark Hardware platform: The STM32H743 microcontroller (ARM Cortex-M7 @ 480MHz, DTCM 128KB, AXI-SRAM 512KB, external NAND Flash 512MB) is used as a unified testing platform to ensure the fairness of the comparison.

[0058] Software environment: Based on RT-Thread 4.1.1 real-time operating system, integrating FAL abstraction layer, FlashDB (as a comparison benchmark) and EmbedTSDB scheme based on the present invention.

[0059] Test dataset: Based on Zipf's law, a time-series data stream that conforms to the characteristics of IoT scenarios is generated, including multi-dimensional labels, differentiated sampling frequencies and numerical distributions. The data scale dynamically expands from 100,000 to 45 million records to test the performance boundaries of the system under different loads.

[0060] Benchmark: FlashDB, a lightweight database designed specifically for MCUs, was selected for comparison. To ensure fairness in the storage space comparison, the available data space of FlashDB was also configured to 440MB.

[0061] II. Functional Correctness Verification In functional testing, all core processes of EmbedTSDB based on this invention exhibited behavior consistent with design expectations: In terms of multi-level storage and data flow: the TCM ring buffer achieved zero data loss reception in high-concurrency write scenarios, the data flow between the AXI-SRAM temporary storage area and the Flash persistent layer was correct and error-free, and the WAL log mechanism was able to fully recover uncommitted data after a simulated crash, proving the system's data consistency guarantee capability.

[0062] Compression and Random Access: The Gorilla compression algorithm, accelerated by integrated hardware instructions (CLZ / CTZ), achieves an average compression rate of 49.6% for different types of sensor data (temperature, humidity, acceleration), effectively saving nearly half of the storage space. At the same time, the interval checkpoint mechanism inserted in the compressed stream reduces the location latency for random time window queries of long time-series data to less than 10% of the time spent on full decompression, enabling efficient and accurate access to compressed data.

[0063] Level 3 indexes and L0-level memory indexes support real-time response for multi-dimensional label combination filtering; the basic plus incremental merging strategy of L1-level incremental indexes and L2-level basic indexes is effective, and the index expansion rate remains stable within 10% under long-term operation; pre-aggregation queries (SUM, AVG, etc.) are completed by directly reading the statistical values ​​maintained by the Welford / Kahan algorithm in the block metadata, avoiding the decompression of the underlying data and significantly improving the response speed.

[0064] III. Quantitative Analysis of Performance Advantages Referring to Table 1, the performance advantages of this invention are quantified by comparing the performance of EmbedTSDB and FlashDB based on this invention under the same hardware and data load: Table 1. Performance Comparison of EmbedTSDB (Invention) and Traditional FlashDB under the Same Hardware and Data Load. Evaluation Dimensions Test scenarios / data scale EmbedTSDB results FlashD Results Advantages Analysis Maximum write throughput Continue writing until storage space is exhausted Writing 45 million data entries took approximately 285 minutes. The flash memory was exhausted after writing 28.5 million data entries. EmbedTSDB utilizes compression and hot / cold data separation to store 58% more data in the same physical space, resulting in a significant increase in effective storage capacity. Storage space usage Write the same amount of data (e.g., 10 million records). The total size is approximately 60KB (after compression). Total size: 120KB (raw storage) Thanks to the variable-length compression algorithm, storage space is saved by about 50%, which greatly alleviates the problem of tight storage resources in MCUs. CPU utilization High-throughput continuous write scenarios Average CPU utilization <25% Average CPU utilization >70% By offloading the data transfer burden through multi-level DMA chain operations, the CPU utilization rate was reduced by about 64%, and the freed computing power could be used for other real-time tasks. CPU utilization High-throughput continuous write scenarios Average CPU utilization <25% Average CPU utilization >70% By offloading the data transfer burden through multi-level DMA chain operations, the CPU utilization rate was reduced by about 64%, and the freed computing power could be used for other real-time tasks.

[0065] IV. Comprehensive Experimental Conclusions The simulation results demonstrate that the embedded timing data storage method based on multi-level storage and multi-level DMA acceleration has significant advantages on resource-constrained microcontroller platforms such as the STM32H7 series. 1) Complete and reliable functions: The storage, compression, indexing and query modules at all levels of the system are working normally, and the data consistency and crash recovery capabilities have been verified.

[0066] 2) Overall performance improvement: Compared with the traditional solution (FlashDB), it has achieved orders-of-magnitude improvement or fundamental improvement in three key dimensions: storage efficiency, query speed, and CPU resource release.

[0067] 3) High engineering practicality: It effectively solves the core contradictions of rapid storage medium wear, high query latency and heavy CPU load in embedded time-series data processing, and provides high-performance and low-resource-consumption data management solutions for IoT edge computing, industrial sensing and other scenarios.

[0068] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions or improvements made by those skilled in the art within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration, applied to an ARM architecture including tightly coupled memory, on-chip high-speed static random access memory, flash memory, main direct memory access controller, standard direct memory access controller, and microprocessor, characterized in that, Includes the following steps: Step 1: Write the collected raw time series data points into a tightly coupled memory circular buffer to temporarily store the time series data; Step 2: The main direct memory access controller performs zero-copy data transfer of the timing data in the circular buffer and generates a hardware interrupt notification; Step 3: The microprocessor core responds to the hardware interrupt notification, uses hardware acceleration instructions in the on-chip high-speed static random access memory to compress the transferred timing data, obtains compressed data blocks, and constructs metadata corresponding to the compressed data blocks. Step 4: Based on the compressed data blocks and their corresponding metadata, complete the construction and update of the three-level hybrid index, and store the compressed data blocks, metadata, and three-level hybrid index in flash memory through the standard direct memory access controller to complete the persistent storage of time-series data; the three-level hybrid index includes the L0 level memory index located in the on-chip high-speed static random access memory, the L1 level incremental index located in flash memory, and the L2 level basic index.

2. The embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1, characterized in that, In step 1, the circular buffer is managed using head pointers and tail pointers, and the memory address and length of the circular buffer both follow the fixed byte alignment constraints of the processor bus architecture.

3. An embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1 or 2, characterized in that, In step 2, the primary direct memory access controller performs zero-copy data transfer of the timing data within the circular buffer, specifically including: When the amount of time-series data in the circular buffer reaches the preset block size, the microprocessor configures the transmission parameters of the main direct memory access controller and controls the main direct memory access controller to initiate a data transmission to achieve zero-copy data transfer of time-series data. The transmission parameters of the main direct memory access controller include the source address and the destination address. The source address is set to the current head pointer position of the circular buffer, and the destination address is set to the data temporary storage area of ​​the on-chip high-speed static random access memory.

4. The embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1, characterized in that, In step 3, the transferred timing data is compressed using hardware acceleration instructions in the on-chip high-speed static random access memory to obtain compressed data blocks, specifically including: Incremental-to-increment dual-track differential coding is used to compress the timestamp sequence of the transported time-series data, and XOR operation is used to compress the measurement value sequence of the transported time-series data. During the compression process, the hardware instruction set in the ARM architecture is used to accelerate variable-length coding operations, and checkpoints are inserted into the time-series data points at fixed intervals. After compression, the corresponding compressed data block is obtained.

5. The embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 4, characterized in that, The hardware instruction set in the ARM architecture includes CLZ leading zero counting instructions and bitwise operation instructions.

6. The embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1, characterized in that, In step 3, constructing the metadata corresponding to the compressed data block specifically includes: The Kahan summation algorithm is used to accumulate the additivity index within the compressed data block, and the Welford online algorithm is used to perform a single pass through the time series data within the compressed data block to calculate metadata containing statistical values; the statistical values ​​include mean, variance, maximum value, minimum value, sum, and standard deviation.

7. The embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1, characterized in that, In step 4, the construction and updating of the three-level hybrid index based on the compressed data blocks and their corresponding metadata specifically includes: Key features of the compressed data block are extracted, including start timestamp, end timestamp, and sequence identifier; these key features are then converted into bit-mapping relationships using a mapping function. Then it is stored in the on-chip high-speed static random access memory to form an L0 level memory index; the L0 level memory index adopts a coarse-grained bitmap structure, in which each bit corresponds to a preset time span. By assigning the corresponding bit position that matches the target compressed data block to a value of 1, the status label of the target compressed data block is realized. When the size of the time-series data covered by the L0-level index reaches the preset memory buffer threshold of the on-chip high-speed static random access memory, the system triggers an index flush operation: the metadata of all compressed data blocks within the time period is serialized to form an L1-level incremental index data packet; the L1-level incremental index data packet is persistently stored in the incremental index partition in the flash memory using an append write method to form an L1-level incremental index; at the same time as the append write, a lightweight mount table is maintained to record the logical order of each L1-level incremental index block in the flash memory; The system background process monitors the number of L1-level incremental indexes in flash memory in real time. When the number of L1-level incremental indexes reaches a preset merging threshold, it triggers multi-way merge sort to sort the timestamps of the L1-level incremental indexes. Multiple L1-level incremental indexes are read into memory, reordered and deduplicated according to their timestamps, and incrementally merged with the existing L2-level base indexes in memory to form an updated L2-level base index. After merging, the node information of the L2-level base indexes is updated, and the merged L1-level incremental index area is marked as erasable, thus realizing the construction and updating of a three-level hybrid index structure.

8. The embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1, characterized in that, In step 4, storing the compressed data blocks, metadata, and three-level hybrid index into flash memory via a standard direct memory access controller specifically includes: Based on the logical address of the flash memory encapsulated in the compressed data block, the data is recorded in the L1 level incremental index of the flash memory in an append-only manner; The compressed data blocks and their corresponding metadata are moved to the flash memory buffer via a standard direct memory access controller, and the flash memory buffer is aligned with the page size of the flash memory. Flash memory initiates write operations to compressed data blocks, metadata, and a three-level hybrid index to complete the persistent storage of time-series data.

9. An embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1 or 7, characterized in that, After completing the persistent storage of time-series data, the process also includes merging and optimizing the space storage of the L1-level incremental index and L2-level basic index in flash memory, specifically including: As time-series data is continuously written, the L1-level incremental index will gradually grow. When the size of the L1-level incremental index reaches a preset threshold or the system is in an idle period, a low-priority background task is triggered. This low-priority background task reads the current L2-level basic index and all L1-level incremental indexes from the flash memory into the on-chip high-speed static random access memory (SRAM). In the SRAM, it performs a multi-way merge sort on the L2-level basic indexes and then uses a basic plus incremental merge algorithm to merge the sorted L2-level basic indexes and all L1-level incremental indexes into a unified index structure. During the merging process, the timestamp of the time-series data is used as the primary index key, and the sequence ID of the time-series data is used as the secondary index key. The sorted L2-level basic indexes and all L1-level incremental indexes are then compared and overlapped. The merged unified index structure is written sequentially to a new location in the flash memory, the system's root pointer is updated to point to the unified index structure, and the merged L2-level basic indexes and all L1-level incremental indexes are reclaimed and cleaned up to optimize flash memory storage space.

10. An embedded time-series data storage method based on multi-level storage and multi-level DMA acceleration according to claim 1, 6, or 7, characterized in that, After the time-series data has been persistently stored, if you need to retrieve or query the time-series data, perform the following operations: When a query request is received, the query engine parses the time range of the query; Coarse-grained filtering is performed using the L0 level memory index resident in the on-chip high-speed static random access memory to identify candidate data blocks that match the query time range; When the query request is an aggregate query request, candidate data blocks are read, and the statistical values ​​in the metadata corresponding to the candidate data blocks are merged and calculated to obtain a global aggregate result corresponding to the query time range. The overall aggregation result is encapsulated according to the preset return format and returned directly to complete the response to the aggregation query request; When the query request is a detailed query request for raw time series data points, the L1-level incremental index and L2-level basic index of the flash memory are used to locate the candidate data block containing the target raw time series data point; the candidate data block containing the target raw time series data point is read, and the query engine locates the upstream checkpoint closest to the query request timestamp based on the timestamp of the query request; using the nearest upstream checkpoint as a reference, a data coverage window corresponding to the query time is defined, and the time series data within the data coverage window is sequentially incrementally decompressed; The original time series data points that meet the query time conditions are parsed and extracted from the decompressed time series data, and then packaged and returned according to the preset return format to complete the detailed query of the original data points.

Images