Data acquisition platform three-level data storage method and system

Abstract

This invention discloses a three-level data storage method and system for a data acquisition platform. The method includes: periodically writing the acquired real-time data into the collector's memory circular buffer at a first time interval; when a fault trigger signal is received, extracting data before and after the fault time from the memory circular buffer and packaging it into a fault data packet; periodically writing the acquired real-time data into the hot data area of ​​the collector's local database at a second time interval, wherein the second time interval is longer than the first time interval; periodically migrating historical data exceeding the retention period in the hot data area to the cold data storage area; matching a corresponding compression strategy according to the current occupancy rate threshold range, and compressing and archiving files in the cold data storage area based on the compression strategy. The above data storage method, through a three-level hierarchical storage architecture, achieves a deep balance between data accuracy, query speed, and storage capacity in a resource-constrained environment.

Description

Technical Field

[0001] This invention relates to the field of data acquisition and dynamic storage technology, and in particular to a three-level data storage method and system for a data acquisition platform. Background Technology

[0002] With the rapid development of large-scale industrial equipment such as energy storage power stations, the requirements for the precision of equipment operation status monitoring are becoming increasingly stringent. Taking energy storage battery management systems (BMS) or data acquisition platforms for large industrial equipment as examples, a single device typically needs to collect tens of thousands of analog and status data points. In order to meet increasingly stringent industry standards (such as the requirements of the national standard GB / T 34131-2023 for energy storage BMS), data acquisition platforms not only need to store historical operating data for a long period (such as no less than 120 days), but are also required to have extremely high fault tracing accuracy, that is, they must completely record high-frequency, fine-grained data before and after the alarm trigger moment (such as 10 seconds before and after).

[0003] However, in practical industrial applications, data acquisition platforms typically employ resource-constrained embedded hardware platforms. Limited by cost and power consumption, the memory capacity and space for local non-volatile storage media (such as eMMC, Flash, etc.) of such devices are often very limited (e.g., only 2GB of memory and 8GB of storage). Faced with hundreds or even thousands of megabytes of raw time-series data daily, an irreconcilable contradiction arises between the need for long-term data retention and the limited hardware storage capacity.

[0004] To address the aforementioned issues, existing data storage and processing solutions suffer from the following main shortcomings: First, existing data acquisition devices typically store historical data in conventional row-based CSV or basic database formats. This row-based time-series data does not effectively utilize the high correlation of industrial time-series data along the same measurement point time axis. When archiving using common compression algorithms (such as gzip), the compression ratio usually only reaches about 10-20 times, and the compressed data volume is still enormous, making it impossible to store months of historical data in just a few GB of storage space. If space is saved by reducing the sampling frequency or discarding older data, it cannot meet the requirements for compliance review of national standards and rapid querying of recent data.

[0005] Second, most existing solutions write all data directly to persistent storage media at a fixed low-to-medium frequency cycle. When sudden faults such as short circuits or overheating occur, which are on the order of seconds or even milliseconds, the lack of an independent high-frequency caching mechanism often prevents the system from fully capturing and extracting the high-frequency data before and after the fault. This results in insufficient granularity of the generated fault slice data, making it difficult to meet the standard's requirement for tracing detailed data before and after the fault.

[0006] Third, existing data acquisition platforms often write frequently collected data directly and directly to local databases and non-volatile storage media (such as eMMC) during operation. Massive fragmented writes and unoptimized database logging mechanisms (such as frequent WAL log writes to disk) significantly increase the write amplification effect of storage media, causing the write cycles of storage chips to be exhausted quickly, severely shortening the overall lifespan of industrial equipment.

[0007] Therefore, there is an urgent need in this field for a new intelligent data storage solution. Summary of the Invention

[0008] The purpose of this invention is to provide a three-level data storage method and system for a data acquisition platform that can simultaneously achieve precise fault capture at the second level, rapid query of recent data, and extreme compression and archiving of long-term data under limited hardware resources.

[0009] To achieve the above objectives, the present invention provides a three-level data storage method for a data acquisition platform, comprising: The collected real-time data is periodically written into the collector's memory circular buffer at intervals of the first duration. When a fault trigger signal is received, data before and after the fault moment is extracted from the memory circular buffer and packaged into a fault data packet. The collected real-time data is periodically written into the hot data area of ​​the local database of the collector at a second time interval, wherein the second time interval is longer than the first time interval; Historical data that has exceeded its retention period in the hot data area is periodically migrated to the cold data storage area. The migration method includes: Read historical data from the local database and convert row-based time-series data into column-based time-series data to obtain a column-structured file; Obtain the current occupancy rate of the storage medium, match the corresponding compression strategy according to the threshold range of the current occupancy rate, and compress and archive the columnar structure file based on the compression strategy.

[0010] Preferably, data within a preset time period before and after the fault occurs is extracted, converted into an independent file, compressed to generate the fault data packet, and an alarm record corresponding to the fault trigger signal is inserted into the local alarm table.

[0011] Preferably, the method for writing the collected real-time data into the hot data area of ​​the collector's local database includes: The real-time data is serialized according to different measurement point types and converted into binary data; Within a transaction commit cycle consisting of multiple second durations, the binary data is cached, and at the end of the transaction commit cycle, it is committed and written to the hot data area in a batch.

[0012] Preferably, the local database is configured to use a write-ahead log mode. The shared memory files of the local database are redirected to a memory-based temporary file system; and incremental space reclamation operations are performed on the local database after periodically migrating historical data.

[0013] Preferably, the method for obtaining the columnar structure file further includes: The historical data in the local database is read in batches according to time period and object; Establish a mapping table with the measurement point identifier as the key and the numerical sequence of the measurement point within the current reading time period as the value; The results read in batches are deserialized, and the measurement values ​​at each time point are appended to the numerical sequence corresponding to the mapping table. The columnar structure file is then output based on the mapping table.

[0014] Preferably, the method for matching a corresponding compression strategy based on the threshold range of the current occupancy rate includes: When the current occupancy rate is less than the first preset threshold and the device monitored by the data acquisition platform is in a silent state, a high compression ratio mode is triggered, and the first compression algorithm is used for compression. The silent state refers to the device monitored by the data acquisition platform having no power fluctuation or being in standby mode within a preset time period. When the current occupancy rate is greater than or equal to the first preset threshold and less than the second preset threshold, a balancing mode is triggered, and a second compression algorithm is used for compression; the compression ratio of the second compression algorithm is lower than that of the first compression algorithm, and the compression speed is higher than that of the first compression algorithm; When the current occupancy rate is greater than or equal to the second preset threshold, a fast mode is triggered, and a third compression algorithm is used for compression; the compression speed of the third compression algorithm is higher than that of the second compression algorithm.

[0015] Preferably, when the current occupancy rate is greater than or equal to the second preset threshold, an emergency mode is triggered, and the following operations are performed simultaneously: Execute the cold data cleanup task to delete the oldest compressed archive file stored in the cold data storage area; The retention period for historical data in the hot data area will be shortened.

[0016] The present invention also provides a three-level data storage system for a data acquisition platform, which stores the acquired data based on the three-level data storage method for a data acquisition platform described above.

[0017] This invention also provides a three-level data storage system for a data acquisition platform, comprising: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the three-level data storage method of the data acquisition platform as described above.

[0018] The present invention also provides a computer-readable storage medium, characterized in that it includes a computer program, which can be executed by a processor to perform the three-level data storage method of the data acquisition platform as described above.

[0019] Compared to existing technologies, the three-tiered data storage method provided by the above-mentioned technical solution achieves a deep balance between data accuracy, query speed, and storage capacity in resource-constrained environments through a three-tiered storage architecture. First, the memory circular buffer ensures the immediacy and precision of fault data capture, meeting high-standard compliance traceability requirements. Second, the local database hot data area, through long-cycle writing, balances the fast query performance of recent data with optimized storage media lifespan. Third, the cold data storage area, through columnar rearrangement technology, fully exploits the correlation of time-series data, greatly improving compression efficiency. Furthermore, coupled with an adaptive compression strategy based on storage occupancy, the system can dynamically switch compression algorithms according to resource availability, ensuring both real-time system operation and reliable archiving of massive amounts of data over extremely long periods within limited storage space. This effectively resolves the core contradiction between limited storage capacity in embedded devices and the requirement for long-term data retention. Attached Figure Description

[0020] Figure 1 This is a flowchart of the data storage method in an embodiment of the present invention.

[0021] Figure 2 This is a timing diagram for fault data generation in an embodiment of the present invention. Detailed Implementation

[0022] To illustrate the technical content, structural features, objectives, and effects of the present invention in detail, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0023] This embodiment discloses a three-level data storage method for a data acquisition platform, primarily applied to resource-constrained data acquisition hardware platforms in large industrial equipment, such as battery management systems (BMS). These platforms typically face challenges due to limited memory and non-volatile storage media (such as eMMC and Flash), yet require stringent specifications to meet the demands of long-term, massive historical data storage and high-precision fault tracing. This embodiment achieves efficient, layered data flow by constructing a three-level architecture consisting of a memory circular buffer, a local database hot data area, and a cold data storage area. Figure 1 The method specifically includes the following steps: S11: The collected real-time data is periodically written into the collector's memory circular buffer at intervals of the first duration.

[0024] In hardware or software architectures, a memory circular buffer is configured within volatile memory (RAM). The data acquisition platform acquires analog and state data from data sources (such as underlying sensors) in real time and continuously writes them into this memory circular buffer at a high-frequency cycle that strictly adheres to a first duration (e.g., 1 second).

[0025] The circular buffer uses a first-in, first-out (FIFO) mechanism, which means that when the set depth of records is filled, the latest data will automatically overwrite the oldest data, thus always maintaining a high-frequency real-time data slice in memory within a fixed time window (such as the most recent 60 seconds).

[0026] Those skilled in the art should understand that a circular memory buffer can be implemented using a third-party memory database (such as Redis's List structure combined with LPUSH and LTRIM instructions), or by allocating a fixed-size contiguous memory array in C / C++ in conjunction with read / write pointers.

[0027] S12: When a fault trigger signal is received, extract the data before and after the fault time from the memory circular buffer and package it into a fault data packet.

[0028] During equipment operation, once the diagnostic module detects an abnormal state exceeding the safety threshold (such as a level one or level two alarm signal), it immediately generates a fault trigger signal. Upon receiving this signal, the system stops overwriting the data segment and directly exports a dataset from the memory circular buffer, extending pre-set durations before and after the alarm time, and then isolates and archives it. The generation of the fault data packet ensures the integrity and high precision of the data during sudden anomalies.

[0029] S13: The collected real-time data is periodically written into the hot data area of ​​the collector's local database at intervals of the second duration, wherein the second duration is longer than the first duration, and preferably the second duration is three times or more than the first duration.

[0030] Simultaneously, in another parallel data stream, with a second duration (e.g., 5 seconds) as the sampling interval, the full data of the device at the current moment is written into a local relational or time-series database in a non-volatile storage medium.

[0031] Because the second duration is longer than the first, the data density written to the hot data area is lower than that in the memory buffer, which acts as the first data down-rate on a large time scale. The hot data area is configured to retain data from the most recent period (such as the last 3 or 7 days). This data is not deeply compressed and supports SQL-level fast retrieval and second-level response in the front-end interactive interface.

[0032] S14: Periodically migrate historical data that has exceeded the retention period in the hot data area to the cold data storage area.

[0033] Over time, local databases accumulate a large amount of historical data. A scheduled task (e.g., daily at midnight) is used to filter out records that have exceeded the hot data retention period and perform cold archiving. The migration process is as follows: First, read the historical data from the local database and convert the original row-based time-series data into column-based time-series data to obtain a column-based structure file.

[0034] In conventional databases, data is stored in rows with timestamps as the primary key, meaning each row contains the values ​​of all measurement points at the current moment. However, the data from a single measurement point of a physical device (such as the voltage and temperature of a single battery cell) often exhibits strong smoothness and continuity over time. Converting rows to columns clusters the values ​​of the same measurement point within a continuous timeframe, generating numerous repeating byte sequences or regularly increasing sequences. This columnar rearrangement mechanism significantly alters the entropy distribution of the data, providing an optimal data arrangement for subsequent dictionary lookups or block sorting compression algorithms.

[0035] S15: Then, obtain the current occupancy rate of the storage medium, match the corresponding compression strategy according to the threshold range of the current occupancy rate, and compress and archive the columnar structure file based on the compression strategy.

[0036] Before compression, the percentage of used file system space on the target storage medium is dynamically obtained through operating system interfaces (such as the vfsstat or df command in Linux). The system pre-configures multiple threshold ranges, each mapping to a compression algorithm and its operating parameters. Since different compression algorithms vary significantly in compression ratio, CPU usage, and execution time, the system adaptively selects the compression strategy best suited to the current system context based on the current disk availability.

[0037] By combining columnar rearrangement with adaptive compression algorithms, massive amounts of raw data can be compressed to a very small size while ensuring that real-time control tasks are not disturbed, thus accommodating months of historical records within limited physical storage.

[0038] Another embodiment, such as Figure 2 The specific process of extracting data before and after the fault from the memory circular buffer and packaging it into a fault data packet is as follows: Extract data within a preset time period before and after the fault occurrence (e.g., 30 seconds before the alarm occurs to 30 seconds after the alarm). Then, convert the extracted pure memory data structure into a common independent file format (e.g., CSV file) and use a packaging and compression tool to generate an independent fault archive package.

[0039] To facilitate subsequent troubleshooting, fault data packets adopt standardized naming rules, such as "fault_timestamp_alarm type.tar.bz2", and are centrally stored in a dedicated fault isolation directory (such as / data / faults / ).

[0040] While packaging the fault data packet, an alarm record corresponding to the fault trigger signal is also inserted into the alarm table of the local database. The record content includes, but is not limited to: the subsystem cluster number that triggered the fault, the device number, the alarm description, the status, and the precise timestamp.

[0041] In this embodiment, by employing independent high-frequency slice extraction and alarm log association technology, the system not only captures physical quantity fluctuations at the second or even millisecond level that far exceed the normal period (such as 5 seconds), but also achieves hard binding between alarm events and underlying raw data. This solves the problem of core field data loss caused by conventional storage solutions failing to flush fault cache to disk in time during system crashes.

[0042] In another embodiment, to address the write amplification and system I / O bottleneck issues caused by high-frequency writing of massive measurement point data to embedded storage chips, a method for writing the acquired real-time data to the hot data area of ​​the collector's local database includes: First, at the application layer, real-time data is structured according to different measurement point types (e.g., analog quantities are divided into individual voltage sets, individual temperature sets, total equipment current, etc.), and the corresponding data arrays are directly serialized by memory copying, packaging them as a whole into unformatted binary data (BLOB format). By avoiding the process of converting thousands of integer or floating-point measurement points into SQL strings one by one and then parsing them, the CPU serialization overhead is significantly reduced.

[0043] Instead of immediately triggering a database write operation when each second duration period arrives, the system caches the serialized binary data in application-layer memory within a transaction commit cycle consisting of multiple second duration periods (e.g., every 3 collection cycles, or 15 seconds). When the transaction commit cycle ends, a single database transaction is initiated to batch commit and write the cached records to the hot data area.

[0044] By employing the aforementioned BLOB field merging and aggregation mechanism, along with the multi-cycle transaction batch commit mechanism, the frequency of database engine creation of B-Tree indexes and manipulation of file descriptors is significantly reduced, effectively lowering write latency. In practical applications, even with tens of thousands of test points, this strategy still ensures that system bus bandwidth is not monopolized by database I / O.

[0045] Furthermore, considering the limited write / erase life of flash memory media (such as eMMC) in embedded devices, and the fact that frequent file updates can easily lead to premature device failure, this embodiment optimizes the underlying mechanism of the local database, as follows: Enable Write-Ahead Logging (WAL) mode for the local database. In WAL mode, modifications do not directly overwrite the original database files, but are appended to a separate WAL log. To completely eliminate the wear and tear on flash memory chips caused by frequent log file updates, this solution redirects the shared memory file of the local database (such as the -shm file in an SQLite database, used to coordinate concurrent access to WAL by multiple processes) to a memory-based temporary file system (tmpfs, such as the / dev / shm path in Linux).

[0046] For data acquisition devices equipped with power-loss protection circuits, the present invention further provides an optimized implementation method for database memory-based processing: Redirecting the SQLite database's shared memory file (-shm) and WAL log file (-wal) to a memory-based temporary file system (tmpfs, such as the / dev / shm directory), while the main database file (.db) remains in eMMC persistent storage, significantly improves write performance while ensuring data security.

[0047] The security mechanism is as follows: 1. WAL Log In-Memory: The WAL log file resides in tmpfs, and all transaction log writes are completed in memory, significantly reducing eMMC writes. Since the master database file is still in eMMC, at most, unsynchronized transactions in the WAL (i.e., new data after the last checkpoint) will be lost during a power outage.

[0048] 2. Active Checkpoint Strategy: After each transaction commit cycle (e.g., after a batch commit every 15 seconds), the system actively calls sqlite3_wal_checkpoint_v2 to perform a passive checkpoint, synchronizing the transactions in the WAL to the master database file. This strategy limits the data window that may be lost due to power outages to within a single transaction commit cycle (≤15 seconds).

[0049] 3. Power-down protection linkage: When the power-down protection circuit detects a drop in main power supply, it sends an interrupt signal. The interrupt service routine performs emergency operations within the 3-5 second retention period: Calling sqlite3_wal_checkpoint forces a full checkpoint, committing all unsynchronized transactions in the WAL to the master database file (this operation typically takes less than 100 milliseconds because the WAL file only contains incremental data from the last 15 seconds, and is very small in size). Once the checkpoint is complete, notify the system to power off.

[0050] 4. Redundancy protection: The system executes PRAGMA incremental_vacuum once during idle periods (such as every hour) and verifies the integrity of the database to ensure that the main database file can be recovered normally after an abnormal power outage.

[0051] In addition, since daily or periodic cold data migrations remove a large number of expired rows from the hot data area, resulting in a large number of fragmented space holes, after periodically migrating historical data, an incremental space reclamation command (such as executing the PRAGMA incremental_vacuum operation) is sent to the database engine to smoothly release idle data pages by limiting the number of pages each time (such as 500 pages).

[0052] The above solution locks a large number of extremely fine read / write interactions and log flow at the memory level. In actual tests, it can reduce the amount of writing to the physical eMMC by more than 60%, which greatly extends the reliability of the device throughout its entire life cycle.

[0053] In another embodiment, considering that the data acquisition platform cannot load the entire massive amount of data spanning several days into memory at once for transpose matrix calculation, the method for obtaining the columnar structure file specifically includes: First, using a two-level index of time interval and spatial dimension, the historical data to be migrated in the local database is read in batches by time period (e.g., dividing 24 hours of a day into 4 6-hour periods) and belonging to the object (e.g., different battery cluster numbers 1 to 16).

[0054] Then, a dynamic mapping table (Map / Dictionary structure) is established at the application layer. This mapping table uses a specific measurement point identifier (e.g., the string "clu1_volt_5") as the key and a continuous sequence (e.g., a QVector vector or array) of all historical values ​​of that measurement point within the current batch reading time period as the value.

[0055] Next, each record containing BLOB data retrieved from the database in the current batch is sequentially traversed, the BLOB is deserialized, and each specific measurement point value is extracted and appended to the end of the value sequence of the corresponding key in the mapping table in chronological order.

[0056] Finally, after a batch of data has been traversed, a columnar structure file for that period is output based on the mapping table: when generating the CSV file, the first row outputs the time series and the identifier of each measurement point as the table header, and each subsequent row only outputs the values ​​extracted from the mapping table at a specific time point in a horizontally flat manner.

[0057] This time-segmented pipeline processing mechanism not only effectively realizes columnar rearrangement and aggregation of data in the time dimension, but also strictly controls the peak memory usage during the processing to within a safe limit (such as 190MB), ensuring the stability of memory-constrained systems when handling heavy-load tasks.

[0058] In another embodiment, before performing cold data archiving, the current file system occupancy is detected. Based on the threshold range of the current occupancy, as shown in Table 1 below, the method for adaptively matching and executing different compression strategies is as follows: Scenario 1: When the current occupancy rate is less than a first preset threshold (e.g., 85%) and the device monitored by the data acquisition platform is in a silent state, it is determined that current resources are sufficient and interference tolerance is high. At this time, a high compression ratio mode is triggered. In this mode, the first compression algorithm (such as the bzip2 algorithm) is used for processing, which consumes a longer time and certain computing resources in exchange for an extreme compression ratio (e.g., a compression ratio of 48:1). It should be noted that a silent state means that the device monitored by the data acquisition platform has no power fluctuation or is in a standby state within a preset time period.

[0059] Scenario 2: When the current occupancy rate is greater than or equal to the first preset threshold and less than the second preset threshold (e.g., 90%), the system is in a resource warning state. At this time, a balancing mode is triggered, and the system is forcibly woken up to execute a compression task, using a second compression algorithm (such as gzip). The second compression algorithm sacrifices some compression ratio, but its running speed is significantly higher than the first compression algorithm, allowing it to smoothly pass the warning period with moderate resource consumption.

[0060] Scenario 3: When the current occupancy rate is greater than or equal to the second preset threshold, the system faces a serious risk of running out of storage space. At this time, the fast mode is triggered, and the third compression algorithm (such as the zstd algorithm) is used. The third compression algorithm quickly completes the encapsulation and transfer of the current data with an execution speed far exceeding that of the second compression algorithm, preventing secondary failures caused by the real-time system being full of disks.

[0061] Furthermore, to ensure that the core operations of the data acquisition platform are not interfered with to the greatest extent possible, those skilled in the art will understand that when executing any compression and archiving process, the CPU scheduling priority of the process can be lowered to the lowest level through operating system-level instructions (such as the nice command in Linux), and the maximum disk input / output bandwidth of the process can be limited (e.g., limited to within 10MB / s) using control group technology. This dynamic scheduling system combining hardware and software completely solves the persistent problem of traditional timed compression tasks preempting the CPU and affecting the real-time control of the device.

[0062] Furthermore, when the current occupancy rate is greater than or equal to a second preset threshold (e.g., ≥90%), an emergency mode is triggered. In emergency mode, while performing high-speed compression (using a third compression algorithm), the following two proactive degradation operations are triggered in parallel: First, it performs cold data cleanup tasks, actively scans the cold data storage area, identifies and forcibly deletes several historical compressed archive files with the earliest storage time based on timestamp verification, thereby directly freeing up usable physical sectors in the file system. Second, the retention period for historical data in the hot data area is shortened. For example, the original hot data retention period is dynamically reduced from 3 days to 2 days, and the extra day's data is also extracted and compressed in the current cleanup script.

[0063] This self-regulating mechanism, which requires no human intervention, creates a closed-loop protection system for extreme operating conditions, ensuring that core processes have the minimum necessary storage resources to maintain operation regardless of the platform's isolated operating state.

[0064] Table 1

[0065] In summary, this invention discloses a three-level data storage method for a data acquisition platform. To more clearly demonstrate the overall collaborative logic of the various technical features of this invention in a real complex industrial environment, the following section provides a comprehensive overview of the technical solution of this invention by combining it with a 20-foot energy storage container battery management system containing 13,000 measurement points (the hardware platform is configured with a quad-core Cortex-A55 processor, 2GB of memory, and 8GB of eMMC storage space).

[0066] After the system is started, the front-end sensors continuously generate real-time data from up to 13,000 measurement points.

[0067] This data stream first enters the first-level storage: a memory circular buffer configured with a maximum size of 20MB. Data is written frequently at a frequency of 1 second and resides in memory for 60 seconds. Since no disk I / O is involved, this operation has zero latency and zero wear.

[0068] When the system detects an alarm signal such as a single-unit overvoltage, it immediately extracts data slices from 30 seconds before and after the alarm point to generate a CSV file, and uses the tar tool to package and compress it into an independent fault package and save it to disk for future reference.

[0069] At the same time, this data stream enters the second-level storage: the hot data area. To avoid catastrophic wear caused by high-frequency writes, the system downsamples the data frequency to 5 seconds, performs memory copy serialization to generate BLOB blocks for different measurement points such as voltage and temperature, and accumulates them in memory for 15 seconds (i.e., 3 cycles) before writing them to the SQLite database in batches through a single database transaction.

[0070] SQLite runs in WAL mode, and its shared memory log is redirected to a tmpfs filesystem based on system RAM. This combined strategy not only achieves second-level response times for regular SQL queries on hot data (data from the last 3 days, totaling approximately 2.4GB), but also reduces actual physical write wear on the 8GB eMMC by more than 60%.

[0071] Every morning at dawn, the system automatically triggers a migration pipeline to the third-level storage (compressed archive cold data area). To balance memory safety and compression efficiency, the day's data is divided into four independent processing batches of six hours each. Within each batch, the program reads expired hot data from SQLite sequentially by battery cluster. By establishing a multidimensional mapping table, it completely deconstructs the original row-based BLOB structure arranged by timestamps in memory and reassembles it into a column-based structure file with continuously arranged values ​​from the same measurement point. Entropy reduction processing of the data is completed within the constraint that peak memory usage does not exceed 190MB.

[0072] After the list file is generated, the current health (occupancy) of the 8GB eMMC is dynamically assessed. Under normal conditions (occupancy <85% and BMS is idle), the bzip2 algorithm is called with low priority to perform high compression ratio harvesting; once a space warning is issued (occupancy reaches 85%~90%), it automatically degrades to gzip balanced operation; if an extreme congestion crisis occurs (occupancy >90%), an emergency mode is triggered, and the zstd algorithm is used for rapid repair, while cleaning up the earliest archive and reducing the number of hot data days.

[0073] Through the organic synergy of the aforementioned three-tier architecture and a series of optimizations, the measured 800MB of raw BMS data per day was ultimately compressed to 17MB (a compression ratio as high as 48:1). Even with strict retention of 120 days of complete historical data that meets national standards, the total cold storage capacity is only 2.04GB. Including the 2.4GB of hot data, the total storage requirement is locked within 6GB, not only leaving nearly 26% of redundant safety space within the limited 8GB eMMC, but also ensuring that the overall solution does not cause any degradation to the real-time control performance of the real-time operating system.

[0074] In another preferred embodiment of the present invention, a three-level data storage system for a data acquisition platform is also disclosed. This storage system stores the acquired data based on the three-level data storage method for a data acquisition platform described in the above embodiments.

[0075] This invention also discloses another three-level data storage system for a data acquisition platform, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. The programs include instructions for performing the data storage method as described above. The processor may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, used to execute relevant programs to implement the functions required by the modules in the data storage system of this application embodiment, or to execute the data storage method of the method embodiment of this application.

[0076] The present invention also discloses a computer-readable storage medium comprising a computer program executable by a processor to perform the data storage method described above. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available medium can be read-only memory (ROM), random access memory (RAM), or magnetic media, such as floppy disks, hard disks, magnetic tapes, magnetic disks, or optical media, such as digital versatile discs (DVDs), or semiconductor media, such as solid-state drives (SSDs).

[0077] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned data storage method.

[0078] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A three-level data storage method for a data acquisition platform, characterized in that, include: The collected real-time data is periodically written into the collector's memory circular buffer at intervals of the first duration. When a fault trigger signal is received, data before and after the fault moment is extracted from the memory circular buffer and packaged into a fault data packet. The collected real-time data is periodically written into the hot data area of ​​the local database of the collector at a second time interval, wherein the second time interval is longer than the first time interval; Historical data that has exceeded its retention period in the hot data area is periodically migrated to the cold data storage area. The migration method includes: Read historical data from the local database and convert row-based time-series data into column-based time-series data to obtain a column-structured file; Obtain the current occupancy rate of the storage medium, match the corresponding compression strategy according to the threshold range of the current occupancy rate, and compress and archive the columnar structure file based on the compression strategy.

2. The three-level data storage method for the data acquisition platform according to claim 1, characterized in that, Data within a preset time period before and after the fault occurrence is extracted, converted into an independent file, and compressed to generate the fault data packet. At the same time, alarm records corresponding to the fault trigger signal are inserted into the local alarm table.

3. The three-level data storage method for the data acquisition platform according to claim 1, characterized in that, The method for writing the collected real-time data into the hot data area of ​​the collector's local database includes: The real-time data is serialized according to different measurement point types and converted into binary data; Within a transaction commit cycle consisting of multiple second durations, the binary data is cached, and at the end of the transaction commit cycle, it is committed and written to the hot data area in a batch.

4. The three-level data storage method for the data acquisition platform according to claim 3, characterized in that, The local database is configured to use write-ahead logging. The shared memory files of the local database are redirected to a memory-based temporary file system; and incremental space reclamation operations are performed on the local database after periodically migrating historical data.

5. The three-level data storage method for the data acquisition platform according to claim 1, characterized in that, The method for obtaining the columnar structure file further includes: The historical data in the local database is read in batches according to time period and object; Establish a mapping table with the measurement point identifier as the key and the numerical sequence of the measurement point within the current reading time period as the value; The results read in batches are deserialized, and the measurement values ​​at each time point are appended to the numerical sequence corresponding to the mapping table. The columnar structure file is then output based on the mapping table.

6. The three-level data storage method for the data acquisition platform according to claim 1, characterized in that, The method for matching a corresponding compression strategy based on the threshold range of the current occupancy rate includes: When the current occupancy rate is less than the first preset threshold and the device monitored by the data acquisition platform is in a silent state, a high compression ratio mode is triggered, and the first compression algorithm is used for compression. The silent state refers to the device monitored by the data acquisition platform having no power fluctuation or being in standby mode within a preset time period. When the current occupancy rate is greater than or equal to the first preset threshold and less than the second preset threshold, a balancing mode is triggered, and a second compression algorithm is used for compression; the compression ratio of the second compression algorithm is lower than that of the first compression algorithm, and the compression speed is higher than that of the first compression algorithm; When the current occupancy rate is greater than or equal to the second preset threshold, a fast mode is triggered, and a third compression algorithm is used for compression; the compression speed of the third compression algorithm is higher than that of the second compression algorithm.

7. The three-level data storage method for the data acquisition platform according to claim 6, characterized in that, When the current occupancy rate is greater than or equal to the second preset threshold, an emergency mode is triggered, and the following operations are performed simultaneously: Execute the cold data cleanup task to delete the oldest compressed archive file stored in the cold data storage area; The retention period for historical data in the hot data area will be shortened.

8. A three-level data storage system for a data acquisition platform, characterized in that, The storage system stores the collected data based on the three-level data storage method of the data acquisition platform as described in any one of claims 1 to 7.

9. A three-level data storage system for a data acquisition platform, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for performing the three-level data storage method of the data acquisition platform as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It includes a computer program that can be executed by a processor to perform the three-level data storage method for the data acquisition platform as described in any one of claims 1 to 7.

Images