System and method for persistent storage of total data of Internet of Things based on multi-stage time granularity

Abstract

The invention discloses an Internet of Things full data persistent storage system and method based on multi-stage time granularity. The system comprises a data acquisition module, a multi-stage storage management module, a dynamic index module, a compression engine module and a query engine module. The data acquisition module is used for receiving equipment data in real time, the multi-level storage management module is used for dynamically dividing time windows and executing a cold and hot data migration strategy, the dynamic index module is used for constructing a joint index of time dimensions and equipment IDs, and the compression engine module is used for selecting an optimal compression algorithm according to data features and transmitting the optimal compression algorithm to the storage management module. And the query engine module is used for analyzing a query condition, positioning a data block through an index and optimizing an IO reading sequence. Compared with a traditional time sequence database, the storage cost is reduced through a dynamic time granularity division and hierarchical compression algorithm; and on the basis of query optimization of multi-level indexes, the query efficiency in the complex time range is improved by several times, and the query response time in the complex time range is reduced from the minute level to the sub-second level.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things data storage, and specifically relates to a system and method for persistent storage of full Internet of Things data based on multi-level time granularity. Background Art

[0002] In IoT scenarios, the data generated by devices is often real-time, diverse, and massive. Reliable, scalable, and high-performance storage solutions are needed to preserve this data over the long term for subsequent analysis, querying, and application. IoT data often has multiple levels of time granularity, such as device operational data at the second level, aggregated data at the minute level, and statistical data at the hour level or longer. Storage systems and methods based on this multi-level time granularity can better adapt to different data types and business needs, improving data storage and management efficiency.

[0003] Disadvantages of existing technologies: 1. Low storage efficiency: Traditional time series databases (such as InfluxDB) use a single time partition, resulting in mixed storage of hot and cold data, and query efficiency drops sharply with the growth of data volume. 2. Insufficient compression: Existing compression algorithms do not take into account the redundant characteristics of the time dimension. For example, SQLite stores 500 million data in 500GB, while column storage is only compressed to 21GB, and there is still room for optimization. 3. High query latency: The index structure is not adapted to multi-level time granularity. Queries across time ranges need to traverse the entire table. In industrial scenarios, the query delay for tens of millions of data exceeds 10 seconds. 4. Poor scalability: Traditional architectures cannot dynamically adjust the storage granularity. For example, DolphinDB requires a predefined partitioning strategy and cannot adapt to changes in data access patterns.

[0004] Therefore, the present invention provides a system and method for persistent storage of full-scale data of the Internet of Things based on multi-level time granularity to solve the problems existing in the prior art. Summary of the Invention

[0005] In response to the problems raised by the above background technology, the purpose of the present invention is to provide a persistent storage system and method for the entire Internet of Things data based on multi-level time granularity, reduce storage costs through dynamic time granularity division and hierarchical compression algorithm; optimize queries based on multi-level indexes to improve the efficiency of complex time range queries; and support automatic migration of hot and cold data and elastic storage expansion to adapt to the life cycle characteristics of Internet of Things data.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the present invention are as follows:

[0007] A persistent storage system for all IoT data based on multi-level time granularity, including a data acquisition module, a multi-level storage management module, a dynamic indexing module, a compression engine module, and a query engine module.

[0008] The data acquisition module is used to receive device data in real time, the multi-level storage management module is used to dynamically divide time windows and execute hot and cold data migration strategies, the dynamic index module is used to construct a joint index of time dimension and device ID, the compression engine module is used to select the optimal compression algorithm based on data characteristics, and the query engine module is used to parse query conditions, locate data blocks through indexes, and optimize IO reading order.

[0009] It is further defined that the multi-level storage management module includes a time granularity division unit and a hot and cold migration unit, the time granularity division unit dynamically generates a time window according to the data frequency, the time window includes T1, T2, ..., Tn, where T1 = 1 minute, T2 = 1 hour, and Tn = 1 day;

[0010] The hot-cold migration unit is used to calculate data heat. The formula of the hot-cold migration unit based on access frequency is as follows:

[0011] F(t)=α·F hist +(1-α)·F curr

[0012] Wherein, α is a decay factor, which is used to control the weight of the historical access frequency, wherein 0<α<1,

[0013] F hist is the average number of visits in the past 24 hours, F curr is the number of visits in the current time window.

[0014] It is further defined that the compression engine module adopts a column storage algorithm, and the compression ratio calculation formula is:

[0015]

[0016] in, Indicates the original size of the i-th data block, Indicates the compressed size, and n indicates the total number of data blocks in the current time window.

[0017] It is further defined that the dynamic index module constructs a multi-level B+ tree index and an index level L, and satisfies the following index level formula:

[0018]

[0019] Where k is the B+ tree branching factor, N is the total number of data records, and M is the leaf node capacity.

[0020] A method for persistent storage of full-volume data of the Internet of Things based on multi-level time granularity includes the following steps:

[0021] S1: Receives the original data stream and extracts the timestamp and device ID;

[0022] S2: Dynamically divide the time window; the dynamic time window formula is as follows:

[0023]

[0024] Among them, R threshold represents the data arrival rate threshold, Q threshold Indicates the query frequency threshold.

[0025] Further defined, after S2, hierarchical compression is performed: the data in the time window is sorted by column, and the redundancy is calculated as follows:

[0026]

[0027] When R d > 0.9 enables dictionary encoding, otherwise Delta-of-Delta compression is used.

[0028] Further defined, it also includes a hot and cold data migration strategy, and the heat threshold calculation formula defined in the hot and cold data migration strategy is as follows:

[0029] H t =μ+3σ

[0030] Where μ is the average number of accesses and σ is the standard deviation. The hot and cold data migration strategy is used to migrate over-threshold data to the SSD storage layer.

[0031] Further defined, the query optimization adopts the time predicate push-down technology, and the query response time satisfies the following formula:

[0032]

[0033] Among them, T index Indicates index search time, Indicates the scanning time of the i-th data block.

[0034] An electronic device comprising the system according to any one of claims 1 to 4.

[0035] A computer-readable storage medium storing program code for implementing any one of the methods of claims 5-8.

[0036] Beneficial effects of the present invention:

[0037] Advantages of the present invention:

[0038] 1. Storage efficiency: Compared with traditional time series databases, this invention significantly reduces storage costs through dynamic time granularity division and layered compression algorithms.

[0039] 2. Query performance: Query optimization based on multi-level indexes improves the efficiency of complex time range queries several times, reducing the response time of complex time range queries from minutes to sub-seconds.

[0040] 3. Scalability: Supports automatic migration of hot and cold data and elastic storage expansion, and can also dynamically add storage nodes.

[0041] In summary, the present invention reduces storage costs through dynamic time granularity division and hierarchical compression algorithms; optimizes queries based on multi-level indexes and improves the efficiency of complex time range queries; and supports automatic migration of hot and cold data and elastic storage expansion, adapting to the lifecycle characteristics of IoT data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention can be further illustrated by the non-limiting examples given in the accompanying drawings;

[0043] Figure 1 This is a system module diagram of an embodiment of a system and method for persistent storage of full IoT data based on multi-level time granularity according to the present invention;

[0044] Figure 2 This is a flowchart of the steps of an embodiment of the system and method for persistent storage of full data of the Internet of Things based on multi-level time granularity of the present invention.

[0045] The main component symbols are explained as follows: data acquisition module 101, multi-level storage management module 102, dynamic index module 103, compression engine module 104, and query engine module 105. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the present invention, the technical solutions of the present invention are further described below in conjunction with the accompanying drawings and embodiments. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of the present invention.

[0047] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0048] In addition, the descriptions of "first", "second", etc. in the present invention are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0049] like Figure 1 As shown, the IoT full data persistent storage system based on multi-level time granularity of the present invention includes a data acquisition module 101, a multi-level storage management module 102, a dynamic index module 103, a compression engine module 104 and a query engine module 105;

[0050] The data acquisition module 101 is used to receive device data in real time, and inputs: IoT device raw data stream (MQTT / Kafka protocol), and outputs: structured data with timestamps (transmitted to the multi-level storage management module 102 via HTTP API);

[0051] The multi-level storage management module 102 is used to dynamically divide time windows and execute hot and cold data migration strategies; and inputs: raw data from the data acquisition module 101, and outputs: data blocks partitioned by time granularity (transmitted to the compression engine via an internal message queue);

[0052] The dynamic index module 103 is used to construct a joint index of the time dimension and the device ID. The input is compressed metadata (obtained by monitoring ZooKeeper node changes) and the output is a multi-level B+ tree index file (interacting with the query engine through shared memory).

[0053] The compression engine module 104 is used to select the optimal compression algorithm according to data characteristics, and inputs: data blocks from the multi-level storage management module 102, and outputs: compressed column storage files (written to storage nodes through the distributed file system interface);

[0054] The query engine module 105 is used to parse query conditions, locate data blocks through indexes, optimize IO reading order, and input: user SQL query requests (received through RESTful API), and output: query result sets (returned through gRPC streaming).

[0055] In practical applications of this embodiment, the multi-level storage management module 102 includes a time granularity division unit 1021 and a hot / cold migration unit 1022. The time granularity division unit 1021 dynamically generates time windows according to data frequency. The time windows include T1, T2, ..., Tn, where T1 = 1 minute, T2 = 1 hour, and Tn = 1 day.

[0056] The hot / cold migration unit 1022 is used to calculate data heat. The formula of the hot / cold migration unit 1022 based on the access frequency is as follows:

[0057] F(t)=α·F hist +(1-α)·F curr

[0058] Wherein, α is a decay factor, which is used to control the weight of the historical access frequency, wherein 0<α<1,

[0059] F hist is the average number of visits in the past 24 hours, F curr is the number of visits in the current time window.

[0060] In the actual application of this embodiment, the compression engine module 104 adopts a column storage algorithm, and the compression ratio calculation formula is:

[0061]

[0062] in, Indicates the original size of the i-th data block, Indicates the compressed size, and n indicates the total number of data blocks in the current time window.

[0063] In the actual application of this embodiment, the dynamic index module 103 constructs a multi-level B+ tree index and an index level L, and satisfies the following index level formula:

[0064]

[0065] Where k is the B+ tree branching factor, N is the total number of data records, and M is the leaf node capacity.

[0066] A method for persistent storage of full-volume data of the Internet of Things based on multi-level time granularity includes the following steps:

[0067] S1: Receives the original data stream and extracts the timestamp and device ID;

[0068] S2: Dynamically divide the time window; the dynamic time window formula is as follows:

[0069]

[0070] Among them, R threshold represents the data arrival rate threshold, Q threshold Indicates the query frequency threshold.

[0071] In the actual application of this embodiment, the layered compression is performed after S2: the data in the time window is sorted by column, and the redundancy is calculated as follows:

[0072]

[0073] When R d > 0.9 enables dictionary encoding, otherwise Delta-of-Delta compression is used.

[0074] In practical applications of this embodiment, a hot and cold data migration strategy is also included. The calculation formula for defining the heat threshold of the hot and cold data migration strategy is as follows:

[0075] H t =μ+3σ

[0076] Where μ is the average number of accesses and σ is the standard deviation. The hot and cold data migration strategy is used to migrate over-threshold data to the SSD storage layer.

[0077] In the actual application of this embodiment, the query optimization adopts the time predicate push-down technology, and the query response time satisfies the following formula:

[0078]

[0079] Among them, T index Indicates index search time, Indicates the scanning time of the i-th data block.

[0080] Example 1:

[0081] Implementation 1: Industrial Equipment Monitoring Scenario

[0082] Scenario description:

[0083] A steel plant's 5,000 sensors generate 2,000 pieces of data (temperature, vibration, etc.) per second, and the data needs to be stored for 10 years.

[0084] System Application:

[0085] Dynamic time window adjustment:

[0086]

[0087] If the initial window W prev = 1 minute;

[0088] When the sensor data arrival rate R data=2500 / s>R threshold =2000 / s, W curr =1×2=2 minute window; after adjustment, the write pressure is reduced by 50%.

[0089] Compression strategy:

[0090]

[0091] Temperature data column N total =1,000,000, n distinct =50 (because the equipment accuracy is 0.1°C and the range is fixed);

[0092] Calculation: R d =1-50 / 1,000,000=0.99995, far exceeding the 0.9 threshold;

[0093] Using dictionary encoding, the original 1GB data is compressed to 20MB, and the compression ratio Cr = (1024-20) / 1024×100

[0094] Hot and cold migration:

[0095] F(t)=α·F hist +(1-α)·F curr

[0096] A vibration data block history access F hist =80 times, F curr =300 times, α=0.8;

[0097] F(t)=0.8×80+0.2×300=64+60=124;

[0098] System average μ=90,σ=30, threshold H t =90+3×30=180

[0099] Because 124<180, the data is considered cold and is migrated to HDD storage.

[0100] Example 2: Smart City Traffic Management

[0101] Scenario description:

[0102] City electronic eyes generate 1 billion vehicle trajectory data every day, and traffic flow needs to be analyzed in real time.

[0103] Method application:

[0104] Dynamic time window: The granularity is automatically reduced to 5 minutes during the morning rush hour and expanded to 1 hour at night.

[0105] Dynamic adjustment of time window:

[0106]

[0107] Query frequency Q during morning peak hours freq =150 times / minute>Q threshold =100 times / min; W prev =15 minutes, then adjust W curr =15 / 2=7.5 minutes (rounded to 5 minutes);

[0108] Substituting the formula into the calculation, we can see that the query accuracy is improved by 3 times and the response time is reduced from T resp =120ms down to T resp =25ms+2×8ms=41ms.

[0109] Example 3: Real-time data analysis of the Internet of Vehicles

[0110] Scenario description:

[0111] Autonomous vehicles upload more than 1,000 sensor data points per second, which need to be stored in real time and support millisecond-level queries.

[0112] Technical implementation:

[0113] Edge node processing: Perform 1-second granularity data aggregation at the vehicle gateway and only transmit abnormal events to the cloud.

[0114] Column Compression: Delta encoding is used for GPS coordinates, reducing storage space by 89%.

[0115] Parallel query: Using the DolphinDB distributed computing framework, a range query of 1 billion data points takes less than 1 second.

[0116] Index level calculation:

[0117]

[0118] Vehicle data N = 10 9 , set k = 200, M = 5000;

[0119]

[0120] Therefore, compared with the 5-layer structure of the traditional B+ tree, the query path is shortened by 40%;

[0121] GPS coordinate compression:

[0122]

[0123] Using the Delta-of-Delta algorithm, 90% of the quadratic differences between adjacent coordinates are 0;

[0124] Original data 1MB after compression Compression ratio = (1-0.11) / 1×100.

[0125] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A multi-level time-granularity, full-scale IoT data persistent storage system, featuring: It includes a data acquisition module (101), a multi-level storage management module (102), a dynamic index module (103), a compression engine module (104) and a query engine module (105); The data acquisition module (101) is used to receive device data in real time, the multi-level storage management module (102) is used to dynamically divide time windows and execute hot and cold data migration strategies, the dynamic index module (103) is used to construct a joint index of the time dimension and the device ID, the compression engine module (104) is used to select the optimal compression algorithm according to data characteristics, and the query engine module (105) is used to parse query conditions, locate data blocks through indexes, and optimize IO reading order.

2. The multi-level time granularity-based persistent storage system for all IoT data according to claim 1 is characterized by: The multi-level storage management module (102) includes a time granularity division unit (1021) and a hot / cold migration unit (1022), wherein the time granularity division unit (1021) dynamically generates a time window according to the data frequency, wherein the time window includes T1, T2, ..., Tn, wherein T1 = 1 minute, T2 = 1 hour, and Tn = 1 day; The hot / cold migration unit (1022) is used to calculate data heat. The formula of the hot / cold migration unit (1022) based on access frequency is as follows: F(t)=α·F hist +(1-α)·F curr Wherein, α is a decay factor, which is used to control the weight of the historical access frequency, where 0<α<1, F hist is the average number of visits in the past 24 hours, F curr is the number of visits in the current time window.

3. The multi-level time granularity-based persistent storage system for all IoT data according to claim 1 is characterized by: The compression engine module (104) adopts a column storage algorithm, and the compression ratio calculation formula is: in, Indicates the original size of the i-th data block, Indicates the compressed size, and n indicates the total number of data blocks in the current time window.

4. The multi-level time granularity-based persistent storage system for all IoT data according to claim 1 is characterized by: The dynamic index module (103) constructs a multi-level B+ tree index and an index level L, and satisfies the following index level formula: Where k is the B+ tree branching factor, N is the total number of data records, and M is the leaf node capacity.

5. A method for persistent storage of full IoT data based on multi-level time granularity, characterized in that: The steps include: S1: Receives the original data stream and extracts the timestamp and device ID; S2: Dynamically divide the time window; the dynamic time window formula is as follows: Among them, R threshold represents the data arrival rate threshold, Q threshold Indicates the query frequency threshold.

6. The method for persistent storage of full IoT data based on multi-level time granularity according to claim 5 is characterized by: After S2, layered compression is performed: the data in the time window is sorted by column, and the redundancy is calculated as follows: When R d > 0.9 enables dictionary encoding, otherwise Delta-of-Delta compression is used.

7. The method for persistent storage of full IoT data based on multi-level time granularity according to claim 5 is characterized by: It also includes a hot and cold data migration strategy. The calculation formula for the heat threshold defined in the hot and cold data migration strategy is as follows: H t =μ+3σ Where μ is the average number of accesses and σ is the standard deviation. The hot and cold data migration strategy is used to migrate over-threshold data to the SSD storage layer.

8. The method for persistent storage of full IoT data based on multi-level time granularity according to claim 5 is characterized by: The query optimization adopts the time predicate push-down technology, and the query response time satisfies the following formula: Among them, T index Indicates index search time, Indicates the scanning time of the i-th data block.

9. An electronic device, characterized in that: The electronic device comprises the system according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes for implementing any one of the methods of claims 5-8.

Images