A method and system for dynamically compressing time-series data

Abstract

This invention discloses a method and system for dynamically compressing time-series data. The method includes the following steps: S1, sensors periodically upload collected monitoring data to a server; S2, the server receives and processes the monitoring data from the sensors, and if the monitoring data is abnormal, the server sends an early warning message to the user; S3, the server performs a dynamic deduplication strategy on the monitoring data; S4, if the monitoring data is not deduplicated, the server transmits it to a time-series database for storage; S5, the user reads the deduplicated monitoring data stream from the time-series database through the server; S6, the deduplicated monitoring data stream is restored to the complete original monitoring data stream at the user's end using a time-series data decompression and restoration algorithm. This invention can effectively improve system throughput and data reading speed in IoT monitoring systems, while reducing system bandwidth consumption and storage space consumption, thereby effectively improving the overall performance of the system.

Description

Technical Field

[0001] This invention belongs to the technical field of the Internet of Things, specifically relating to a method and system for dynamically compressing time-series data. Background Technology

[0002] The Internet of Things (IoT) refers to the use of various information sensing devices to collect information in real time about objects or processes that need to be monitored and connected, enabling connections between things and between things and people on a network, thereby facilitating the identification and management of various objects and processes. Currently, IoT technology has fully penetrated our lives and profoundly impacted them, including connected vehicles, industrial IoT, agricultural IoT, smart homes, and smart security. IDC predicted in 2019 that by 2025, there will be 41.6 billion IoT devices in use globally, collectively building a global IoT network. Among various IoT applications, monitoring systems based on IoT technology, such as environmental monitoring systems, fire monitoring systems, and equipment operation status monitoring systems, can replace human labor for uninterrupted monitoring and early warning of monitored objects, thus possessing stronger disaster prevention capabilities and effectively improving industrial production efficiency, reducing industrial production costs, and increasing social operational efficiency.

[0003] Taking an IoT-based gas leak monitoring system as an example, an IoT monitoring system generally consists of four components: a set of gas monitoring sensors laid in various pipelines, a central or distributed server, a time-series database (TSDB) for storing monitoring data, and a group of system users. Each sensor collects leak monitoring information from the gas pipeline at fixed time intervals (e.g., 10 seconds) and uploads it to the server. The server receives and analyzes the data uploaded by these sensors, and if the data shows abnormalities, the server will send an early warning notification to the users. The TSDB receives the processed data from the server and stores it on disk, while system users query the gas leak monitoring time-series data stored in the TSDB through the server.

[0004] In typical IoT monitoring system architectures, the data streams transmitted between different components consume enormous communication bandwidth and server processing power, while uncompressed data occupies significant disk space. A natural solution is to apply traditional general-purpose compression algorithms to reduce the amount of data the system needs to process, transmit, and store. However, this approach is not easily effective in this type of IoT monitoring system because each data point generated periodically by the monitoring sensors often requires real-time transmission. Therefore, each communication between the sensor, server, and time-series database transmits exactly one data point. In this case, traditional compression methods designed for static batch data cannot reduce the size of a single data point, thus failing to reduce the number of communications and bandwidth between system modules, and consequently, failing to effectively improve system throughput and performance. The maximum effect these compression algorithms can achieve is merely compressing the overall data stream size at the database level, reducing disk overhead. Therefore, existing general-purpose compression algorithms are not entirely suitable for this type of IoT monitoring system. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method and system for dynamically compressing time-series data. This method designs a dynamic deduplication strategy for time-series data to achieve dynamic compression of time-series data. This invention can effectively improve system throughput and data reading speed in IoT monitoring systems, while reducing system bandwidth consumption and storage space consumption, thereby greatly improving the overall performance of IoT monitoring systems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for dynamically compressing time-series data, comprising the following steps:

[0008] S1. The sensor periodically uploads the collected monitoring data to the server; the monitoring data is a data tuple containing the data collection timestamp and the monitoring value;

[0009] S2. The server receives and processes monitoring data from the sensors. If the monitoring data is abnormal, the server sends an early warning message to the user.

[0010] S3. The server executes a dynamic deduplication strategy for time-series data on the monitoring data. The dynamic deduplication strategy for time-series data is as follows: whenever a new data point arrives at the server, the server will determine whether there are any data points that can be deduplicated with the data point at the previous time. If there are, these data points will be deduplicated and will not be stored. If there are no, these data points that do not need to be deduplicated will be transmitted to the time-series database for further storage.

[0011] S4. If the monitoring data is not deduplicated, the server will transmit it to the time series database for storage.

[0012] S5. The user reads the deduplicated and compressed monitoring data stream from the time-series database through the server;

[0013] S6. The deduplicated and compressed monitoring data stream is restored to the complete original monitoring data stream at the user end through a time-series data decompression and restoration algorithm.

[0014] As a preferred technical solution, the sensor is a group of information sensing devices with monitoring and communication capabilities; the server is a central or distributed server with the ability to process monitoring time-series data.

[0015] As a preferred technical solution, in step S2, the anomaly is that the current monitored data value exceeds the set warning threshold, the user is the system's management and business user U, and the method of sending the warning information includes, but is not limited to, web page warning, email warning, or SMS warning.

[0016] As a preferred technical solution, in step S3, the dynamic deduplication strategy for time-series data specifically includes:

[0017] S31. Initialize the initial data point f0 = (0, init);

[0018] S32. Determine the currently reached monitoring data point c. i Is it related to the data point f at the previous time? i-1 The values ​​are the same;

[0019] S33. If the result of S32 is yes, then jump to step S37;

[0020] S34. If the result of S32 is negative, then continue to check the data point f at the previous time step. i-1 Is it related to the data point f at the previous time? i-2 The values ​​are the same;

[0021] S35. If the result of S34 is yes, then f i-1 and c i Transmitted to a time-series database for storage;

[0022] S36. If the result of S34 is negative, then only c i Transmitted to a time-series database for storage;

[0023] S37, let f i =c i Whenever there is a new monitoring data point c i+1 Upon arrival, repeat steps S32-S37.

[0024] As a preferred technical solution, in step S4, the time series database is a local or cloud-based time series database service that provides efficient storage, retrieval, and management capabilities for time series data.

[0025] As a preferred technical solution, in step S5, the deduplicated and compressed monitoring data stream is the time-series data stream Q that has been processed in steps S3 and S4 and is actually stored in the time-series database.

[0026] As a preferred technical solution, step S6, the time-series data decompression and restoration algorithm includes the following steps:

[0027] S61. Initialize interval to the time interval for sensor S to periodically collect monitoring data, and let the time-series data stream Q′=Q;

[0028] S62. Traverse each data point q in the time-series data stream Q′. i ,like Then the value will be changed every interval. Fill to data stream interval

[0029] S63. Return the data stream Q′ after filling and restoring.

[0030] As a preferred technical solution, after the time-series data decompression and restoration algorithm, the data points that were discarded after deduplication in the deduplicated and compressed monitoring data stream Q will be refilled, thereby restoring a complete original monitoring data stream Q′.

[0031] As a preferred technical solution, in step S6, the user terminal is the user's device, including but not limited to smartphones and web browsers.

[0032] Secondly, the present invention provides an Internet of Things (IoT) monitoring system. This system uses the aforementioned method of dynamically compressing time-series data to process the monitoring data. After processing, in the data storage process from the server to the time-series database TSDB, the data reading process from TSDB to the server, and the data query process from the server to the user, the data transmitted is transformed from the original monitoring data stream Q′ to the deduplicated and compressed monitoring data stream Q. Since |Q′|>|Q|, the amount of data transmitted in these processes will be effectively reduced.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] In this invention, the server executes a dynamic deduplication strategy for time-series data monitoring. This strategy works as follows: whenever a new data point arrives at the server, the server checks if there are any deduplicated data points between this data point and the previous data point. If so, these deduplicated data points are removed and not stored. If not, these data points that do not need deduplication are transmitted to the time-series database for further storage. Based on this dynamic deduplication strategy, this invention enables dynamic compression of time-series data streams in IoT monitoring systems, improving system throughput and data reading speed while reducing system bandwidth and storage space consumption. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a diagram of the improved IoT monitoring system architecture using the method of this invention.

[0037] Figure 2 This is a flowchart of the time-series data acquisition, transmission, and storage process for an IoT monitoring system using the original method.

[0038] Figure 3 This is a flowchart of the time-series data acquisition, transmission, and storage process of the IoT monitoring system improved using the method of this invention.

[0039] Figure 4 The results show a comparison of the system throughput of the original method and the method of this invention.

[0040] Figure 5 This is a comparison of the data reading speed between the original method and the method of this invention.

[0041] Figure 6 The results show a comparison of system bandwidth consumption between the original method and the method of this invention.

[0042] Figure 7 The results show a comparison of storage space consumption between the original method and the method of this invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0044] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0045] An IoT-based gas leak monitoring system typically comprises four components: a set of gas monitoring sensors installed in various pipelines, a central or distributed server, a time-series database (TSDB) for storing monitoring data, and a group of system users. Each sensor collects leak monitoring information from the gas pipeline at fixed time intervals (e.g., 10 seconds) and uploads it to the server. The server receives and analyzes the data uploaded by these sensors; if the data shows anomalies, the server sends an alert to the users. The TSDB receives the processed data from the server and stores it on disk. System users can query the gas leak monitoring time-series data stored in the TSDB through the server. This invention improves upon this system, specifically by using a method for dynamically compressing time-series data:

[0046] S1. A set of gas monitoring sensors P laid in various pipelines periodically (e.g., every 10 seconds) uploads the collected monitoring data (t,v) to the server S, where the monitoring data (t,v) includes the data collection timestamp t and the monitoring value v.

[0047] Specifically, if the gas monitoring sensor detects that the pipeline is normal, the monitoring value v = 0; if a gas leak is detected in the pipeline, the monitoring value v represents the specific gas leak concentration, such as v = 1.

[0048] S2. Server S receives and processes monitoring data (t, v) uploaded periodically from various sensors P. If the monitoring data is abnormal, the server sends early warning information to the user, including web page warnings, email warnings, SMS warnings, etc.

[0049] Specifically, abnormal monitoring data means that the monitoring value v > 0, which means that a gas leak has been detected in the pipeline.

[0050] S3. The server S executes a dynamic deduplication strategy for time-series data for each arriving monitoring data point (t,v).

[0051] Specifically, the aforementioned dynamic deduplication strategy for time-series data includes the following steps:

[0052] S31. Initialize the initial data point f0 = (0, init), where init represents the initial value, such as -1;

[0053] S32. Determine the currently reached monitoring data point c. i Is it related to the data point f at the previous time? i-1 The values ​​are the same;

[0054] S33. If the result of S32 is yes, then jump to step S37;

[0055] S34. If the result of S32 is negative, then continue to check the data point f at the previous time step. i-1 Is it related to the data point f at the previous time? i-2 The values ​​are the same;

[0056] S35. If the result of S34 is yes, then f i-1 and c i Transmitted to a time-series database for storage;

[0057] S36. If the result of S34 is negative, then only c i Transmitted to a time-series database for storage;

[0058] S37, let f i =c i Whenever there is a new monitoring data point c i+1 Upon arrival, repeat steps S32-S37;

[0059] More specifically, the strategy means that whenever a new data point is reached... When the data point arrives at server S, server S will compare it with the data point f from the previous time step. i-1 If there are duplicate data points, they will be deduplicated and not stored; otherwise, these data points that do not need to be deduplicated will be transferred to TSDB for further storage.

[0060] In one embodiment, for example, there is a prior data stream. Upon reaching the server, after passing through this policy, TSDB will store... And in the server When new data points Upon arrival, after dynamic deduplication using this strategy, and It will be transferred to TSDB storage. At this point, TSDB will store it. And in the server Through this strategy, server S can achieve dynamic compression of time-series data. That is, as data points arrive continuously, the compression process can proceed dynamically without requiring the accumulation of a batch of static data points as traditional compression algorithms do. Specifically, the time-series database is a local or cloud-based time-series database service that provides efficient storage, retrieval, and management capabilities for time-series data, including OpenTSDB, Alibaba Cloud TSDB, etc.

[0061] S4. If the monitoring data is not deduplicated, the server will transmit it to the time series database for storage; the time series database is a local or cloud-based time series database service that provides efficient storage, retrieval and management capabilities for time series data.

[0062] S5. The user reads the deduplicated and compressed monitoring data stream from the time series database through the server; the deduplicated and compressed monitoring data stream is the time series data stream Q that is actually stored in the time series database after being processed in steps S3 and S4.

[0063] S6. When reading the monitoring data stream, the system's management or business user U reads the deduplicated and compressed monitoring data stream Q from the time-series database through the server, and restores it to the complete original monitoring data stream Q′ at the user end through the time-series data decompression and restoration algorithm.

[0064] Specifically, the user terminal is the user's device, including smartphones, web browsers, etc.; the time-series data decompression and restoration algorithm includes the following steps:

[0065] S61. Initialize interval to the time interval for sensor S to collect monitoring data, such as 10 seconds, and let the time-series data stream Q′=Q;

[0066] S62. Traverse each data point q in the time-series data stream Q′. i ,like Then the value will be changed every interval. Fill to data stream interval

[0067] S63. Return the data stream Q′ after filling and restoration;

[0068] Specifically, after the time-series data decompression and restoration algorithm, the data points that were discarded during deduplication in the decompressed monitoring data stream Q will be refilled, thus restoring a complete original monitoring data stream Q′. For example Will be restored to

[0069] To clearly demonstrate the overall process of this embodiment, Figure 1 The diagram illustrates the improved IoT monitoring system architecture using the method of this invention. Specifically, in this embodiment, after adopting the method of this invention, the data transmitted is transformed from the original monitoring data stream Q′ to the data storage process from the server to the TSDB, the data reading process from the TSDB to the server, and the data query process from the server to the user. Since |Q′| > |Q|, the data transmission volume in these processes is effectively reduced. Furthermore, due to the reduction in data transmission volume, the system's corresponding processing throughput, data reading speed, communication bandwidth consumption, and storage space consumption are all effectively optimized.

[0070] To more clearly demonstrate the differences between this embodiment and other existing technologies, Figure 2 This diagram illustrates the time-series data acquisition, transmission, and storage process of a traditional IoT monitoring system that does not employ the method described in this invention. Figure 3 A flowchart illustrating the time-series data acquisition, transmission, and storage process of the IoT monitoring system improved using the method of this invention is presented. Specifically, in this embodiment, after adopting the method of this invention, each monitoring data point arriving at the server will be determined in real time whether to be deduplicated based on the dynamic deduplication strategy of the time-series data. If deduplicated, the data point will be discarded in the data stream; if not deduplicated, the data point will be normally transmitted to the TSDB for storage. Furthermore, in the TSDB service, a further general static compression method can be implemented to further compress the monitoring data, thereby achieving lower disk storage space consumption.

[0071] To verify the effectiveness of the present invention, in one embodiment, the original method is compared with the method of the present invention in the following three different server settings:

[0072] (1) Synchronous single-threaded mode: The server can only process a single monitoring data point at any given moment. Each monitoring data point will be synchronously transmitted to TSDB for disk storage. At this time, the system performance bottleneck is the system communication bandwidth.

[0073] (2) Asynchronous single-threaded mode: The server can only process a single monitoring data point at any given moment. Each monitoring data point will be asynchronously transmitted to TSDB for disk storage. At this time, the system's performance bottleneck is the performance of a single-core CPU.

[0074] (3) Asynchronous multi-threaded mode: At any given moment, the server can start multiple threads to process multiple monitoring data points simultaneously. Each monitoring data point will be asynchronously transmitted to TSDB for disk storage. At this time, the performance bottleneck of the system is the maximum write speed of TSDB.

[0075] Figures 4-7 The performance comparison results of the original method and the method of this invention are presented in three different server settings, including system throughput, data read speed, system bandwidth consumption, and storage space consumption. Figures 4-7 It can be seen that the method of the present invention has achieved effective system improvement compared with the original method under various server settings.

[0076] In summary, the method for dynamically compressing time-series data in an IoT monitoring system proposed in this invention can effectively solve the problem of dynamic compression of time-series data in IoT monitoring systems. It can effectively improve system throughput and data reading speed in IoT monitoring systems, while reducing system bandwidth consumption and storage space consumption, thereby greatly improving the overall performance of IoT monitoring systems.

[0077] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0078] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0079] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method of dynamically compressing time series data, characterized by, Includes the following steps: S1. The sensor periodically uploads the collected monitoring data to the server; the monitoring data is a data tuple containing the data collection timestamp and the monitoring value; S2. The server receives and processes monitoring data from the sensors. If the monitoring data is abnormal, the server sends an early warning message to the user. S3. The server executes a dynamic deduplication strategy for time-series data on the monitoring data. The dynamic deduplication strategy for time-series data is as follows: whenever a new data point arrives at the server, the server will determine whether there are any data points that can be deduplicated with the data point at the previous time. If there are, these data points will be deduplicated and will not be stored. If there are no, these data points that do not need to be deduplicated will be transmitted to the time-series database for further storage. S4. If the monitoring data is not deduplicated, the server will transmit it to the time series database for storage. S5. The user reads the deduplicated and compressed monitoring data stream from the time-series database through the server; S6. The deduplicated and compressed monitoring data stream is restored to the complete original monitoring data stream at the user end using a time-series data decompression and restoration algorithm. In step S3, the dynamic deduplication strategy for time-series data specifically includes: S31, initializing initial data points ; S32, judging whether the current arrived monitoring data point is same as the value of the previous time data point ; S33. If the result of S32 is yes, then jump to step S37; S34. If the result of S32 is negative, then continue to check the data point at the previous time step. Is it consistent with previous time points? The values ​​are the same; S35. If the result of judgment in S34 is yes, then... and Transmitted to a time-series database for storage; S36. If the result of S34 is negative, then only... Transmitted to a time-series database for storage; S37, Order Whenever there are new monitoring data points Upon arrival, repeat steps S32-S37.

2. The method for dynamically compressing time-series data according to claim 1, characterized in that, The sensor is a group of information sensing devices with monitoring and communication capabilities; the server is a central or distributed server with the ability to process monitoring time-series data.

3. The method for dynamically compressing time-series data according to claim 1, characterized in that, In step S2, the anomaly is that the current monitored data value exceeds the set warning threshold, and the user refers to the system's management and business users. The methods for sending early warning information include, but are not limited to, web page warnings, email warnings, or SMS warnings.

4. The method for dynamically compressing time-series data according to claim 1, characterized in that, In step S4, the time series database is a local or cloud-based time series database service that provides efficient storage, retrieval, and management capabilities for time series data.

5. The method for dynamically compressing time-series data according to claim 1, characterized in that, In step S5, the deduplicated and compressed monitoring data stream is the time-series data stream that has been processed in steps S3 and S4 and is actually stored in the time-series database. .

6. The method for dynamically compressing time-series data according to claim 1, characterized in that, In step S6, the time-series data decompression and restoration algorithm includes the following steps: S61, Initialization For sensors The time interval for collecting monitoring data at regular intervals allows for the creation of time-series data streams. ; S62, Traversing the Time-Sequenced Data Stream Each data point in ,like Then every interval Time will be numerical Fill to data stream interval ; S63, Return to the data stream after filling and restoring. .

7. The method for dynamically compressing time-series data according to claim 6, characterized in that, The monitoring data stream after being decompressed and restored using the aforementioned time-series data decompression and restoration algorithm is now deduplicated and compressed. The data points that were discarded after deduplication will be refilled, thus restoring a complete original monitoring data stream. .

8. The method for dynamically compressing time-series data according to claim 1, characterized in that, In step S6, the user terminal refers to the user's device, including but not limited to smartphones and web browsers.

9. An Internet of Things (IoT) monitoring system, characterized in that, This system processes monitoring data using the dynamic compression time-series data method described in any one of claims 1-8. After processing, the data is transmitted from the server to the time-series database (TSDB), from the TSDB to the server, and from the server to the user during the data storage process, the data reading process, and the data query process. The data transmitted is derived from the original monitoring data stream. Transformed into a deduplicated and compressed monitoring data stream ,because Therefore, the amount of data transmitted in these processes will be effectively reduced.

Images