Internet of Things time series data processing method and system

Abstract

The invention is suitable for the field of Internet of Things, and provides an Internet of Things time series data processing method and system, and the method comprises the steps: receiving a data package sent by Internet of Things equipment, and writing the data of the data package into a data buffer area; obtaining sliding window data, and analyzing and generating a window feature vector according to the sliding window data and a preset window statistical analysis algorithm; determining a target compression mode according to the window feature vector and a preset regularized decision tree model; performing data compression coding on the sliding window data based on the target compression mode through the write-in thread, and performing hierarchical storage on the compressed and coded data through the write-in thread; and sending the compressed and coded data to a cloud according to a preset uploading mechanism. According to the method, data changes are effectively handled, and loss of important data is reduced. The uploading time and frequency are flexibly adjusted according to the network condition and the system load, and the system performance is optimized.

Description

Technical Field

[0001] This application belongs to the field of Internet of Things (IoT), and in particular relates to an IoT time-series data processing method, apparatus, and device. Background Technology

[0002] With the rapid development of Internet of Things (IoT) devices, various sensors, measuring instruments, and controllers are constantly generating massive amounts of time-series data at millisecond-level frequencies. In fields such as industry, transportation, and energy management, the real-time processing and analysis of this data is crucial for achieving intelligent decision-making and efficient operation. For example, in microgrid control, the ability to monitor and schedule power flow in real time directly affects the stability and efficiency of the system; in industrial automation, real-time monitoring of equipment status and performance can reduce failure rates and improve production efficiency.

[0003] However, existing time-series databases still face many challenges in high-frequency data acquisition and processing. First, traditional time-series databases have limited write throughput, making it difficult to meet the write requirements of millions of data points per second. This performance bottleneck prevents the system from responding promptly in high-frequency data acquisition scenarios, thus affecting the effectiveness of real-time decision-making.

[0004] Secondly, existing data compression strategies are often fixed and lack adaptability, failing to dynamically adjust the compression algorithm according to data characteristics. This makes it difficult to achieve a good balance between compression efficiency and system overhead. In the case of high-frequency data streams, how to improve storage efficiency while ensuring data integrity becomes a pressing issue.

[0005] Furthermore, in weak network environments, the order and reliability of data are difficult to guarantee. Due to network instability, data packets may be transmitted out of order, affecting data accuracy and consistency. This problem is particularly prominent in applications with extremely high data reliability requirements, such as industrial control and intelligent traffic scheduling.

[0006] Finally, traditional time-series databases are mostly used for monitoring and recording, making it difficult to support real-time control loops at the edge. For applications requiring sub-millisecond response times, such as microgrid control and industrial automation, existing technologies cannot meet the demands for real-time response, preventing the system from effectively combining intelligent decision-making and control.

[0007] In summary, existing technologies have significant shortcomings in core requirements such as high-frequency acquisition adaptation, adaptive compression, weak network order preservation, and real-time control. There is an urgent need for a new time-series data processing solution to operate efficiently in an edge-cloud collaborative architecture and meet the needs of complex application scenarios. Summary of the Invention

[0008] This application provides an IoT time-series data processing method and system that can solve the above-mentioned problems.

[0009] In a first aspect, embodiments of this application provide an IoT time-series data processing method, applied to a time-series database deployed on an edge node, comprising the following steps: Receive data packets sent by IoT devices and write the data of the data packets into a data buffer; Obtain sliding window data, and analyze and generate window feature vectors based on the sliding window data and a preset window statistical analysis algorithm; wherein, the sliding window data is obtained from the data buffer based on the sliding window, and the window feature vector is the quantization result of the data features of the sliding window data; The target compression mode is determined based on the window feature vector and the preset rule-based decision tree model; The writing thread compresses and encodes the sliding window data based on the target compression mode, and then stores the compressed and encoded data in a hierarchical manner. The hierarchical storage area includes a memory-level pre-cache, and the control thread accesses the mapping area of ​​the memory-level pre-cache to read data. The compressed and encoded data is sent to the cloud according to the preset upload mechanism.

[0010] Secondly, embodiments of this application provide an Internet of Things (IoT) time-series data processing apparatus, including: The first processing unit is used to receive data packets sent by IoT devices and write the data packets into a data buffer. The second processing unit is used to acquire sliding window data and, based on the sliding window data and a preset window statistical analysis algorithm, analyze and generate a window feature vector; wherein, the sliding window data is obtained from the data buffer based on the sliding window, and the window feature vector is the quantization result of the data features of the sliding window data; The third processing unit is used to determine the target compression mode based on the window feature vector and the preset rule-based decision tree model. The fourth processing unit is used to compress and encode the sliding window data based on the target compression mode through a write thread, and to store the compressed and encoded data in a hierarchical manner through the write thread; wherein, the hierarchical storage area includes a memory-level pre-cache, and the control thread accesses the mapping area of ​​the memory-level pre-cache to read data; The fifth processing unit is used to send the compressed and encoded data to the cloud according to a preset upload mechanism.

[0011] Thirdly, embodiments of this application provide an Internet of Things (IoT) time-series data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the first aspect above.

[0012] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0013] Fifthly, embodiments of this application provide an Internet of Things (IoT) time-series data processing system, including: IoT devices, edge nodes, and a cloud, wherein a time-series database is deployed on the edge nodes; The IoT device establishes a data connection with the edge node, and the edge node establishes a data connection with the cloud; The edge node is used to perform the steps described in the first aspect above.

[0014] In this embodiment, data compression and hierarchical storage significantly reduce storage space requirements and improve data transmission efficiency, while lowering bandwidth consumption. The design of a memory-level pre-caching layer and control thread ensures real-time data processing, meeting the stringent latency requirements of IoT applications. A sliding window mechanism dynamically manages data features, and a decision tree model optimizes the compression strategy, effectively addressing data changes and reducing the loss of critical data. Upload time and frequency are flexibly adjusted based on network conditions and system load to optimize system performance. This method exhibits good scalability, adapting to IoT applications of different scales and types. As the number of devices increases, the system can handle higher data traffic by adding edge nodes and expanding storage capacity. Furthermore, the compression algorithm and decision tree model can be retrained and optimized based on new datasets to adapt to constantly changing sensor data characteristics. This method is not only applicable to various IoT application scenarios but also helps improve system real-time performance, reduce the risk of data loss, and possesses good scalability, providing strong support for future smart cities, smart agriculture, and industrial automation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1This is a schematic flowchart of an IoT time-series data processing method provided in the first embodiment of this application; Figure 2 This is a schematic flowchart of S1021~S1022 in an IoT time-series data processing method provided in the first embodiment of this application; Figure 3 This is a schematic flowchart of S1031~S1034 in an IoT time-series data processing method provided in the first embodiment of this application; Figure 4 This is a schematic flowchart of steps S1035 to S10312 in an IoT time-series data processing method provided in the first embodiment of this application; Figure 5 This is a schematic flowchart of steps S106-108 in an IoT time-series data processing method provided in the first embodiment of this application; Figure 6 This is a schematic flowchart of S109~S110 in an IoT time-series data processing method provided in the first embodiment of this application; Figure 7 This is a schematic flowchart of steps S111 to S113 in an IoT time-series data processing method provided in the first embodiment of this application; Figure 8 This is a schematic diagram of an Internet of Things (IoT) time-series data processing system provided in the second embodiment of this application; Figure 9 This is a schematic diagram of the Internet of Things time-series data processing device provided in the third embodiment of this application; Figure 10 This is a schematic diagram of the Internet of Things time-series data processing device provided in the fourth embodiment of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0018] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0019] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0020] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0021] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating an IoT time-series data processing method provided in the first embodiment of this application. In this embodiment, the executing entity of the IoT time-series data processing method is a device with IoT time-series data processing capabilities, such as a desktop computer, server, etc. The IoT time-series data processing method provided in this embodiment is applied to a time-series database deployed on an edge node, such as... Figure 1 The IoT time-series data processing method shown may include: S101: Receive data packets sent by IoT devices and write the data of the data packets into the data buffer.

[0024] In this embodiment, the data packet is a unit of data sent by the IoT device, containing timestamps and sensor data. A data buffer is used to temporarily store the received data to reduce data loss and improve processing efficiency.

[0025] Edge nodes listen for data packets from IoT devices via network interfaces. These data packets may contain sensor reads, status information, or control commands.

[0026] A buffer is pre-defined in memory to temporarily store received data packets. This buffer can be in the form of a circular buffer to make efficient use of memory.

[0027] S102: Obtain sliding window data, and analyze and generate a window feature vector based on the sliding window data and a preset window statistical analysis algorithm; wherein, the sliding window data is obtained from the data buffer based on the sliding window, and the window feature vector is the quantization result of the data features of the sliding window data.

[0028] In this embodiment, the sliding window data is a set of data obtained from the data buffer based on the sliding window within a specific time range.

[0029] The window feature vector is the quantized result of the data features of the sliding window data. It is a quantized array used to drive compression decisions, generated through real-time statistical analysis of the time series within a data window. Its main dimensions include: variance of timestamp differences (measuring sampling periodicity), variance and extreme values ​​of numerical differences (measuring the magnitude of jumps), degree of jump, and fitting residuals or volatility (evaluating data trend and stationarity). This vector serves as input to the decision tree, enabling a dynamic mapping from data features to the optimal compression mode.

[0030] Data is extracted from the data buffer according to a set time window (e.g., data within the last 10 seconds). The window slides forward whenever new data arrives.

[0031] The sliding window data is analyzed using a preset algorithm (such as mean, standard deviation, maximum value, minimum value, etc.) to generate window feature vectors.

[0032] In one embodiment, generating a window feature vector based on the sliding window data and a preset window statistical analysis algorithm may include steps S1021~S1022, such as... Figure 2 As shown, S1021~S1022 are as follows: S1021: Based on the sliding window data and the preset window statistical analysis algorithm, at least the variance of the timestamp difference value, the variance of the data load difference value, and the degree of jump are analyzed and obtained; wherein, the timestamp difference value is the difference between the collection timestamps of two adjacent data points, the data load difference value is the difference between the data values ​​of two adjacent data points, and the degree of jump refers to the fluctuation range between the maximum difference value and the minimum difference value of two adjacent data points.

[0033] Select a sliding window that contains the N most recent data points (e.g., N=10). Each data point contains at least two pieces of information: a timestamp and a data payload value.

[0034] For two adjacent data points within the sliding window, calculate their timestamp difference. Timestamp difference = Current data point timestamp - Previous data point timestamp.

[0035] For two adjacent data points within the sliding window, calculate their data load difference value.

[0036] Data load differential = Current data point data load - Previous data point data load.

[0037] The degree of jump measures the fluctuation range between adjacent data points, specifically: The degree of jump = maximum difference - minimum difference The maximum and minimum difference values ​​are the maximum and minimum values ​​of the timestamp difference and the data load difference, respectively.

[0038] In addition, the degree of hopping can be calculated by first determining the difference between adjacent data points, then calculating the largest positive and negative differences, and the variance between the positive and negative differences is the degree of hopping. The degree of hopping can be compared with the hopping threshold of this type of data, and the frequency of exceeding the threshold can be used to obtain the hopping rate.

[0039] Assume the data points within the sliding window are as follows: Data point 1: Timestamp = 10s, Data payload = 100 Data point 2: Timestamp = 12s, Data payload = 105 Data point 3: Timestamp = 15s, Data payload = 102 calculate: Timestamp difference: 12s - 10s = 2s, 15s - 12s = 3s Data load differential values: 10⁵ - 10⁰ = 5, 10² - 10⁵ = -3 Jump level: Maximum timestamp difference value = 3s, minimum value = 2s; Maximum data load difference value = 5, minimum value = -3.

[0040] S1022: Generate the window feature vector based on the variance of the timestamp difference value, the variance of the data load difference value, and the degree of jump.

[0041] Calculate the variance of the timestamp difference and the variance of the data load difference. Combine the calculated variances of the timestamp difference, the data load difference, and the hopping degree into a feature vector.

[0042] Window feature vector = [timestamp variance, data load variance, jump level] In this embodiment, the feature vector effectively summarizes the original data in the sliding window, reducing the data volume without losing important information. By monitoring changes in the feature vector, anomalies in the data stream can be detected in a timely manner (e.g., excessively high variance may indicate data instability). Using the generated feature vector, combined with a machine learning model, predictions can be made for future time-series data, improving the intelligence level of the IoT system. Real-time data analysis is achieved, and system performance is monitored through feature vectors, allowing for timely adjustments to strategies to optimize resource utilization and system response.

[0043] S103: Determine the target compression mode based on the window feature vector and the preset rule-based decision tree model.

[0044] In this embodiment, the rule-based decision tree model is a model trained on historical data and is used to make decisions based on feature vectors.

[0045] The window feature vectors are classified using a pre-trained decision tree model to determine the appropriate compression mode (such as lossless compression or lossy compression).

[0046] Based on the model output, select the optimal compression strategy to ensure a balance between data compression effectiveness and data integrity.

[0047] If the decision tree model identifies that the current data features indicate that the data changes steadily, it may choose lossless compression; if the changes are drastic, it will choose lossy compression to reduce storage space.

[0048] In one embodiment, based on S1021-S1022, S103 may include S1031-S1034, such as... Figure 3 As shown, S1031~S1034 are as follows: S1031: When the variance of the timestamp difference value is less than the dynamic first threshold, the target compression mode is determined to be the RLE continuous time difference coding mode; wherein, the dynamic first threshold is dynamically configured based on the timestamp stability in the IoT application scenario.

[0049] In this embodiment, the timestamp difference value refers to the difference between the collection timestamps of two adjacent data points (Δt = t_n - t_{n-1}). It is used to analyze the stability and regularity of the sampling period. Stable difference values ​​with small variance are suitable for efficient compression modes such as RLE.

[0050] First, timestamp information is collected from time-series data, and the differences between these timestamps (i.e., the differences between adjacent timestamps) are calculated. Then, the variance of these differences is calculated.

[0051] The calculated variance is compared with a dynamic first threshold. If the variance is less than the dynamic first threshold, the timestamp variation is considered small and highly regular. Generally, the dynamic first threshold can be set to 50. This dynamic first threshold is dynamically configured based on the timestamp stability in the IoT application scenario.

[0052] Under the above conditions, the target compression mode is determined to be RLE (Run-Length Encoding) continuous time difference encoding mode.

[0053] S1032: When the variance of the data load differential value is concentrated in a preset low bit range, the target compression mode is determined to be either packed encoding mode or simple8b encoding mode.

[0054] In this embodiment, the numerical load difference value refers to the difference (Δv = v_n - v_{n-1}) between the data values ​​(load) of two adjacent data points. It is used to analyze the fluctuation and jump characteristics of the data itself. When these differences are concentrated in a small range (such as the low bit range of 0-31), it indicates that the data changes smoothly and is suitable for compression encoding such as Packed / Simple8b.

[0055] Collect data loads (such as sensor readings) from time-series data and calculate the variance of their differences.

[0056] The calculated variance is compared with a preset low-bit range to determine whether the changes in data load are concentrated in the low-bit range.

[0057] When the variance is concentrated in the low bit range, the target compression mode is determined to be either packed encoding mode or simple8b encoding mode.

[0058] S1033: When the variance of the timestamp difference value is less than the dynamic second threshold and the variance of the data load difference value is less than the dynamic third threshold, the target compression mode is determined to be an improved XOR differential coding mode; wherein, the dynamic second threshold is dynamically configured based on the timestamp volatility in the IoT application scenario, and the dynamic third threshold is dynamically configured based on the data load stability in the IoT application scenario.

[0059] Calculate the variance of the timestamp difference and the data load difference separately. If the variance of the timestamp difference is less than a dynamic second threshold and the variance of the data load difference is less than a dynamic third threshold, the data can be considered relatively stable and exhibiting continuity. The dynamic second threshold is dynamically configured based on the timestamp volatility in the IoT application scenario, and the dynamic third threshold is dynamically configured based on the data load stability in the IoT application scenario.

[0060] Based on the above conditions, the target compression mode is determined to be the improved XOR differential coding mode, which stores only the effective bit range of the numerical value, thereby achieving a balance between high compression ratio and low computational overhead.

[0061] S1034: When the degree of jump exceeds the dynamic fourth threshold, the target compression mode is determined to be Raw mode.

[0062] The system monitors data jumps and calculates the degree of these jumps (i.e., the frequency of data value changes). If the degree of jump exceeds a dynamic fourth threshold, it indicates that the data is changing drastically and is not suitable for compression. This dynamic fourth threshold is dynamically configured based on the degree of data jumps in the IoT application scenario.

[0063] If the above conditions are met, the target compression mode is determined to be Raw mode, that is, no compression is performed.

[0064] By following the steps described above and combining different compression modes, appropriate compression strategies can be selected for time-series data with different characteristics, thereby achieving efficient data storage and transmission. Implementing this method can significantly reduce storage costs and improve data processing efficiency.

[0065] In one embodiment, based on S1021-S1022, S103 may include S1035-S10312, such as... Figure 4 As shown, S1035~S10312 are as follows: S1035: Determine whether the degree of the jump exceeds the dynamic fourth threshold.

[0066] First, we define "jump level" as the frequency of change of data payload or timestamps over a period of time. This can be obtained by analyzing the fluctuation range and frequency of change in statistical data. For example, if data changes frequently within a short period, the jump level is high.

[0067] Set a dynamic fourth threshold (e.g., 0.5), and then compare the calculated degree of change with this dynamic fourth threshold.

[0068] If the degree of abrupt change exceeds this fourth dynamic threshold, it indicates that the data is changing drastically. If the degree of abrupt change does not exceed this threshold, it indicates that the data is changing steadily.

[0069] Suppose that in the past hour there were 10 data points [100, 102, 101, 150, 200, 200, 205, 202, 205, 300], and the calculated jump level was 0.8 (e.g., standard deviation), while the set fourth threshold was 0.5, so the jump level exceeded the threshold.

[0070] S1036: If the degree of jump exceeds the dynamic fourth threshold, the target compression mode is determined to be Raw mode.

[0071] S1037: If the degree of jump does not exceed the dynamic fourth threshold, determine whether the variance of the timestamp difference value is less than the dynamic first threshold.

[0072] If the jump level exceeds the dynamic fourth threshold, the target compression mode is determined to be Raw mode, meaning no compression is performed and the original data is stored directly. If the jump level does not exceed the fourth threshold, proceed to the next step of the judgment.

[0073] S1038: If the variance of the timestamp difference value is less than the dynamic first threshold, the target compression mode is determined to be RLE continuous time difference coding mode.

[0074] S1039: If the variance of the timestamp difference value is not less than the dynamic first threshold, determine whether the variance of the data load difference value is concentrated in the preset low bit range.

[0075] Calculate the difference between timestamps (the difference between adjacent timestamps), and calculate the variance of these differences.

[0076] Set a dynamic first threshold (e.g., 0.1) and compare the calculated variance to that threshold.

[0077] If the variance is less than the dynamic first threshold, it indicates that the timestamp changes are highly regular and RLE encoding is suitable. If the variance is not less than the dynamic first threshold, continue to judge the variance of the data load differential value.

[0078] S10310: If the variance of the data load differential value is concentrated in a preset low bit range, determine that the target compression mode is either packed encoding mode or simple8b encoding mode.

[0079] S10311: If the variance of the data load difference value is not concentrated in the preset low bit range, determine whether the variance of the timestamp difference value is less than the dynamic second threshold and whether the variance of the data load difference value is less than the dynamic third threshold.

[0080] Similar to timestamps, calculate the difference values ​​of the data load and the variance of these difference values.

[0081] Set a preset low bit range (e.g., 0 to 1) and determine whether the variance of the data load differential values ​​is concentrated within this range.

[0082] If the focus is on the low bit range, determine the target compression mode as either packed encoding or simple8b encoding. If the focus is not on the low bit range, continue to determine the relationship between timestamp variance and data payload variance.

[0083] S10312: If the variance of the timestamp difference value is less than the dynamic second threshold and the variance of the data payload difference value is less than the dynamic third threshold, the target compression mode is determined to be an improved XOR differential coding mode. If neither condition is met, the target compression mode is determined to be Raw mode.

[0084] Set two preset thresholds: a dynamic second threshold (for the variance of timestamp differences) and a dynamic third threshold (for the variance of data load differences). For example, set the dynamic second threshold to 0.05 and the dynamic third threshold to 0.02.

[0085] Check if the variance of the timestamp difference is less than the dynamic second threshold. If yes, proceed to the next step; otherwise, the target compression mode is uncertain.

[0086] Check if the variance of the data load differential value is less than the dynamic third threshold. If both conditions are met, determine the target compression mode as the improved XOR differential coding mode.

[0087] If none of the conditions are met, the target compression mode is determined to be Raw mode.

[0088] This embodiment provides a flexible compression strategy that can effectively improve the efficiency of data storage and transmission.

[0089] S104: The writing thread compresses and encodes the sliding window data based on the target compression mode, and stores the compressed and encoded data in a hierarchical manner through the writing thread; wherein, the hierarchical storage area includes a memory-level pre-cache, and the control thread accesses the mapping area of ​​the memory-level pre-cache to read data.

[0090] Based on the target compression mode, the sliding window data is encoded using an appropriate compression algorithm (such as gzip, LZ4, etc.). The compression process involves transforming the original data into a smaller byte format, making subsequent storage and transmission more efficient.

[0091] The compressed data is stored in layers. First, it is stored in a memory-level front cache for fast access; then, the data is periodically written to persistent storage (such as SSD or HDD).

[0092] The hierarchical storage area includes a memory-level front cache, and the control thread accesses the mapping area of ​​the memory-level front cache to read data.

[0093] Assuming the original size of the data within the sliding window is 100 KB, after compression using the LZ4 algorithm, the size is reduced to 30 KB. The system first stores this 30 KB compressed data in a memory-level cache, and then writes it to the SSD after certain conditions are met.

[0094] In one embodiment, the hierarchical storage area further includes a write-ahead log and disk shard files. The compressed and encoded data is stored hierarchically by the write thread, including the following steps: the compressed and encoded data is first written to a memory-level pre-cache by the write thread, then asynchronously written to the write-ahead log, and then written in batches to the disk shard files.

[0095] During system startup, the write thread initializes and connects to the memory-level pre-cache, write-ahead log, and disk storage area. The received data is compressed and encoded within the write thread, using algorithms such as gzip or LZ4.

[0096] The compressed and encoded data is first written to a memory-level pre-cache. The state of the memory-level pre-cache is then updated to ensure data availability.

[0097] The compressed and encoded data is asynchronously written to the write-ahead log, including information such as timestamp and data type.

[0098] Periodically or when specific conditions are met, collect data to be written from the memory-level front cache.

[0099] The collected data is written in batches to disk shard files to improve write efficiency.

[0100] S105: The compressed and encoded data is sent to the cloud according to the preset upload mechanism.

[0101] Set an upload strategy, which can be proactive or based on instructions. For example, upload at fixed time intervals (e.g., every 5 minutes) or when the cache reaches a certain size (e.g., 1 MB). This mechanism can effectively manage bandwidth and optimize upload frequency.

[0102] Compressed data is transmitted to cloud servers using lightweight protocols such as MQTT or HTTP / HTTPS. These protocols are commonly used in IoT devices and offer low latency and high reliability.

[0103] The compressed data packet is sent to the designated cloud address via the network interface. During the upload process, data integrity checks (such as CRC checks) may be performed to ensure that the data is not corrupted during transmission.

[0104] After a successful upload, the system can choose to delete the uploaded data from the memory-level cache, freeing up space to store new data.

[0105] Assuming the system is configured to upload data every 5 minutes, when the compressed data stored in the cache reaches 1 MB, the system will immediately send it to the cloud server via the MQTT protocol. After successful upload, if confirmation is received, the corresponding data in memory will be released.

[0106] In this embodiment, after uploading to the cloud, the cloud can perform hash verification to fill in missing data. When edge nodes transmit data, they include a first hash value; when the cloud receives the data, it also calculates a second hash value. If the two hash values ​​are inconsistent, it indicates data loss during transmission. In the event of data loss, transmission can be re-attempted.

[0107] In one implementation, by detecting the sequence number and timestamp of data packets, data integrity and consistency are ensured. This is particularly effective in poor network conditions, managing out-of-order data packets, and ultimately analyzing, compressing, and storing the data. Data packets are configured with at least a sequence number and a timestamp. This embodiment may also include steps S106-108, such as... Figure 5 As shown, S106~108 are detailed below: S106: Detect data continuity in real time based on the sequence number, determine whether the current network is in a weak network state, and if so, trigger sliding window expansion to temporarily store out-of-order data packets.

[0108] In this embodiment, the sequence number is a unique identifier for each data packet, indicating the order in which the data was sent. The timestamp records the precise time when the data packet was generated.

[0109] Extract the sequence number and timestamp from the received data packets. Compare the sequence number of the current data packet with that of the previous data packet to determine if they are consecutive. If a discontinuity in sequence numbers is detected, it indicates a potential network problem. Further analyze the current network status; if it is determined to be in a weak network state (e.g., high packet loss rate, long latency), trigger the sliding window expansion mechanism.

[0110] Configure a dynamic cache to temporarily store out-of-order data packets, ensuring that data is not lost due to network fluctuations.

[0111] In one implementation, S106 may include: real-time detection of data continuity based on the sequence number to obtain a data continuity index; if the data continuity index meets a preset first data out-of-order identification standard, triggering a sliding window expansion to temporarily store the out-of-order data packets. Furthermore, if the data continuity index meets a preset second data out-of-order identification standard, triggering a timestamp verification; when the timestamp corresponding to the out-of-order data packet is within a target time range, writing the data of the out-of-order data packet into the data buffer; when the timestamp corresponding to the out-of-order data packet is not within the target time range, marking the out-of-order data packet as an abnormal data packet and sending an exponential backoff retransmission instruction to the IoT device.

[0112] In the above embodiment, the sequence number of received data packets is monitored in real time, and the range of most recently received sequence numbers is recorded. A data continuity index is calculated, such as the continuity difference of sequence numbers. If the sequence number of a received data packet differs from that of the previous data packet by more than 5 but less than or equal to 10, it is considered that out-of-order data has occurred. At this time, the data continuity index will show discontinuity. The out-of-order window is then expanded (from the default 10 data packets to 20) to temporarily store the data.

[0113] If the data continuity metric meets the first out-of-order data identification criterion (e.g., sequence numbers are found to be discontinuous), then a sliding window expansion is triggered. After expansion, out-of-order data packets are temporarily stored in the expanded sliding window. Out-of-order window expansion (from the default 10 packets to 20 packets) can be triggered to temporarily store data.

[0114] If the data continuity metric meets the second out-of-order data identification criterion (e.g., continuity difference reaches a certain level), then timestamp verification is performed on the out-of-order data packets in the sliding window. The timestamps of the out-of-order data packets are checked to see if they are within a preset target time range. For example, timestamp verification is initiated when the data continuity metric is >10. If the timestamp is within a reasonable time range (±500ms), it is placed in the corresponding window; otherwise, it is marked as abnormal and exponential backoff retransmission is initiated.

[0115] If the timestamp is within the target time range, write the out-of-order data packet to the data buffer. If the timestamp is outside the target time range, mark the data packet as an abnormal data packet and send an exponential backoff retransmission command to the IoT device.

[0116] It can effectively identify and process out-of-order data, ensuring data continuity and timeliness. Utilizing sliding window expansion and timestamp verification mechanisms not only improves the flexibility and accuracy of data processing but also enhances system reliability in the face of network instability through the marking and retransmission strategies for abnormal data packets.

[0117] S107: Verify the out-of-order data packet according to the preset dual-verification writing standard; wherein, the dual-verification writing standard includes sequential continuity verification and time rationality verification.

[0118] The system iterates through the temporarily stored out-of-order data packets, extracting their sequence numbers and timestamps. It then performs a sequence continuity check; if any discrepancies are found, the data is marked as abnormal. Finally, it performs a time validity check; if the timestamps do not conform to the order, the data is also marked as abnormal.

[0119] For flagged abnormal data, you can choose to log it or request a retransmission as needed.

[0120] Out-of-order data can be written to persistent storage under two conditions: 1) Sequence continuity condition: When the global sequence number (SequenceID) of a data packet temporarily stored in the adaptive out-of-order window fills the gap in the current expected write sequence, that is, when it forms a continuous sequence with the already persisted data.

[0121] 2) Time reasonableness verification conditions: Based on the sequence number verification, the device local timestamp carried by the data packet is within a reasonable range inferred from the timestamps of the preceding and following data (e.g., the ±500ms range mentioned in the document).

[0122] Only when both the sequence number and the timestamp are consecutive are met is the out-of-order data considered to have been rearranged, retrieved from the temporary cache, and written in order to the front cache and subsequent storage links, thereby ensuring time sequence consistency.

[0123] S108: When the preset double-check write standard is met, the analysis, compression encoding and write operations are performed in sequence.

[0124] After the data packet passes double verification, data analysis is performed first to extract useful information. The analyzed data is then compressed and encoded to reduce storage space and transmission bandwidth requirements. The compressed data is then written to the target storage (such as a database, cloud storage, etc.).

[0125] In this implementation, real-time detection, dual verification, and efficient processing ensure data integrity and consistency in unstable network environments, ultimately achieving efficient data storage and utilization. This method not only improves data reliability but also enables the data processing system to flexibly respond to different network conditions, demonstrating strong adaptability and broad application prospects.

[0126] In one embodiment, after S107, S109-S110 may also be included, such as... Figure 6 As shown, S109~S110 are as follows: S109: When the preset double-check write criterion is not met, move the out-of-order data packet to the reorder buffer, perform context correlation analysis on the out-of-order data packet in the reorder buffer, determine the chronological position of the out-of-order data packet, and place the out-of-order data packet in its proper position.

[0127] In S107, if it is detected that a certain data packet fails the double-check, the system regards it as an out-of-order data packet. Temporarily move the conflicting data packet out of the current out-of-order sorting process and allocate an independent, temporary reorder buffer (new block). The out-of-order data packet will be moved to the reorder buffer to ensure no data loss. <000032I>Within this new block, perform a more complex correlation analysis in combination with the historical data characteristics of the device, the continuity of the data payload, and the network quality metrics to infer its most likely chronological position. That is, perform context correlation analysis on all out-of-order data packets in the reorder buffer, and use the timestamp and sequence number information of the data packets to determine the potential chronological order of each data packet.

[0129] After the analysis is completed, the system will attempt to reinsert it into a reasonable position in the main sorting link or mark it as a pending data block for special handling. If it still cannot be determined ultimately, it will be marked as "suspicious data" and trigger an exponential backoff retransmission request to apply for a data copy near the sequence number from the source device for final comparison and repair. By analyzing the timestamps of the data packets, construct a sorting logic to place the data packets in chronological order.

[0130] Suppose the timestamps of three received data packets are T1, T3, T2 (i.e., T1 < Tz < T3), and at this time T2 is the out-of-order data packet. Through context correlation analysis, the system can know that T2 should be before T3, so T2 is placed in its proper position in the reorder buffer.

[0131] S110: If the out-of-order data packet cannot be placed in its proper position, mark the out-of-order data packet as an abnormal data packet and send an exponential backoff retransmission instruction to the IoT device.

[0132] If a data packet cannot be placed in its proper position through context correlation analysis in the reorder buffer, the system marks it as an abnormal data packet. After marking, the system sends an "exponential backoff retransmission instruction" to the source device of the original data packet.

[0133] A retransmission strategy that gradually increases the retransmission interval when the device fails to successfully send data to reduce network congestion.

[0134] If data packet T2 cannot find a suitable position in the reordering buffer, the system marks it as abnormal. In this case, the system may send a command to the device that sent T2, requesting it to try retransmitting after a certain period of time, such as waiting 1 second for the first retransmission, 2 seconds for the second retransmission, 4 seconds for the third retransmission, and so on.

[0135] Through S109 and S110, data integrity and order are ensured when faced with out-of-order packets and network instability. The use of reordering buffers and intelligent handling of abnormal packets not only improves the system's fault tolerance but also enhances the efficiency and reliability of data processing in the IoT environment.

[0136] In one embodiment, it may further include S111~S113, such as Figure 7 As shown, S111~S113 are as follows: S111: The control thread accesses the latest data in the mapping area of ​​the memory-level front cache at preset intervals.

[0137] Define a preset time interval (e.g., every 5 seconds) to allow the control thread to run periodically. At system startup, create and start a control thread responsible for periodically accessing the data.

[0138] The control thread accesses the mapped area in the memory-level pre-cache at preset intervals to obtain the latest time-series data. The mapped area is a region in memory specifically used to store the latest received data.

[0139] S112: Based on the latest data and the preset detection algorithm, determine whether there is any abnormality.

[0140] The latest data is analyzed based on preset detection algorithms. These algorithms can be based on thresholds, statistical methods, machine learning models, etc. The latest data is compared with the normal range to determine if any anomalies exist. For example, if the temperature reading exceeds the set upper limit, an anomaly is considered to have occurred.

[0141] Specifically, the analysis primarily focuses on the real-time values ​​and trends of key physical quantities (such as voltage, current, and temperature) read from the latest data window in the circular buffer. The core objective is to detect any abnormal states, such as: whether the voltage drops by more than 15%, whether the current exceeds a safe threshold, or whether the load undergoes a sudden change.

[0142] During analysis, a lightweight real-time state machine based on rules and thresholds is employed. The algorithm continuously calculates the instantaneous values, short-term averages, and rates of change (differences) of key indicators and compares them in real-time with preset safety thresholds or normal fluctuation ranges at edge nodes. Once any indicator exceeds the threshold, it is determined to be an abnormal state.

[0143] S113: If so, determine the abnormal type corresponding to the abnormal situation, generate and issue a real-time control command corresponding to the abnormal type.

[0144] Once an anomaly is detected, the system will analyze the nature of the anomaly and determine its corresponding anomaly type.

[0145] Based on the type of anomaly, generate corresponding real-time control commands (e.g., start the cooling system, issue an alarm, etc.).

[0146] The generated control commands are sent to the relevant devices or systems to execute corresponding countermeasures.

[0147] For example, when the voltage drops suddenly, commands to "close the backup energy storage device switch" or "adjust the inverter output" are generated; when there is an overcurrent, commands to "disconnect a branch" or "reduce power" are generated.

[0148] In this embodiment, sub-second (millisecond) autonomous fault isolation and system protection are achieved, minimizing the impact of anomalies on the system (such as microgrids and industrial equipment) and ensuring stable system operation. This breaks through the limitation of traditional time-series databases being used only for post-event tracing, making them the core of perception and decision-making in real-time control closed loops.

[0149] In this embodiment, data compression and hierarchical storage significantly reduce storage space requirements and improve data transmission efficiency, while lowering bandwidth consumption. The design of a memory-level pre-caching layer and control thread ensures real-time data processing, meeting the stringent latency requirements of IoT applications. A sliding window mechanism dynamically manages data features, and a decision tree model optimizes the compression strategy, effectively addressing data changes and reducing the loss of critical data. Upload time and frequency are flexibly adjusted based on network conditions and system load to optimize system performance. This method exhibits good scalability, adapting to IoT applications of different scales and types. As the number of devices increases, the system can handle higher data traffic by adding edge nodes and expanding storage capacity. Furthermore, the compression algorithm and decision tree model can be retrained and optimized based on new datasets to adapt to constantly changing sensor data characteristics. This method is not only applicable to various IoT application scenarios but also helps improve system real-time performance, reduce the risk of data loss, and possesses good scalability, providing strong support for future smart cities, smart agriculture, and industrial automation.

[0150] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0151] Please see Figure 8, Figure 8 This is a schematic diagram of an Internet of Things (IoT) time-series data processing system provided in the second embodiment of this application. The IoT time-series data processing system includes: IoT device 81, edge node 82, and cloud 83, wherein a time-series database is deployed on the edge node. The IoT device establishes a data connection with the edge node, and the edge node establishes a data connection with the cloud; The edge node is used to perform the steps in the IoT time-series data processing method mentioned in the first embodiment.

[0152] In this embodiment, an IoT time-series data processing system is used to efficiently manage and process time-series data generated by IoT devices. This system comprises three main components: IoT devices, edge nodes, and the cloud. A time-series database is deployed on the edge nodes, responsible for collecting, storing, and processing data from IoT devices, and synchronizing and analyzing the data with the cloud.

[0153] Internet of Things (IoT) devices refer to various sensors, smart devices, etc., which can collect environmental data, such as temperature, humidity, and light intensity, and transmit the data through a network.

[0154] Edge nodes are computing and storage devices deployed close to the data source, responsible for timely data processing and reducing latency. Time-series databases are deployed on edge nodes, enabling rapid response to real-time data requests. IoT time-series data processing methods are applied to time-series databases deployed on edge nodes.

[0155] The cloud refers to centralized servers or cloud computing platforms used for storing and analyzing large amounts of data. Complex calculations and long-term data analysis can be performed in the cloud.

[0156] For specific details, please refer to the detailed description in the first embodiment, which will not be repeated here.

[0157] Please see Figure 9 , Figure 9 This is a schematic diagram of an IoT time-series data processing apparatus provided in the third embodiment of this application. The included units are used for performing... Figures 1-7 The steps in the corresponding embodiments. Please refer to the details. Figures 1-7 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 9 The IoT time-series data processing device 9 includes: The first processing unit 910 is used to receive data packets sent by IoT devices and write the data packets into a data buffer. The second processing unit 920 is used to acquire sliding window data and, based on the sliding window data and a preset window statistical analysis algorithm, analyze and generate a window feature vector; wherein, the sliding window data is obtained from the data buffer based on the sliding window, and the window feature vector is the quantization result of the data features of the sliding window data; The third processing unit 930 is used to determine the target compression mode based on the window feature vector and the preset rule-based decision tree model; The fourth processing unit 940 is used to compress and encode the sliding window data based on the target compression mode through a write thread, and to store the compressed and encoded data in a hierarchical manner through the write thread; wherein, the hierarchical storage area includes a memory-level pre-cache, and the control thread accesses the mapping area of ​​the memory-level pre-cache to read data; The fifth processing unit 950 is used to send the compressed and encoded data to the cloud according to a preset upload mechanism.

[0158] Furthermore, the data packet is configured with at least a sequence number and a timestamp; the IoT time-series data processing device also includes: The sixth processing unit is used to detect data continuity in real time based on the sequence number, determine whether the current network is in a weak network state, and if so, trigger the sliding window expansion to temporarily store out-of-order data packets. The seventh processing unit is used to verify the out-of-order data packets according to a preset dual-verification writing standard; wherein, the dual-verification writing standard includes sequential continuity verification and time rationality verification. The eighth processing unit is used to perform analysis, compression encoding and writing operations in sequence when the preset double-check writing standard is met.

[0159] Furthermore, the IoT time-series data processing device also includes: The ninth processing unit is used to move the out-of-order data packet out to the reordering buffer when the preset double-check writing standard is not met, and to perform context association analysis on the out-of-order data packet in the reordering buffer to determine the time order position of the out-of-order data packet and to reorder the out-of-order data packet. The tenth processing unit is used to mark the out-of-order data packet as an abnormal data packet if it is impossible to return the out-of-order data packet to its original position, and send an exponential backoff retransmission instruction to the Internet of Things device.

[0160] Furthermore, the sixth processing unit is specifically used for: The data continuity index is obtained by detecting the data continuity in real time based on the serial number. If the data continuity index meets the preset first data out-of-order identification standard, the sliding window is expanded to temporarily store the out-of-order data packets; The Internet of Things (IoT) time-series data processing device also includes: The eleventh processing unit is used to trigger timestamp verification if the data continuity index meets the preset second data disorder identification standard. The twelfth processing unit is used to write the data of the out-of-order data packet into the data buffer when the timestamp corresponding to the out-of-order data packet is within the target time range; The thirteenth processing unit is used to mark the out-of-order data packet as an abnormal data packet and send an exponential backoff retransmission instruction to the IoT device when the timestamp corresponding to the out-of-order data packet is not within the target time range.

[0161] Furthermore, the second processing unit is specifically used for: Based on the sliding window data and the preset window statistical analysis algorithm, at least the variance of the timestamp difference, the variance of the data load difference, and the degree of jump are analyzed; wherein, the timestamp difference is the difference between the collection timestamps of two adjacent data points, the data load difference is the difference between the data values ​​of two adjacent data points, and the degree of jump refers to the fluctuation range between the maximum difference and the minimum difference between two adjacent data points; The window feature vector is generated based on the variance of the timestamp difference, the variance of the data load difference, and the degree of hopping.

[0162] Furthermore, the third processing unit is specifically used for: When the variance of the timestamp difference is less than a dynamic first threshold, the target compression mode is determined to be RLE continuous time difference coding mode; wherein, the dynamic first threshold is dynamically configured based on the timestamp stability in the IoT application scenario; When the variance of the data load differential value is concentrated in a preset low bit range, the target compression mode is determined to be either packed encoding mode or simple8b encoding mode. When the variance of the timestamp difference is less than the dynamic second threshold and the variance of the data load difference is less than the dynamic third threshold, the target compression mode is determined to be an improved XOR differential coding mode; wherein, the dynamic second threshold is dynamically configured based on the timestamp volatility in the IoT application scenario, and the dynamic third threshold is dynamically configured based on the data load stability in the IoT application scenario. When the degree of data jump exceeds the dynamic fourth threshold, the target compression mode is determined to be Raw mode; wherein, the dynamic fourth threshold is dynamically configured based on the degree of data jump in the IoT application scenario.

[0163] Furthermore, the third processing unit is specifically used for: Determine whether the degree of the jump exceeds the dynamic fourth threshold; If the degree of jump exceeds the dynamic fourth threshold, the target compression mode is determined to be Raw mode; If the degree of change does not exceed the dynamic fourth threshold, determine whether the variance of the timestamp difference is less than the dynamic first threshold; If the variance of the timestamp difference is less than the dynamic first threshold, the target compression mode is determined to be the RLE continuous time difference coding mode. If the variance of the timestamp difference is not less than the dynamic first threshold, determine whether the variance of the data load difference is concentrated in a preset low bit range; If the variance of the data load differential value is concentrated in a preset low bit range, the target compression mode is determined to be either packed encoding mode or simple8b encoding mode. If the variance of the data load difference value is not concentrated in the preset low bit range, determine whether the variance of the timestamp difference value is less than the dynamic second threshold and whether the variance of the data load difference value is less than the dynamic third threshold. If the variance of the timestamp difference is less than the dynamic second threshold and the variance of the data load difference is less than the dynamic third threshold, the target compression mode is determined to be the improved XOR differential coding mode. If none of the conditions are met, the target compression mode is determined to be Raw mode.

[0164] Furthermore, the IoT time-series data processing device also includes: The fourteenth processing unit is used to access the latest data in the mapping area of ​​the memory-level front cache every preset time through the control thread; The fifteenth processing unit is used to determine whether there is an abnormal situation based on the latest data and the preset detection algorithm; The sixteenth processing unit is used to determine the abnormal type corresponding to the abnormal situation if the condition is true, and to generate and issue a real-time control command corresponding to the abnormal type.

[0165] Furthermore, the hierarchical storage area also includes write-ahead logs and disk fragment files, and the fourth processing unit is specifically used for: The compressed and encoded data is first written to a memory-level pre-cache by the write thread, then asynchronously written to the write-ahead log, and finally written in batches to the disk shard files.

[0166] Figure 10 This is a schematic diagram of the IoT time-series data processing device provided in the fourth embodiment of this application. Figure 10As shown, the IoT time-series data processing device 10 of this embodiment includes: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100, such as an IoT time-series data processing program. When the processor 100 executes the computer program 102, it implements the steps in the various IoT time-series data processing method embodiments described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 100 executes the computer program 102, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of modules 610 to 640 are shown.

[0167] For example, the computer program 102 can be divided into one or more modules / units, which are stored in the memory 101 and executed by the processor 100 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 102 in the Internet of Things (IoT) time-series data processing device 10. For example, the computer program 102 can be divided into a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit, with the specific functions of each unit as follows: The first processing unit is used to receive data packets sent by IoT devices and write the data packets into a data buffer. The second processing unit is used to acquire sliding window data and, based on the sliding window data and a preset window statistical analysis algorithm, analyze and generate a window feature vector; wherein, the sliding window data is obtained from the data buffer based on the sliding window, and the window feature vector is the quantization result of the data features of the sliding window data; The third processing unit is used to determine the target compression mode based on the window feature vector and the preset rule-based decision tree model. The fourth processing unit is used to compress and encode the sliding window data based on the target compression mode through a write thread, and to store the compressed and encoded data in a hierarchical manner through the write thread; wherein, the hierarchical storage area includes a memory-level pre-cache, and the control thread accesses the mapping area of ​​the memory-level pre-cache to read data; The fifth processing unit is used to send the compressed and encoded data to the cloud according to a preset upload mechanism.

[0168] The IoT time-series data processing device may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will understand that... Figure 10This is merely an example of the IoT time-series data processing device 10 and does not constitute a limitation on the IoT time-series data processing device 10. It may include more or fewer components than shown, or combine certain components, or different components. For example, the IoT time-series data processing device may also include input / output devices, network access devices, buses, etc.

[0169] The processor 100 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0170] The memory 101 can be an internal storage unit of the IoT time-series data processing device 10, such as a hard drive or memory of the IoT time-series data processing device 10. The memory 101 can also be an external storage device of the IoT time-series data processing device 10, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the IoT time-series data processing device 10. Furthermore, the IoT time-series data processing device 10 can include both internal storage units and external storage devices. The memory 101 is used to store the computer program and other programs and data required by the IoT time-series data processing device. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0171] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0172] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0173] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0174] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0176] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0177] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for processing time-series data in the Internet of Things, characterized in that, Applied to time-series databases deployed on edge nodes, the process includes the following steps: Receive data packets sent by IoT devices and write the data of the data packets into a data buffer; Obtain sliding window data, and analyze and generate a window feature vector based on the sliding window data and a preset window statistical analysis algorithm; wherein, the sliding window data is obtained from the data buffer based on the sliding window, and the window feature vector is the quantization result of the data features of the sliding window data; The target compression mode is determined based on the window feature vector and the preset rule-based decision tree model; The writing thread compresses and encodes the sliding window data based on the target compression mode, and then stores the compressed and encoded data in a hierarchical manner. The hierarchical storage area includes a memory-level pre-cache, and the control thread accesses the mapping area of ​​the memory-level pre-cache to read data. The compressed and encoded data is sent to the cloud according to the preset upload mechanism.

2. The IoT time-series data processing method according to claim 1, characterized in that, The data packet is configured with at least a sequence number and a timestamp, and the method further includes the following steps: Based on the sequence number, the continuity of data is detected in real time to determine whether the current network is in a weak state. If so, the sliding window is triggered to expand the capacity and temporarily store out-of-order data packets. The out-of-order data packets are verified according to a preset dual-verification writing standard; wherein, the dual-verification writing standard includes sequential continuity verification and time rationality verification. When the preset double-check writing standard is met, the analysis, compression encoding and writing operations are performed in sequence.

3. The IoT time-series data processing method according to claim 2, characterized in that, The method further includes the following steps: When the preset double-check write standard is not met, the out-of-order data packet is moved out to the reordering buffer, and the context association analysis of the out-of-order data packet in the reordering buffer is performed to determine the time order position of the out-of-order data packet and to reorder the out-of-order data packet. If the out-of-order data packets cannot be returned to their original positions, the out-of-order data packets are marked as abnormal data packets, and an exponential backoff retransmission command is sent to the IoT device.

4. The IoT time-series data processing method according to claim 2, characterized in that, The step of detecting data continuity in real time based on the sequence number to determine whether the current network is in a weak network state, and if so, triggering a sliding window expansion to temporarily store out-of-order data packets, includes the following steps: The data continuity index is obtained by detecting the data continuity in real time based on the serial number. If the data continuity index meets the preset first data out-of-order identification standard, the sliding window is expanded to temporarily store the out-of-order data packets; The method further includes the following steps: If the data continuity index meets the preset second data disorder identification standard, timestamp verification is triggered; When the timestamp corresponding to the out-of-order data packet is within the target time range, the data of the out-of-order data packet is written into the data buffer; When the timestamp corresponding to the out-of-order data packet is not within the target time range, the out-of-order data packet is marked as an abnormal data packet, and an exponential backoff retransmission instruction is sent to the IoT device.

5. The IoT time-series data processing method according to any one of claims 1 to 4, characterized in that, The step of generating a window feature vector based on the sliding window data and a preset window statistical analysis algorithm includes the following steps: Based on the sliding window data and the preset window statistical analysis algorithm, at least the variance of the timestamp difference, the variance of the data load difference, and the degree of jump are analyzed; wherein, the timestamp difference is the difference between the collection timestamps of two adjacent data points, the data load difference is the difference between the data values ​​of two adjacent data points, and the degree of jump refers to the fluctuation range between the maximum difference and the minimum difference between two adjacent data points; The window feature vector is generated based on the variance of the timestamp difference, the variance of the data load difference, and the degree of hopping.

6. The IoT time-series data processing method according to claim 5, characterized in that, The step of determining the target compression mode based on the window feature vector and a preset rule-based decision tree model includes the following steps: When the variance of the timestamp difference is less than a dynamic first threshold, the target compression mode is determined to be RLE continuous time difference coding mode; wherein, the dynamic first threshold is dynamically configured based on the timestamp stability in the IoT application scenario; When the variance of the data load differential value is concentrated in a preset low bit range, the target compression mode is determined to be either packed encoding mode or simple8b encoding mode. When the variance of the timestamp difference is less than the dynamic second threshold and the variance of the data load difference is less than the dynamic third threshold, the target compression mode is determined to be an improved XOR differential coding mode; wherein, the dynamic second threshold is dynamically configured based on the timestamp volatility in the IoT application scenario, and the dynamic third threshold is dynamically configured based on the data load stability in the IoT application scenario. When the degree of data jump exceeds the dynamic fourth threshold, the target compression mode is determined to be Raw mode; wherein, the dynamic fourth threshold is dynamically configured based on the degree of data jump in the IoT application scenario.

7. The IoT time-series data processing method according to claim 5, characterized in that, The step of determining the target compression mode based on the window feature vector and a preset rule-based decision tree model includes the following steps: Determine whether the degree of the jump exceeds the dynamic fourth threshold; If the degree of jump exceeds the dynamic fourth threshold, the target compression mode is determined to be Raw mode; If the degree of change does not exceed the dynamic fourth threshold, determine whether the variance of the timestamp difference is less than the dynamic first threshold; If the variance of the timestamp difference is less than the dynamic first threshold, the target compression mode is determined to be the RLE continuous time difference coding mode. If the variance of the timestamp difference is not less than the dynamic first threshold, determine whether the variance of the data load difference is concentrated in a preset low bit range; If the variance of the data load differential value is concentrated in a preset low bit range, the target compression mode is determined to be either packed encoding mode or simple8b encoding mode. If the variance of the data load difference value is not concentrated in the preset low bit range, determine whether the variance of the timestamp difference value is less than the dynamic second threshold and whether the variance of the data load difference value is less than the dynamic third threshold. If the variance of the timestamp difference is less than the dynamic second threshold and the variance of the data load difference is less than the dynamic third threshold, the target compression mode is determined to be the improved XOR differential coding mode. If none of the conditions are met, the target compression mode is determined to be Raw mode.

8. The IoT time-series data processing method according to any one of claims 1 to 4, characterized in that, The method includes the following steps: The control thread accesses the latest data in the mapping area of ​​the memory-level front cache at preset intervals; Based on the latest data and the preset detection algorithm, determine whether there are any abnormal situations; If so, determine the anomaly type corresponding to the anomaly, and generate and issue a real-time control command corresponding to the anomaly type.

9. The IoT time-series data processing method according to any one of claims 1 to 4, characterized in that, The tiered storage area also includes write-ahead logs and disk fragment files. The step of storing the compressed and encoded data in a tiered manner through the write thread includes the following steps: The compressed and encoded data is first written to a memory-level pre-cache by the write thread, then asynchronously written to the write-ahead log, and finally written in batches to the disk shard files.

10. An Internet of Things (IoT) time-series data processing system, characterized in that, include: Internet of Things (IoT) devices, edge nodes, and the cloud, with a time-series database deployed on the edge nodes; The IoT device establishes a data connection with the edge node, and the edge node establishes a data connection with the cloud; The edge node is used to perform the steps in the IoT time-series data processing method as described in any one of claims 1 to 9.

Images