Real-time data processing system and method for Internet of Things
Through the sliding window and dynamic threshold data filter combined with an exception confirmation mechanism, the problem of data processing delay and false alarms and missed reports of long-tail equipment is solved, and efficient and low-latency real-time data processing of IoT is achieved, adapting to different scenarios, reducing the load and false alarm rate in the cloud.
Patent Information
- Application Number
- CN202510455914.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-11
- Publication Date
- 2025-08-29
AI Technical Summary
The data update frequency of long-tail devices in the Internet of Things system is low, resulting in critical events not being discovered in time, and low-frequency data signals are easily submerged in high-frequency data noise, increasing the risk of false alarms or missed reports and reducing system reliability.
The data filter with sliding window and dynamic threshold is used for abnormal detection, combined with the exception confirmation mechanism and event detector, high-value data is transmitted to the cloud through communication protocol, Snappy LZ4 is used to compress the data, and data filtering and compression are performed at the edge layer, edge gateways perform data filtering and event detection, and the cloud receives high-priority data and triggers alarms.
Real-time data processing with low latency is realized, which reduces network bandwidth requirements, reduces cloud load, improves system efficiency and response speed, adapts to different scenarios, and reduces false alarms and missed alarm rates.
Smart Images

Figure CN120567800A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and more particularly, to a real-time data processing system and method for the Internet of Things. Background Art
[0002] In the Internet of Things (IoT) system, long-tail devices refer to those with low usage frequency, small data generation volumes, or infrequent data updates. These devices are very common in real-world applications, such as industrial sensors, environmental monitoring equipment, and certain smart home devices. Although these devices generate data infrequently, their data may contain critical information (such as abnormal events and fault warnings), and therefore also requires real-time processing.
[0003] Because long-tail devices update data infrequently, the system may prioritize data from high-frequency devices, potentially preventing critical events (such as device failures and abnormal conditions) from being detected promptly. Furthermore, during anomaly detection, signals from low-frequency data may be drowned out by the noise of high-frequency data, increasing the risk of false positives or missed alerts. This, in turn, reduces the system's ability to monitor long-tail devices and can potentially reduce overall system reliability.
[0004] In view of this, the present invention proposes a real-time data processing system for the Internet of Things to solve the above problems. Summary of the Invention
[0005] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned objectives, the present invention provides the following technical solutions: a real-time data processing system for the Internet of Things, comprising long-tail devices, wherein data from the long-tail devices is transmitted to an edge layer via a communication protocol, an edge gateway in the edge layer is used to receive the data, and the edge layer also includes a data filter, which discards useless data based on filtering rules and retains valuable data; The data filter uses a sliding window and dynamic thresholds for anomaly detection. The sliding window size is adjusted according to the data change rate, the amplitude threshold is dynamically adjusted based on the mean and standard deviation within the window, and the change rate threshold is based on the quantile of the historical change rate queue. Based on the filtering rules, the data is marked as normal data and potential anomalies. Through the anomaly confirmation mechanism, high-value data is finally output. The event detector detects anomalies in valuable data through model rules. If the data is abnormal, it will be divided into a high-priority queue, otherwise it will be placed in a low-priority queue. High-priority data will be transmitted to the cloud for application first, while low-priority data will be processed in a stagnant state. The cloud is used to receive high-priority data and trigger alarm notifications.
[0006] Preferably, useless data is discarded based on the filtering rules, and valuable data is retained, normal data in long-tail devices is recorded as useless data, and data with potential anomalies is recorded as valuable data; Use Snappy LZ4 to compress the valuable data, and then use the event detector on the compressed data; Stalled processing is local caching, waiting for batch upload, or deciding whether to upload immediately based on system load.
[0007] Preferably, the data filter discards useless data based on filtering rules, and the method of retaining valuable data includes: Initialize the sliding window size, initialize the change rate threshold and amplitude threshold, initialize the abnormal confirmation counter to 0, and initialize the historical change rate queue to empty; The edge gateway continuously collects data and arranges it in time series, adding the latest data to the sliding window. If the sliding window is full, the oldest data is removed. Calculate the rate of change between the data at each time in the previous sliding window and adjust the size of the next sliding window according to the rate of change; Calculate the mean and standard deviation of the data in the previous sliding window, and set a dynamic mechanism to adjust the amplitude threshold range in the next sliding window based on the mean and standard deviation; Calculate the rate of change between the current time and the previous time, and add the rate of change to the historical rate of change queue. If the queue exceeds the preset length, remove the rate of change at the oldest time, and calculate a certain percentile of the rate of change in the historical rate of change queue as the new rate of change threshold; The smoothing rule is introduced into the adjustment and update of the change rate threshold and the amplitude threshold to obtain the change rate threshold and amplitude threshold finally used at the current time; For the data at the current time, calculate the average value and average rate of change within the sliding window, check whether the average value is within the amplitude threshold range, and check whether the average rate of change exceeds the rate of change threshold. If it is within the amplitude threshold range and the rate of change does not exceed the threshold, it is considered normal data and the anomaly confirmation counter is reset to 0. If it exceeds the amplitude threshold range or the rate of change exceeds the rate of change threshold, it is marked as a potential anomaly and the anomaly confirmation counter is incremented by 1. An anomaly confirmation mechanism is introduced for data marked as potential anomalies to output high-value data.
[0008] Preferably, the method of introducing a smoothing rule into the adjustment of the change rate threshold and the amplitude threshold includes: Get the adjusted rate-of-change threshold or amplitude threshold at the current time and record it as the new threshold. For the rate-of-change threshold or amplitude threshold, maintain a fixed-size queue to store the thresholds at the past N times. Add the new threshold to the queue. If the queue is full, remove the threshold at the oldest time. Calculate the average and median of all thresholds in the queue, perform weighted average of the average and median, record them as smoothing thresholds, and use the smoothing thresholds as the final change rate thresholds and amplitude thresholds used at the current time.
[0009] Preferably, the method of introducing an anomaly confirmation mechanism for data marked as potentially abnormal includes: For data marked as potential anomalies, it is not immediately judged as abnormal. Instead, it enters the confirmation stage and sets a blank counter. If P consecutive data points are marked as potential anomalies, the counter is increased by 1. If a normal data point appears, the counter is reset to 0. When the counter reaches the preset P value, the data is finally confirmed to be abnormal and recorded as valuable data.
[0010] Preferably, the event detector performs anomaly detection on valuable data using model rules, and the method for classifying the data into a high-priority queue or a low-priority queue includes: Select the appropriate anomaly detection algorithm based on the long-tail device type and data characteristics, and compress and lightweight the anomaly detection algorithm.
[0011] Preferably, the method for compressing and lightweighting the anomaly detection algorithm includes: Use TensorFlow to train a full-precision model, using a combination of group pruning and iterative pruning to gradually remove unimportant weight groups from the model, and then evaluate it. Selectively quantize some layers while maintaining floating-point precision for the remaining layers. Convert the optimized model to the TensorFlow Lite format, deploy the .tflite model to the target device, and use the TensorFlow Lite interpreter to load and run the model on the device.
[0012] Preferably, the method of combining group pruning and iterative pruning to gradually remove unimportant weight groups in the model and perform evaluation includes: Select an appropriate grouping strategy based on the model structure and divide the model weights into multiple groups; Preset pruning standards and pruning ratios. The pruning standard is to calculate the importance of each group, preset an important threshold, and remove the lowest q percent of groups whose importance is lower than the important threshold, where q percent is the pruning ratio. According to the selected pruning standard and pruning ratio, remove some weight groups, use a smaller learning rate to fine-tune the pruned model, evaluate the model accuracy on the validation set, adjust the pruning ratio or fine-tune the parameters based on the evaluation results, and repeat the steps until the target accuracy requirement is reached.
[0013] Preferably, the method of selectively quantizing some layers while maintaining floating-point precision for the remaining layers includes: Evaluate the model layer by layer after iterative pruning and analyze the sensitivity of each layer to quantization. That is, perform quantization sensitivity analysis and weight distribution analysis on each layer. Quantization sensitivity analysis is to quantize each layer separately and then evaluate the degree of accuracy drop of the model on the validation set; weight distribution analysis is to analyze the distribution range of weights of each layer; The sensitivity value is obtained by taking a weighted average of the degree of precision degradation and the distribution range, and a sensitivity threshold is preset. If the sensitivity value is less than the sensitivity threshold, it is recorded as a layer insensitive to quantization and the cover layer is quantized. Otherwise, it is recorded as a layer sensitive to quantization and the floating-point precision is maintained.
[0014] Preferably, the real-time data processing method for the Internet of Things includes: Step S1: The data of the long-tail device is sent to the edge gateway deployed at the edge layer of the network through the communication protocol; Step S2: After the edge gateway receives the raw data, the built-in data filter module is immediately activated and uses sliding window technology to perform real-time anomaly detection. The data filter can identify and mark potential abnormal data, and through further anomaly confirmation, finally output the filtered high-value data; Step S3: The filtered high-value data enters the event detector. Based on the detection results, the data is divided into different priority queues. Data detected as abnormal enters the high-priority queue, while normal data enters the low-priority queue and is processed separately. Step S4: Cloud-based high-priority data processing and alarm notification.
[0015] The technical effects and advantages of the real-time data processing system for the Internet of Things of the present invention are as follows: 1. From long-tail device data collection, edge-side filtering and detection, to cloud-based alerting, the entire process emphasizes low latency. Mechanisms such as sliding windows and dynamic thresholds ensure rapid response to data changes, while high-priority queues and fast transmission mechanisms ensure timely handling of abnormal events.
[0016] 2. By performing extensive data pre-processing (filtering, compression) and event detection at the edge layer, the amount of data that needs to be transmitted to the cloud is significantly reduced. This reduces network bandwidth requirements, reduces the load on cloud servers, and improves overall system efficiency and responsiveness.
[0017] 3. Targeting the characteristics of long-tail devices, the system optimizes resources at multiple levels. Data compression reduces storage and transmission overhead. Model compression and lightweighting (group pruning, iterative pruning, and selective quantization) enable complex anomaly detection models to run efficiently on resource-constrained edge devices.
[0018] 4. The system has strong adaptability. The sliding window size, amplitude threshold, and rate-of-change threshold can all be dynamically adjusted based on data characteristics. The anomaly confirmation mechanism (counter) prevents instantaneous fluctuations from being misidentified as anomalies, improving system robustness. Smoothing rules further enhance the stability of threshold adjustment.
[0019] In summary, the system architecture has good scalability. It can easily connect to more types of long-tail devices and adapt to different application scenarios by adjusting models and rules. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a schematic structural diagram of a real-time data processing system for the Internet of Things according to the present invention; Figure 2 The figure is a schematic diagram of the steps of a real-time data processing method for the Internet of Things of the present invention. DETAILED DESCRIPTION
[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. Example
[0022] See also Figure 1 and Figure 2 As shown, the real-time data processing system for the Internet of Things described in this embodiment includes: In the Internet of Things (IoT) system, long-tail devices refer to those with low usage frequency, small data generation volumes, or infrequent data updates. These devices are very common in real-world applications, such as industrial sensors, environmental monitoring equipment, and certain smart home devices. Although these devices generate data infrequently, their data may contain critical information (such as abnormal events and fault warnings), and therefore also requires real-time processing.
[0023] Data from long-tail devices typically has the following characteristics: Low-frequency updates: A device may send data only under specific conditions (such as when an exception occurs) and remain silent for a long time under normal circumstances.
[0024] High criticality: Although the amount of data is small, the data content may involve key events (such as equipment failure, environmental anomalies, etc.), which is crucial to the system's decision-making and response.
[0025] In IoT systems, resources (such as computing, storage, and bandwidth) are typically prioritized for high-frequency, high-data-volume devices, while data from long-tail devices is easily overlooked or processed with delays. However, neglecting these devices can lead to missed reports of critical events or delayed responses, resulting in serious consequences such as undetected industrial equipment failures and unresponsive environmental monitoring anomalies.
[0026] Because long-tail devices update data less frequently, the system may prioritize processing data from high-frequency devices, potentially leading to key events (such as device failures and abnormal conditions) not being discovered in a timely manner. This in turn reduces the system's ability to monitor long-tail devices, potentially lowering overall system reliability.
[0027] In anomaly detection, signals from low-frequency data can be lost in the noise of high-frequency data, increasing the risk of false positives or omissions. Furthermore, although data from long-tail devices is updated less frequently, it still requires real-time monitoring to ensure that critical events are not missed. However, processing low-frequency data in real time can increase system costs, so pre-filtering the data is necessary to reduce the amount of data to be processed and the number of processing steps.
[0028] The device layer includes high-frequency devices and long-tail devices. High-frequency devices refer to devices with frequently updated data, such as sensors on industrial production lines. Long-tail devices refer to various IoT devices with low-frequency data updates, such as environmental sensors and remote monitoring equipment. This invention processes the low-frequency data in long-tail devices in real time. The design content of the layout is as follows: The real-time data processing system for the Internet of Things includes long-tail devices. Data from these devices is transmitted to the edge layer via communication protocols (such as MQTT, CoAP, and HTTP). The edge gateway in the edge layer receives the data and also includes data filters. These filters discard useless data based on filtering rules and retain valuable data. The data filter uses a sliding window and dynamic thresholds for anomaly detection. The sliding window size is adjusted according to the data change rate, the amplitude threshold is dynamically adjusted based on the mean and standard deviation within the window, and the change rate threshold is based on the quantile of the historical change rate queue. Based on the filtering rules, the data is marked as normal data and potential anomalies. Through the anomaly confirmation mechanism, high-value data is finally output. The event detector detects anomalies in valuable data through model rules. If the data is abnormal, it will be divided into a high-priority queue, otherwise it will be placed in a low-priority queue. High-priority data will be transmitted to the cloud for application first, while low-priority data will be processed in a stagnant state. For example, if a high-priority event (such as a device failure) is detected, the data is immediately placed in the high-priority queue. If a low-priority event or normal data is detected, it is placed in the low-priority queue.
[0029] The cloud is used to receive high-priority data and trigger alarm notifications.
[0030] When long-tail devices output data, high accuracy is required because the data is critical. However, due to the inherent characteristics of long-tail devices, environmental interference, and other factors, the collected data often exhibits certain fluctuations. This fluctuation can easily lead to misjudgments, specifically false positives (misjudgment of normal fluctuations as anomalies) and false negatives (misjudgment of abnormal data as normal). Simply using rate-of-change filtering also has limitations, potentially resulting in Type 1 and Type 2 false positives. Type 1 false positives occur when two consecutive normal data points, even if both are within the normal range, have a significant difference in value (one close to the lower threshold, the other close to the upper threshold). This may cause the rate of change to exceed the preset rate-of-change threshold, leading to false positives. This is a "false alarm." Type 2 false positives occur when two consecutive abnormal data points, even if both are outside the normal range, have a small difference in value. Their rate of change may fall below the preset rate-of-change threshold, leading to false positives. This can result in false positives. This type of false positive is particularly prone to gradual anomalies, such as sensor drift.
[0031] Furthermore, rate-of-change filtering can introduce cumulative errors. Especially in long-term data series, even if the rate of change is within the acceptable range, small deviations can accumulate over time, ultimately causing the overall data to deviate from the true value.
[0032] Therefore, based on the above description, a filtering rule is designed, which is based on the adaptive threshold and change rate of the sliding window for joint filtering. Normal data in long-tail devices is recorded as useless data and removed, and data with potential anomalies is recorded as valuable data and retained. The specific design idea is to automatically adjust the window size according to data changes, balancing stability and response speed. The upper and lower limits of the normal value are dynamically adjusted according to the data in the window to adapt to data fluctuations. The upper limit of the change rate is dynamically adjusted according to the historical change rate to adapt to the change rate in different scenarios. The absolute value (amplitude threshold) and relative change (change rate threshold) of the data are considered at the same time to make a comprehensive judgment. A confirmation mechanism is introduced to avoid misjudgment caused by single fluctuations or occasional anomalies. The specific design content is as follows: Discard useless data based on filtering rules and retain valuable data. Normal data in long-tail devices is recorded as useless data, and data with potential anomalies is recorded as valuable data. Use Snappy LZ4 to compress the valuable data, and then use the event detector on the compressed data; Stalled processing is local caching, waiting for batch upload, or deciding whether to upload immediately based on system load.
[0033] Data filters discard useless data based on filtering rules. Methods for retaining valuable data include: Initialize the sliding window size to a default value (e.g., 10 data points), initialize the change rate threshold and amplitude threshold to a default value (e.g., 5%), initialize the anomaly confirmation counter to 0, and initialize the historical change rate queue to be empty; The edge gateway continuously collects data and arranges it in time series, adding the latest time data to the sliding window. If the sliding window is full, the oldest time data is removed; the window is kept full of data.
[0034] A fixed-size sliding window may not be suitable for all situations. When data changes gradually, a larger window can provide a more stable threshold estimate; when data changes dramatically, a smaller window can respond to changes more quickly. Calculate the rate of change between the data at each time point within the previous sliding window. The rate of change can be the average rate of change or the rate of change of the standard deviation. Adjust the size of the next sliding window based on the rate of change. For example, if the rate of change within the previous sliding window is continuously below a preset low threshold, gradually increase the size of the next sliding window (within a certain range, such as a maximum of 30). If the rate of change suddenly exceeds a preset high threshold, immediately reduce the sliding window (for example, to a minimum of no less than 5). Set upper and lower limits for the sliding window to avoid windows that are too large or too small.
[0035] A fixed rate of change threshold may not be suitable for all devices or all time periods. In some cases, a small rate of change may be an abnormal signal; in other cases, a large rate of change may be a normal fluctuation.
[0036] Calculate the mean and standard deviation of the data in the previous sliding window. Use the mean and standard deviation to set a dynamic mechanism to adjust the amplitude threshold range in the next sliding window. For example, if the mean is μ and the standard deviation is σ, then the upper threshold limit = μ + k * σ and the lower threshold limit = μ - k * σ. k is a configurable coefficient (for example, 2 or 3). You can set the upper and lower limits of k to prevent the threshold range from being too large or too small. Calculate the rate of change between the current and previous data points and add it to a historical rate of change queue. If the queue exceeds a preset length (e.g., 100 data points), remove the oldest data point and calculate a percentile of the data points in the historical rate of change queue (e.g., the 95th percentile) as the new rate of change threshold. Set upper and lower limits for the rate of change threshold to prevent it from being too large or too small. The smoothing rule is introduced into the adjustment and update of the change rate threshold and the amplitude threshold to obtain the change rate threshold and amplitude threshold finally used at the current time; For the data at the current time, calculate the average value and average rate of change within the sliding window, check whether the average value is within the amplitude threshold range, and check whether the average rate of change exceeds the rate of change threshold. If it is within the amplitude threshold range and the rate of change does not exceed the threshold, it is considered normal data and the anomaly confirmation counter is reset to 0. If it exceeds the amplitude threshold range or the rate of change exceeds the rate of change threshold, it is marked as "potential anomaly" and the anomaly confirmation counter is incremented by 1. An exception confirmation mechanism is introduced for data marked as "potential anomaly" to output high-value data.
[0037] For example: Consider a temperature sensor, where the normal temperature range is between 20-30°C.
[0038] Case 1 (Normal Fluctuation): The temperature fluctuates between 22°C, 23°C, 24°C, and 23°C. The adaptive threshold might be adjusted to around 21-25°C. Since the temperature fluctuations are small and the rate of change is very low, the data is considered normal.
[0039] Case 2 (abnormal sudden change): The temperature suddenly rises from 23°C to 35°C. Even with the adaptive threshold adjusted, 35°C still exceeds the threshold range, and the rate of change is large, so it is marked as a "potential anomaly." If the next few data points remain near 35°C, the anomaly confirmation mechanism will finally confirm this as an anomaly.
[0040] Case 3 (Slow Change to Anomaly): The temperature slowly rises from 23°C, increasing by 0.5°C each time, eventually reaching 32°C. Although the rate of change may be small, because 32°C exceeds the adaptive threshold range, it is still marked as a "potential anomaly" and is eventually confirmed through the anomaly confirmation mechanism.
[0041] Case 4 (Small Abnormal Value Change): Assume a fault occurs and the output is stable at 40°C. After a period of time, it changes to 41°C. Although the rate of change is small, it is still considered abnormal because the value continues to exceed the threshold.
[0042] For another example, consider a sensor that measures water level. The normal water level is between 10 and 15 meters.
[0043] Case 1 (Normal Fluctuation): The water level fluctuates between 11 meters, 12 meters, 11.5 meters, and 12.5 meters. The adaptive amplitude threshold might be adjusted to around 10.5-13 meters, and the rate of change is also very small, so the data is considered normal.
[0044] Case 2 (abnormal sudden change): The water level suddenly rises from 12 meters to 18 meters. Even with the adaptive amplitude threshold adjusted, 18 meters still exceeds the threshold range, and the rate of change is large, so it is marked as a "potential anomaly." If the next few data points remain near 18 meters, the anomaly confirmation mechanism will finally confirm this as an anomaly.
[0045] Case 3 (Slow Change to Anomaly): The water level slowly rises from 12 meters, in increments of 0.2 meters, to 16 meters. Although the rate of change may be small, 16 meters is still marked as a "potential anomaly" because it exceeds the adaptive amplitude threshold and is ultimately confirmed through the anomaly confirmation mechanism.
[0046] Case 4 (Small Change in Abnormal Value): Assume a fault occurs and the water level stabilizes at 20 meters. After a period of time, it changes to 20.5 meters. Although the rate of change is small, it is still considered abnormal because the value continues to exceed the threshold.
[0047] Case 5 (data change period): The water level remains stable around 12 meters for a long time, with a very low rate of change. The dynamic window mechanism gradually increases the window size to improve the stability of the threshold estimation.
[0048] Case 6 (period of drastic data changes): The water level suddenly fluctuates significantly. The dynamic window mechanism immediately reduces the window size to respond more quickly to the change.
[0049] The core idea of the smoothing rule is to limit the change range of the threshold within adjacent time steps to avoid drastic fluctuations in the threshold. Methods for adjusting the rate of change threshold and the amplitude threshold to introduce the smoothing rule include: Get the adjusted rate-of-change threshold or amplitude threshold at the current time and record it as the new threshold. For the rate-of-change threshold or amplitude threshold, maintain a fixed-size queue to store the thresholds at the past N times. Add the new threshold to the queue. If the queue is full, remove the threshold at the oldest time. Calculate the average and median of all thresholds in the queue, perform weighted average of the average and median, record them as smoothing thresholds, and use the smoothing thresholds as the final change rate thresholds and amplitude thresholds used at the current time.
[0050] Methods for introducing anomaly confirmation mechanisms for data marked as "potential anomalies" include: For data marked as "potential anomaly", it is not immediately judged as abnormal. Instead, it enters the confirmation stage and sets a blank counter. If P consecutive data points (for example, 3 consecutive data points) are marked as "potential anomaly", the counter is increased by 1. If a normal data point appears, the counter is reset to 0. When the counter reaches the preset P value (for example, 3), the data is finally confirmed to be abnormal and recorded as valuable data.
[0051] In summary, the dynamic sliding window size adjustment in the above technical solution automatically adjusts the window size based on the data change rate, balancing stability and responsiveness. During periods of stable data, a larger window provides more stable statistics (mean, standard deviation, and rate of change); during periods of drastic data fluctuations, a smaller window can more quickly detect anomalies. This improves the system's adaptability to diverse data patterns and avoids the limitations of fixed window sizes (overly large windows leading to slow response, while too small windows lead to sensitivity to noise).
[0052] Dynamically adjusting the threshold based on the mean and standard deviation of the data within the window adapts to the data's fluctuation range, avoiding false positives (fluctuations outside the fixed range) or false negatives (outliers within the fixed range) caused by fixed thresholds. This improves the accuracy of anomaly detection and reduces the false positive and false negative rates.
[0053] Dynamically adjusting the rate-of-change threshold based on the quantiles of the historical rate-of-change queue takes into account the long-term trend of data and can adapt to the rate of change of different devices and time periods. This improves the system's adaptability to different scenarios and avoids the limitations of fixed rate-of-change thresholds.
[0054] By considering both the absolute value (amplitude) and relative change (rate of change) of the data, a comprehensive assessment is made, effectively avoiding the limitations of single-metric filtering. For example, even if the rate of change is small, if the value continuously exceeds the amplitude threshold, it will still be identified as an anomaly (as shown in Case 4). This reduces the false positive and false negative rates, improving the comprehensiveness and accuracy of anomaly detection.
[0055] By maintaining a threshold queue and calculating the weighted average of the mean and median values, we limit the range of threshold changes within adjacent time steps, avoiding drastic threshold fluctuations and preventing misjudgments due to sudden threshold changes. This improves system stability and reduces false alarms caused by threshold fluctuations.
[0056] By marking anomalies as "potential anomalies" multiple times in a row, we can confirm anomalies and avoid false positives caused by single fluctuations or occasional anomalies. Only when multiple consecutive data points are marked as "potential anomalies" is the data finally confirmed as an anomaly. This improves the accuracy of anomaly detection and reduces the false positive rate.
[0057] Furthermore, when performing real-time processing, long-tail devices often have limited resources (computing power, memory, power consumption, etc.), making them unable to directly run complex anomaly detection models. Traditional machine learning models (such as TensorFlow models) are typically large and unsuitable for direct deployment on resource-constrained edge devices. Furthermore, IoT real-time data processing systems require low-latency anomaly detection, but complex models have slow inference speeds, making them difficult to meet real-time requirements.
[0058] The event detector detects anomalies in valuable data using model rules and classifies data into high-priority or low-priority queues using the following methods: Select appropriate anomaly detection algorithms based on long-tail device types and data characteristics, and compress and lightweight the anomaly detection algorithms; The following examples illustrate anomaly detection algorithms. For example, decision trees and random forests are suitable for scenarios with clear rules. Support vector machines (SVMs) are suitable for binary classification problems. K-nearest neighbor (KNN) is suitable for simple classification and regression problems. Lightweight neural networks such as MobileNet and SqueezeNet are also suitable for prediction. The choice of model and algorithm depends on your needs.
[0059] Methods for compressing and lightweighting anomaly detection algorithms include: This paper uses TensorFlow Lite, a lightweight solution for mobile or embedded devices. It consists of two main components: TensorFlow Lite Converter: Converts a trained TensorFlow model (usually a SavedModel or Keras model) to a smaller and more efficient .tflite format.
[0060] TensorFlow Lite Interpreter: A library for running .tflite models on target devices. It is optimized for mobile and embedded platforms, with a small binary size and fast loading speed.
[0061] Use TensorFlow to train a full-precision (typically 32-bit floating-point) model that performs well on the task (anomaly detection), i.e., achieves satisfactory accuracy on the validation set. Use a combination of group pruning and iterative pruning to gradually remove unimportant weight groups from the model and evaluate the model. Selectively quantize some layers, while maintaining floating-point precision for the remaining layers. Convert the optimized model to the TensorFlow Lite format, deploy the .tflite model to the target device (corresponding long-tail device), perform performance testing, and use the TensorFlow Lite interpreter to load and run the model on the device.
[0062] Methods for gradually removing unimportant weight groups from the model by combining group pruning and iterative pruning and evaluating them include: Select an appropriate grouping strategy based on the model structure and divide the model weights into multiple groups; Preset pruning standards and pruning ratios. The pruning standard is to calculate the importance of each group, preset an important threshold, and remove the lowest q percent of groups whose importance is lower than the important threshold, where q percent is the pruning ratio. There are many ways to group the grouping strategies. For example, for the convolution layer, filter level grouping: each filter (convolution kernel) is a group.
[0063] Channel level grouping: each output channel as a group.
[0064] Block-level grouping: Combine multiple filters or channels into one block as a group.
[0065] Custom grouping: Customize grouping based on model structure or weight distribution characteristics.
[0066] For the fully connected layer: Row grouping: Each row weight is treated as a group.
[0067] Column grouping: Each column weight is treated as a group.
[0068] Block grouping: Divide the weight matrix into multiple blocks, and each block is regarded as a group.
[0069] Commonly used materiality criteria include: Sum of absolute values of weights within a group (L1 norm): The absolute values of all weights within a group are added together.
[0070] Square root of the sum of squared weights within a group (L2 norm): The sum of the squares of all weights within the group, then take the square root.
[0071] The maximum absolute value of the weight within the group: the one with the largest absolute value among all the weights in the group.
[0072] Gradient-based / Activation-based: Statistics (such as mean and maximum) of the gradient / activation values corresponding to the weights within the group.
[0073] According to the selected pruning standard and pruning ratio, remove some weight groups, use a smaller learning rate to fine-tune the pruned model, evaluate the model accuracy on the validation set, adjust the pruning ratio or fine-tune the parameters based on the evaluation results, and repeat the steps until the target accuracy requirement is reached.
[0074] The validation set is the data consisting of the accuracy requirements.
[0075] Methods for selectively quantizing some layers while maintaining floating-point precision for the remaining layers include: Evaluate the model layer by layer after iterative pruning and analyze the sensitivity of each layer to quantization. That is, perform quantization sensitivity analysis and weight distribution analysis on each layer. Quantization sensitivity analysis involves quantizing each layer individually (for example, to 8-bit integers) and evaluating the accuracy degradation of the model on the validation set. Layers with greater accuracy degradation are more sensitive to quantization. Weight distribution analysis analyzes the distribution of weights within each layer. Layers with narrower, more uniform distributions are more suitable for quantization.
[0076] Perform weighted averaging on the degree of precision degradation and distribution range to obtain the sensitivity value. Preset the sensitivity threshold. If the sensitivity value is less than the sensitivity threshold, it is recorded as a layer insensitive to quantization and the cover layer is quantized. Otherwise, it is recorded as a layer sensitive to quantization and the floating point precision is maintained. The above quantization method is to select full integer quantization after training and use a representative dataset for calibration.
[0077] In summary, the above technical solution effectively reduces the model size, memory usage, and computational complexity by gradually removing unimportant weight groups from the model. This enables the model to run on resource-constrained long-tail devices, reducing storage and computational overhead.
[0078] By quantizing some layers (e.g., 8-bit integer quantization), the model computation load is reduced, the inference speed is improved, the latency of anomaly detection is reduced, and the requirements of real-time data processing systems are met.
[0079] During the pruning process, fine-tuning and evaluation dynamically adjust the pruning ratio to avoid a significant drop in accuracy due to excessive pruning at once. Through quantization sensitivity analysis and weight distribution analysis, some layers are selectively quantized, while floating-point precision is maintained for quantization-sensitive layers, thus avoiding the accuracy loss caused by quantization. This allows for maximum model accuracy while compressing the model.
[0080] The optimized model was converted to TensorFlow Lite format and deployed on long-tail devices, enabling edge-side anomaly detection. This reduced reliance on the cloud, lowered network bandwidth requirements, and improved the overall efficiency and responsiveness of the system.
[0081] Through a series of model compression and lightweighting technologies (pruning, quantization, and TensorFlow Lite deployment), we address issues such as limited edge device resources, difficult model deployment, real-time requirements, and the balance between accuracy and efficiency. This enables the deployment of efficient and accurate anomaly detection models on resource-constrained, long-tail devices, providing powerful edge intelligence support for real-time IoT data processing systems. Example
[0082] See also Figure 1 As shown, for the parts not described in detail in this embodiment, please refer to the description of Example 1. A real-time data processing method for the Internet of Things is provided, including: Step S1: The data of the long-tail device is sent to the edge gateway deployed at the edge layer of the network through the communication protocol; Step S2: After the edge gateway receives the raw data, the built-in data filter module is immediately activated and uses sliding window technology to perform real-time anomaly detection. The data filter can identify and mark potential abnormal data, and through further anomaly confirmation, finally output the filtered high-value data; Step S3: The filtered high-value data enters the event detector. Based on the detection results, the data is divided into different priority queues. Data detected as abnormal enters the high-priority queue, while normal data enters the low-priority queue and is processed separately. Step S4: Cloud-based high-priority data processing and alarm notification.
[0083] Through technical solution design, we achieve the goals of high efficiency, low latency, resource optimization, adaptability, and scalability. This reduces reliance on cloud resources, and innovations in model compression and lightweighting enable the deployment of complex anomaly detection models on resource-constrained edge devices, providing powerful edge intelligence support for IoT applications. Example
[0084] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the real-time data processing system for the Internet of Things provided above is implemented.
[0085] Since the electronic device described in this embodiment is an electronic device used to implement a real-time data processing system for the Internet of Things in the embodiments of this application, based on the real-time data processing system for the Internet of Things described in the embodiments of this application, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiments of this application will not be described in detail here. As long as those skilled in the art implement the electronic device used in the real-time data processing system for the Internet of Things in the embodiments of this application, they are within the scope of protection to be provided by this application.
[0086] The above formulas are all dimensionless and numerical calculations. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.
[0087] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for users of ordinary skill in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A real-time data processing system for the Internet of Things, characterized in that: This includes long-tail devices. Data from these devices is transmitted to the edge layer through communication protocols. The edge gateway in the edge layer is used to receive data. The edge layer also includes data filters. The data filters discard useless data based on filtering rules and retain valuable data. The data filter uses a sliding window and dynamic thresholds for anomaly detection. The sliding window size is adjusted according to the data change rate, the amplitude threshold is dynamically adjusted based on the mean and standard deviation within the window, and the change rate threshold is based on the quantile of the historical change rate queue. Based on the filtering rules, the data is marked as normal data and potential anomalies. Through the anomaly confirmation mechanism, high-value data is finally output. The event detector detects anomalies in valuable data through model rules. If the data is abnormal, it will be divided into a high-priority queue, otherwise it will be placed in a low-priority queue. High-priority data will be transmitted to the cloud for application first, while low-priority data will be processed in a stagnant state. The cloud is used to receive high-priority data and trigger alarm notifications.
2. A real-time data processing system for the Internet of Things according to claim 1, characterized in that: The useless data is discarded based on the filtering rules, and the valuable data is retained, the normal data in the long-tail device is recorded as useless data, and the data with potential abnormalities is recorded as valuable data; Use Snappy LZ4 to compress the valuable data, and then use the event detector on the compressed data; Stalled processing is local caching, waiting for batch upload, or deciding whether to upload immediately based on system load.
3. A real-time data processing system for the Internet of Things according to claim 2, characterized in that: The method for discarding useless data based on filtering rules and retaining valuable data includes: Initialize the sliding window size, initialize the change rate threshold and amplitude threshold, initialize the abnormal confirmation counter to 0, and initialize the historical change rate queue to empty; The edge gateway continuously collects data and arranges it in time series, adding the latest data to the sliding window. If the sliding window is full, the oldest data is removed. Calculate the rate of change between the data at each time in the previous sliding window and adjust the size of the next sliding window according to the rate of change; Calculate the mean and standard deviation of the data in the previous sliding window, and set a dynamic mechanism to adjust the amplitude threshold range in the next sliding window based on the mean and standard deviation; Calculate the rate of change between the current time and the previous time, and add the rate of change to the historical rate of change queue. If the queue exceeds the preset length, remove the rate of change at the oldest time, and calculate a certain percentile of the rate of change in the historical rate of change queue as the new rate of change threshold; The smoothing rule is introduced into the adjustment and update of the change rate threshold and the amplitude threshold to obtain the change rate threshold and amplitude threshold finally used at the current time; For the data at the current time, calculate the average value and average rate of change within the sliding window, check whether the average value is within the amplitude threshold range, and check whether the average rate of change exceeds the rate of change threshold. If it is within the amplitude threshold range and the rate of change does not exceed the threshold, it is considered normal data and the anomaly confirmation counter is reset to 0. If it exceeds the amplitude threshold range or the rate of change exceeds the rate of change threshold, it is marked as a potential anomaly and the anomaly confirmation counter is incremented by 1. An anomaly confirmation mechanism is introduced for data marked as potential anomalies to output high-value data.
4. A real-time data processing system for the Internet of Things according to claim 3, characterized in that: The method for adjusting the change rate threshold and the amplitude threshold and introducing a smoothing rule includes: Get the adjusted rate-of-change threshold or amplitude threshold at the current time and record it as the new threshold. For the rate-of-change threshold or amplitude threshold, maintain a fixed-size queue to store the thresholds at the past N times. Add the new threshold to the queue. If the queue is full, remove the threshold at the oldest time. Calculate the average and median of all thresholds in the queue, perform weighted average of the average and median, record them as smoothing thresholds, and use the smoothing thresholds as the final change rate thresholds and amplitude thresholds used at the current time.
5. The real-time data processing system for the Internet of Things according to claim 4, characterized in that: The method of introducing an anomaly confirmation mechanism for data marked as potentially abnormal includes: For data marked as potential anomalies, it is not immediately judged as abnormal. Instead, it enters the confirmation stage and sets a blank counter. If P consecutive data points are marked as potential anomalies, the counter is increased by 1. If a normal data point appears, the counter is reset to 0. When the counter reaches the preset P value, the data is finally confirmed to be abnormal and recorded as valuable data.
6. A real-time data processing system for the Internet of Things according to claim 5, characterized in that: The event detector detects anomalies of valuable data using model rules and divides the data into a high-priority queue or a low-priority queue. The method includes: Select the appropriate anomaly detection algorithm based on the long-tail device type and data characteristics, and compress and lightweight the anomaly detection algorithm.
7. The real-time data processing system for the Internet of Things according to claim 6, characterized in that: The method for compressing and lightweighting the anomaly detection algorithm includes: Use TensorFlow to train a full-precision model, using a combination of group pruning and iterative pruning to gradually remove unimportant weight groups from the model, and then evaluate it. Selectively quantize some layers while maintaining floating-point precision for the remaining layers. Convert the optimized model to the TensorFlow Lite format, deploy the .tflite model to the target device, and use the TensorFlow Lite interpreter to load and run the model on the device.
8. The real-time data processing system for the Internet of Things according to claim 7, characterized in that: The method of combining group pruning and iterative pruning to gradually remove unimportant weight groups in the model and perform evaluation includes: Select an appropriate grouping strategy based on the model structure and divide the model weights into multiple groups; Preset pruning standards and pruning ratios. The pruning standard is to calculate the importance of each group, preset an important threshold, and remove the lowest q percent of groups whose importance is lower than the important threshold, where q percent is the pruning ratio. According to the selected pruning standard and pruning ratio, remove some weight groups, use a smaller learning rate to fine-tune the pruned model, evaluate the model accuracy on the validation set, adjust the pruning ratio or fine-tune the parameters based on the evaluation results, and repeat the steps until the target accuracy requirement is reached.
9. The real-time data processing system for the Internet of Things according to claim 8, characterized in that: The method of selectively quantizing some layers and maintaining floating-point precision in the remaining layers includes: Evaluate the model layer by layer after iterative pruning and analyze the sensitivity of each layer to quantization. That is, perform quantization sensitivity analysis and weight distribution analysis on each layer. Quantization sensitivity analysis is to quantize each layer separately and then evaluate the degree of accuracy drop of the model on the validation set; weight distribution analysis is to analyze the distribution range of weights of each layer; The sensitivity value is obtained by taking a weighted average of the degree of precision degradation and the distribution range, and a sensitivity threshold is preset. If the sensitivity value is less than the sensitivity threshold, it is recorded as a layer insensitive to quantization and the cover layer is quantized. Otherwise, it is recorded as a layer sensitive to quantization and the floating-point precision is maintained.
10. A real-time data processing method for the Internet of Things, applied to the real-time data processing system for the Internet of Things according to any one of claims 1 to 9, characterized in that: The real-time data processing method for the Internet of Things includes: Step S1: The data of the long-tail device is sent to the edge gateway deployed at the edge layer of the network through the communication protocol; Step S2: After the edge gateway receives the raw data, the built-in data filter module is immediately activated and uses sliding window technology to perform real-time anomaly detection. The data filter can identify and mark potential abnormal data, and through further anomaly confirmation, finally output the filtered high-value data; Step S3: The filtered high-value data enters the event detector. Based on the detection results, the data is divided into different priority queues. Data detected as abnormal enters the high-priority queue, while normal data enters the low-priority queue and is processed separately. Step S4: Cloud-based high-priority data processing and alarm notification.
Citation Information
Cited By
Internet of Things data transmission method and device, electronic equipment and storage medium
CN121509475A
Iot data transmission method and device, electronic equipment and storage medium
CN121509475B
