IoT Data Stream Real-time Processing and Anomaly Detection System

Abstract

This invention provides a real-time data stream processing and anomaly detection system for the Internet of Things (IoT), belonging to the field of IoT data processing technology. The system includes a distributed data access layer, a streaming processing engine, an anomaly detection platform, a correlation analysis module, and an early warning response center. The core innovation lies in its layered streaming data processing architecture, which supports real-time processing of millions of data points per second. The adaptive anomaly detection algorithm based on online learning maintains high accuracy in dynamically changing data distribution environments. The context-aware multidimensional anomaly correlation analysis framework enables rapid inference from single-point anomalies to system-level anomalies. The system achieves an anomaly detection accuracy of 95% and an average detection latency of less than 100ms. It has been successfully deployed in smart city, power IoT, and other scenarios.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) data processing technology, specifically to a real-time IoT data stream processing and anomaly detection system, applicable to smart city infrastructure monitoring, industrial IoT, power IoT, and large-scale sensor networks. Background Technology

[0002] Existing IoT data processing systems typically employ a centralized architecture, uploading raw data to the cloud for batch processing. This approach suffers from the following technical drawbacks: First, high data transmission latency, often taking several seconds or even tens of seconds from device data collection to cloud processing, making it difficult to meet real-time response requirements. Second, high network bandwidth consumption, with simultaneous data uploads from a large number of devices easily causing network congestion. Third, limited cloud processing capacity, leading to a significant increase in system response time during data peaks, impacting service quality. Fourth, a lack of adaptive capabilities, resulting in a significant decrease in detection accuracy when facing dynamic changes in data distribution.

[0003] In terms of anomaly detection, traditional methods mainly rely on static thresholds or offline trained models, which have the following shortcomings: First, static thresholds are difficult to adapt to changes in device status, and are prone to generating a large number of false alarms or missed alarms; Second, offline models cannot be updated in a timely manner, and the detection effect deteriorates rapidly when the data distribution shifts; Third, single-point anomaly detection ignores the correlation between devices and is difficult to detect system-level anomalies; Fourth, it lacks the ability to analyze the root causes of anomalies and cannot quickly locate the source of the problem.

[0004] Therefore, there is an urgent need to develop a technical solution that can efficiently process massive IoT data streams, detect anomalies accurately in real time, has strong adaptive learning capabilities, and perform system-level anomaly analysis, in order to meet the comprehensive requirements of modern IoT applications for real-time performance, accuracy, and scalability. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time data stream processing and anomaly detection system for the Internet of Things, aiming to solve the technical problems existing in the prior art, such as high processing latency, low detection accuracy, weak adaptability, and insufficient system-level anomaly analysis capability.

[0006] This invention achieves real-time ingestion and processing of millions of data points per second by designing a hierarchical streaming data processing architecture; maintains high accuracy in dynamically changing data distribution environments by introducing an adaptive anomaly detection algorithm based on online learning; enables rapid inference from single-point anomalies to system-level anomalies by constructing a context-aware multidimensional anomaly correlation analysis framework; and supports elastic scaling and fault isolation by adopting a microservice architecture, thereby comprehensively improving the real-time monitoring and anomaly early warning capabilities of IoT systems.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: to provide a real-time processing and anomaly detection system for Internet of Things (IoT) data streams, comprising:

[0008] A distributed data access layer is used to collect time-series data streams from multiple IoT devices. The distributed data access layer includes a data acquisition module and a protocol adaptation module. The data acquisition module is used to acquire raw data generated by IoT devices, and the protocol adaptation module is used to convert data from different communication protocols into a unified data format.

[0009] A streaming processing engine, connected to the distributed data access layer, is used to process time-series data streams in real time. The streaming processing engine includes an adaptive buffer scheduling unit and a parallel computing unit. The adaptive buffer scheduling unit is used to dynamically adjust the buffer capacity according to the data inflow rate and system load. The parallel computing unit is used to distribute the data stream to multiple computing nodes for parallel processing.

[0010] An anomaly detection platform, connected to the streaming engine, is used to identify abnormal patterns in time-series data. The anomaly detection platform includes a feature extraction unit, an online learning unit, and an anomaly determination unit. The feature extraction unit is used to extract multi-dimensional feature vectors from the time-series data. The online learning unit is used to update the detection model parameters in real time based on the data stream. The anomaly determination unit is used to determine the abnormal state of the data points based on the model output.

[0011] The correlation analysis module is connected to the anomaly detection platform and is used to analyze the correlation between abnormal data. The correlation analysis module includes a context construction unit and a correlation reasoning unit. The context construction unit is used to obtain the device status and environmental information corresponding to the abnormal data, and the correlation reasoning unit is used to infer the anomaly propagation path based on the context information.

[0012] The early warning response center is connected to the anomaly detection platform and the correlation analysis module, respectively, and is used to generate early warning information and trigger response strategies. The early warning response center includes a risk assessment unit and a strategy execution unit. The risk assessment unit is used to quantify the impact range and severity of abnormal events, and the strategy execution unit is used to execute corresponding disposal measures according to the risk level.

[0013] The distributed data access layer is responsible for collecting time-series data streams from multiple IoT devices, achieving unified access to heterogeneous data sources through protocol adaptation. The streaming processing engine employs an adaptive buffer scheduling mechanism, dynamically adjusting resource allocation based on real-time load, and distributing data streams to multiple nodes for processing through parallel computing units, significantly improving processing throughput. The anomaly detection platform continuously updates model parameters using online learning, enabling detection capabilities to adaptively adjust to changes in data distribution. The correlation analysis module constructs a correlation graph between devices, inferring anomaly propagation paths through contextual information to achieve system-level anomaly identification. The early warning and response center triggers corresponding handling strategies based on risk assessment results, ensuring timely processing of abnormal events.

[0014] Compared with the prior art, the present invention has the following significant advantages and beneficial effects:

[0015] First, processing performance is significantly improved. Through a hierarchical streaming data processing architecture and an adaptive buffer scheduling mechanism, the system supports real-time processing of 1 million to 2 million data points per second, which is more than 50 times higher than the throughput of traditional batch processing systems, and the average processing latency is reduced to less than 100ms, meeting the strict latency requirements of real-time monitoring.

[0016] Second, the detection accuracy is significantly improved. By adopting an adaptive anomaly detection algorithm based on online learning, the system can continuously track changes in data distribution, achieving an anomaly detection accuracy of over 95% and a false alarm rate controlled within 3%. Compared with the static threshold method, the accuracy is improved by 25 percentage points, effectively reducing the workload of maintenance personnel.

[0017] Third, enhanced adaptability. The online learning mechanism enables the model to automatically adapt to changes in device status and environmental fluctuations, maintaining stable detection performance even when data distributions drift significantly. The model's lifespan is extended from several weeks in traditional methods to several months, greatly reducing model maintenance costs.

[0018] Fourth, the system excels in system-level anomaly identification. Through context-aware multidimensional anomaly correlation analysis, the system can identify systemic problems behind single-device anomalies, achieving an anomaly root cause localization accuracy rate of over 90% and reducing the average localization time to less than 30 seconds, significantly improving fault handling efficiency.

[0019] Fifth, it boasts excellent scalability and reliability. Adopting a microservice architecture, each functional module is deployed independently, supporting horizontal scaling to thousands of nodes. A single point of failure does not affect the overall service, and system availability reaches over 99.9%, meeting the needs of large-scale commercial deployments.

[0020] In summary, the IoT data stream real-time processing and anomaly detection system provided by this invention has achieved significant breakthroughs in processing performance, detection accuracy, adaptability, system-level analysis capabilities, and scalability. It provides an efficient and reliable technical means for real-time monitoring and intelligent operation and maintenance of IoT systems, and has significant application value and promising prospects for promotion. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall architecture of the IoT data stream real-time processing and anomaly detection system of the present invention, wherein label 1 represents the distributed data access layer, label 2 represents the streaming processing engine, label 3 represents the anomaly detection platform, label 4 represents the correlation analysis module, and label 5 represents the early warning response center;

[0022] Figure 2 This is a schematic diagram of the distributed data access layer of the present invention;

[0023] Figure 3 This is a schematic diagram of the data processing flow of the streaming engine of this invention;

[0024] Figure 4 This is a schematic diagram illustrating the working principle of the anomaly detection platform of the present invention;

[0025] Figure 5 This is a schematic diagram of the model update process of the online learning algorithm of this invention. Detailed Implementation

[0026] Please refer to Figures 1-5 The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0027] Reference Figure 1 The IoT data stream real-time processing and anomaly detection system of the present invention includes a distributed data access layer 1, a streaming processing engine 2, an anomaly detection platform 3, a correlation analysis module 4, and an early warning response center 5. The entire system adopts a microservice architecture design, and the modules communicate through standard interfaces, supporting independent deployment and elastic expansion.

[0028] Reference Figure 2 The distributed data access layer 1 is the entry point for the system's data flow, responsible for collecting time-series data from massive numbers of IoT devices and performing preliminary processing. This layer adopts a distributed deployment approach, setting up multiple access nodes in geographically dispersed areas. Each node works independently, avoiding single-point failures that could lead to overall service unavailability.

[0029] The distributed data access layer 1 includes a data acquisition module and a protocol adaptation module. The data acquisition module establishes connections with IoT devices through various communication protocols, supporting mainstream IoT communication protocols such as MQTT, CoAP, HTTP, and WebSocket, as well as industrial protocols commonly used in industrial settings such as Modbus and OPC UA. For devices using wireless communication, the system supports low-power wide-area network protocols such as LoRaWAN, NB-IoT, and Zigbee, ensuring smooth access for all types of devices.

[0030] The data acquisition module uses a push-pull approach to acquire device data. For devices that periodically report data, a push mode is used to receive data; for devices that require on-demand querying, a pull mode is used for active polling. The system dynamically adjusts the acquisition frequency based on device characteristics and data importance, using high-frequency acquisition for important devices and low-frequency acquisition for non-critical devices, thus reducing network and storage overhead while ensuring data integrity.

[0031] The protocol adaptation module is responsible for converting data from different protocol formats into a unified data format within the system. Internally, the system uses JSON format to represent data, with each data record containing fields such as device identifier, timestamp, data type, value, and unit. During protocol adaptation, the system performs data integrity checks, including timestamp validity checks, value range checks, and data type matching checks. Data that does not conform to the specifications is marked and logged for subsequent analysis.

[0032] The distributed data access layer 1 is also responsible for preliminary data filtering and aggregation. For high-frequency continuous data, the system supports downsampling at the edge, calculating statistics such as mean, maximum, and minimum values ​​through a sliding window to reduce the amount of data transmitted to the central node. Preferably, the sliding window size is set to 1 to 10 seconds, which can be dynamically adjusted according to network bandwidth and computing resources. For discrete event data, the system retains the original data without aggregation.

[0033] In one embodiment, the distributed data access layer 1 is deployed in multiple areas of a smart city, with one access node in each area. Each node connects 500 to 2000 IoT devices. Nodes synchronize data via message queues to ensure that if any node fails, other nodes can take over the devices it is responsible for, guaranteeing high system availability. Each access node pushes pre-processed data streams to the streaming processing engine 2. The data push frequency is adaptively adjusted according to the system load: real-time push is used under low load, while batch push is used under high load, with a batch size of 100 to 1000 records.

[0034] Reference Figure 3The streaming engine 2 is the core computing module of the system, responsible for real-time processing and analysis of high-speed data streams. This engine employs a distributed streaming computing framework based on Apache Flink, supporting event-time semantics and exact-once processing guarantees to ensure the accuracy and reliability of data processing results.

[0035] The streaming engine 2 includes an adaptive buffer scheduling unit and a parallel computing unit. The adaptive buffer scheduling unit, a core innovation of this invention, is used to dynamically adjust the system buffer size to adapt to fluctuating data traffic. This unit includes a traffic monitoring subunit, a load assessment subunit, and a buffer adjustment subunit.

[0036] The flow monitoring subunit collects real-time statistics on the inflow rate and calculates the average and peak inflow rates for the most recent period using a sliding time window. The preferred time window size is 10 to 60 seconds, which can be dynamically adjusted based on data flow fluctuations. When the inflow rate within the window exceeds the system's processing capacity, the buffer is expanded; when the inflow rate within the window remains below 50% of the system's processing capacity, the buffer is contracted, releasing idle resources for use by other modules.

[0037] The load assessment subunit periodically collects resource metrics such as CPU utilization, memory utilization, network I / O, and disk I / O from each computing node, and scores the overall load of the computing nodes. The score calculation uses a weighted approach: CPU utilization has a weight of 0.4, memory utilization has a weight of 0.3, network I / O has a weight of 0.2, and disk I / O has a weight of 0.1. When the overall load of a node exceeds 80%, the node is marked as high-load, and the system automatically distributes newly arriving data to other low-load nodes for processing, achieving load balancing.

[0038] The buffer adjustment subunit dynamically adjusts the buffer capacity based on traffic monitoring and load assessment results. The buffer capacity adjustment algorithm employs the following strategy: if the ratio of the current inflow rate to processing capacity is greater than 1.5, the buffer capacity is increased to twice the current capacity; if this ratio remains less than 0.5 for five consecutive monitoring periods, the buffer capacity is reduced to 0.5 times the current capacity. The buffer capacity has upper and lower limits: the upper limit is 30% of the total system memory, and the lower limit is 5% of the total system memory, preventing over-allocation or under-allocation of resources.

[0039] The parallel computing unit is responsible for distributing the data stream to multiple computing nodes for parallel processing. Data distribution employs a keyword-based partitioning strategy, routing data generated by the same device to the same computing node to ensure the continuity of processing logic for a single device. Each computing node independently processes the data substream assigned to it, performing operations such as data cleaning, feature engineering, and preliminary analysis.

[0040] Data cleaning operations include outlier filtering, missing value imputation, and noise smoothing. For outliers exceeding a reasonable range, the system uses the 3σ principle for filtering, marking data points whose values ​​deviate from the mean by more than three standard deviations as outliers. For missing values, the system employs different imputation strategies based on data type: linear interpolation or nearest neighbor imputation is used for continuous data, while mode imputation is used for discrete data. For noisy data, the system uses median filtering or Kalman filtering for smoothing, with the filtering window size adaptively determined based on the signal frequency characteristics, preferably between 3 and 15 data points.

[0041] Feature engineering extracts features from raw time-series data that aid in anomaly detection. The extracted features fall into three main categories: time-domain features, frequency-domain features, and spatiotemporal features. Time-domain features include statistics such as mean, variance, maximum, minimum, peak-to-peak value, skewness, kurtosis, and zero-crossing rate. Frequency-domain features are obtained through Fast Fourier Transform (FFT) and include indices such as dominant frequency, spectral energy, and spectral entropy. Spatiotemporal features are obtained by analyzing the correlation between data from adjacent devices and include cross-correlation coefficients, time delays, and spatial gradients.

[0042] In one embodiment, the streaming engine 2 is deployed on a cluster of 32 compute nodes, each configured with an 8-core CPU, 32GB of memory, and 1TB of SSD storage. The cluster's total processing capacity reaches 2 million data points per second, with an average processing latency of 80ms per data point. When the data inflow rate suddenly increases to 3 million data points per second, the adaptive buffer scheduling unit automatically expands the buffer capacity from 2GB to 4GB and simultaneously starts 8 additional standby compute nodes, increasing the system's processing capacity to 4 million data points per second, successfully handling the traffic surge.

[0043] Reference Figure 4 The anomaly detection platform 3 is responsible for identifying abnormal patterns from the processed data stream and is a key component of the system's intelligence. This platform includes a feature extraction unit, an online learning unit, and an anomaly determination unit.

[0044] The feature extraction unit receives preprocessed data output from the streaming engine 2 and further extracts high-level features suitable for anomaly detection. This unit includes a temporal feature extraction subunit, a frequency domain feature extraction subunit, and a spatiotemporal feature extraction subunit. The temporal feature extraction subunit calculates the statistical characteristics of the data within a sliding time window. The window size is determined based on the data sampling frequency; for data with a sampling frequency of 1Hz, the preferred time window is 60s to 300s; for data with a sampling frequency of 10Hz, the preferred time window is 10s to 60s. The frequency domain feature extraction subunit performs a Fast Fourier Transform on the data within the time window to obtain the spectral distribution and calculates the energy proportion of each frequency band. The spatiotemporal feature extraction subunit analyzes data from multiple related devices at the same time, calculating the correlation matrix and spatial distribution pattern between the devices.

[0045] The online learning unit is one of the core innovations of this invention, employing an incremental learning approach to continuously update the anomaly detection model. This unit is based on an autoencoder neural network architecture, comprising an encoder and a decoder. The encoder maps high-dimensional feature vectors to a low-dimensional latent space, while the decoder attempts to reconstruct the original feature vectors from the latent representation. Reconstruction errors are smaller for normal data and larger for abnormal data; anomaly detection is achieved by setting a reconstruction error threshold.

[0046] The online learning unit comprises a sample acquisition subunit, a gradient calculation subunit, and a parameter update subunit. The sample acquisition subunit extracts training samples from the data stream. The sampling strategy employs a time decay mechanism, prioritizing the acquisition of the most recent data and assigning lower weights to historical data. The sampling window size ranges from 1000 to 10000 data points, and the sampling interval is dynamically adjusted according to the data stream rate; the sampling interval is larger in high-speed data stream scenarios and smaller in low-speed data stream scenarios.

[0047] The gradient calculation subunit calculates the gradient of the model parameters based on the collected training samples. The gradient calculation employs the backpropagation algorithm, and the loss function is the mean squared error of the reconstruction error. In one embodiment of the invention, the autoencoder neural network uses the following innovative adaptive loss function:

[0048] ,

[0049] in, This is the total loss function value. The number of training samples. For the first The original feature vector of each sample, For the first The reconstructed feature vector of each sample, For the first The weight coefficients of each sample are used to assign different importance to samples at different times. This represents the square of the Euclidean distance. The regularization coefficient is . This represents the total number of model parameters. For the first Each model parameter This represents the absolute value. In a preferred embodiment, A value between 0.001 and 0.01 can achieve a good balance between fitting ability and generalization ability.

[0050] Weighting coefficient Calculated using the time decay function:

[0051] ,

[0052] in, For the first The weight coefficients of each sample, The base of the natural logarithm, The time decay rate, For the first The time interval between each sample and the current time. In a preferred embodiment, The value ranges from 0.001 to 0.01 hours. This causes the weight of samples from 1 hour ago to decrease to 0.990 to 0.999 times the current weight, and the weight of samples from 24 hours ago to decrease to 0.787 to 0.976 times the current weight, thus enabling priority learning of recent data while retaining the reference value of historical data.

[0053] The parameter update subunit updates the model weights based on the calculated gradient values. The parameter update uses the Adam optimizer, which combines the advantages of the momentum method and the adaptive learning rate method, offering fast convergence and insensitivity to hyperparameters. The parameter update formula is as follows:

[0054] ,

[0055] in, For the first The model parameter vector at the next iteration For the first The model parameter vector at the next iteration For learning rate, For the first The bias correction value of the first moment estimate of the gradient in the next iteration. For the first The bias correction value of the second moment estimate of the gradient in the next iteration. To prevent the use of tiny constants that divide by zero. In a preferred embodiment, The initial value was set to 0.001, and it gradually decreased to 0.0001 as training progressed; Set as This ensures the stability of numerical calculations.

[0056] The formulas for calculating the first-order moment estimate and the second-order moment estimate are as follows:

[0057] ,

[0058] ,

[0059] in, For the first First-order moment estimation at the next iteration For the first Second-order moment estimation at the next iteration For the first The gradient vector at the next iteration and For the exponential decay rate hyperparameter, This represents the element-wise square of the gradient vector. In a preferred embodiment, The value is 0.9. The value is 0.999, and these values ​​have been widely verified to achieve good results in most tasks.

[0060] The formulas for calculating the first-order moment estimate and the second-order moment estimate after bias correction are as follows:

[0061] ,

[0062] ,

[0063] in, This is the first-order moment estimate after bias correction. This is the second-order moment estimate after bias correction. This represents the current iteration number. Bias correction eliminates the problem of the initial moment estimation being biased towards zero, accelerating convergence in the early stages of model training.

[0064] The anomaly detection unit determines whether a data point is abnormal based on the reconstruction error of the autoencoder. The reconstruction error is calculated using the following formula:

[0065] ,

[0066] in, For the first Reconstruction error of each data point The original feature vector, To reconstruct the feature vector, This represents the square of the Euclidean distance. When the reconstruction error exceeds the dynamic threshold, the data point is considered an anomaly.

[0067] Dynamic threshold The following adaptive calculation method is adopted:

[0068] ,

[0069] in, This is the threshold for anomaly detection. This represents the average reconstruction error of normal data within the most recent time window. This represents the standard deviation of the reconstruction error of normal data within the most recent time window. This is the sensitivity coefficient. In a preferred embodiment, The value ranges from 2 to 3. A higher value indicates higher system tolerance and a lower false positive rate, but a higher false negative rate; a lower value indicates higher system sensitivity and a lower false positive rate, but a higher false positive rate. The system dynamically adjusts the value based on application scenario requirements. Values ​​in critical infrastructure monitoring scenarios Take the smaller value in typical monitoring scenarios. Take the larger value.

[0070] The anomaly detection unit also employs a multi-level detection strategy to improve detection accuracy. This strategy includes three levels: point anomaly detection, pattern anomaly detection, and ensemble anomaly detection. Point anomaly detection targets individual data points and determines anomalies through reconstruction errors; pattern anomaly detection targets time series and determines anomalies by analyzing changes in the sequence's morphology, such as sudden changes, trend reversals, and the disappearance of periodicity; ensemble anomaly detection targets combinations of multiple variables and determines anomalies by analyzing changes in the correlation between variables, such as variables that were originally strongly correlated becoming weakly correlated, or variables that were originally independent becoming strongly correlated.

[0071] In one embodiment, the autoencoder neural network deployed on the anomaly detection platform 3 comprises an input layer, three hidden layers, and an output layer. The input layer has 128 neurons, corresponding to the extracted 128-dimensional feature vector; the first hidden layer has 64 neurons, the second hidden layer has 32 neurons, and the third hidden layer has 16 neurons, forming the encoder; the decoder structure is symmetrical to the encoder, with the fourth hidden layer having 32 neurons, the fifth hidden layer having 64 neurons, and the output layer having 128 neurons. All hidden layers use the ReLU activation function, and the output layer uses a linear activation function. Model training employs mini-batch gradient descent with a batch size of 256, performing parameter updates every 10,000 new data points. Experimental results show that, under this configuration, the anomaly detection accuracy reaches 95.3%, the false positive rate is 2.8%, the false negative rate is 1.9%, and the average detection latency is 95ms, meeting the requirements for real-time monitoring.

[0072] The correlation analysis module 4 is responsible for analyzing the correlations between detected abnormal data, inferring the anomaly propagation path and scope of impact, and enabling the leap from single-point anomalies to system-level anomalies. This module includes a context building unit and a correlation reasoning unit.

[0073] The context building unit is used to collect relevant contextual information when abnormal data occurs. This unit includes a status query subunit, an environmental awareness subunit, and a historical association subunit. The status query subunit obtains the current operating status of the device corresponding to the abnormal data from the device management system, including information such as device model, firmware version, runtime, and recent operation records. The environmental awareness subunit obtains environmental parameters at the time of the abnormality, such as temperature, humidity, air pressure, light intensity, and electromagnetic interference intensity. These environmental factors may affect device behavior or cause data abnormalities. The historical association subunit queries the historical database to retrieve historical abnormal records with similar patterns to the current abnormality, analyzes the root causes and handling measures of historical abnormalities, and provides a reference for the current abnormality analysis.

[0074] The association reasoning unit uses a graph structure to represent the topological relationships and functional dependencies between devices. The system constructs a device association graph, where each node represents an IoT device, and the edges between nodes represent the association relationships between devices. Association relationships include several types: physical connections, such as the output of device A being directly connected to the input of device B; functional dependencies, such as the normal operation of device A depending on the signals or resources provided by device B; spatial proximity, such as device A and device B being in the same physical location and potentially affected by the same environment; and logical associations, such as device A and device B performing related functional tasks, where the failure of one task may affect the other.

[0075] Association reasoning employs a graph traversal algorithm to determine the scope of anomaly impact. When an anomaly is detected in a device, the system starts from the graph node corresponding to that device and propagates along the association edges to adjacent nodes, calculating the probability that adjacent nodes are affected. The impact probability calculation comprehensively considers factors such as association strength, time delay, and anomaly type. Association strength is obtained through historical data statistics, representing the historical frequency of another device experiencing an anomaly when one device experiences one; time delay represents the average time interval between an anomaly propagating from one device to another; and anomaly type indicates that different types of anomalies have different propagation characteristics.

[0076] In one embodiment of the present invention, the probability of anomaly impact The calculation formula is:

[0077] ,

[0078] in, In order to be in Time device Abnormal effects on equipment The probability, For equipment and equipment The strength of the correlation between them The base of the natural logarithm, For the current moment, For equipment When an anomaly occurs, For abnormalities from the device Transmitted to device Average latency, The standard deviation of the delay time. To account for the correction coefficients for anomaly types. This formula represents the probability of influence over the delay time. It reaches its peak near the point of origin and then decays as it deviates from the peak over time.

[0079] Correlation strength Calculations based on historical data:

[0080] ,

[0081] in, For equipment and equipment The strength of the correlation between them For historical equipment After the equipment malfunctions There were also abnormal occurrences. For equipment The total number of anomalies. The correlation strength ranges from 0 to 1, with a higher value indicating a stronger correlation.

[0082] The system traverses the device association graph using a breadth-first search algorithm to calculate all devices reachable from the source device of the anomaly and their probability of being affected. Devices with an impact probability exceeding a preset threshold are included in the anomaly's impact range, triggering corresponding early warnings and response measures. The preset threshold is preferably between 0.3 and 0.5, and can be dynamically adjusted according to the system's fault tolerance and early warning sensitivity requirements.

[0083] In one embodiment, in a smart city water supply system, a water pressure sensor at a pumping station detected abnormally low pressure. The correlation analysis module 4, through an equipment correlation diagram, discovered that the pumping station supplies water to six downstream distribution stations. Analysis and calculation showed that the probability of each distribution station being affected was 0.85, 0.78, 0.72, 0.65, 0.51, and 0.42, respectively. The system included the five distribution stations with an impact probability exceeding 0.5 in the abnormal impact range and sent early warning signals to these stations, notifying them to prepare to switch to a backup water source or adjust their water supply strategy. This early warning measure reduced the average water supply interruption time for residents in the affected area from 45 minutes to 12 minutes, significantly improving the reliability of the water supply system and user satisfaction.

[0084] Reference Figure 1The Early Warning Response Center 5 is the system's output module, responsible for generating early warning information based on anomaly detection and correlation analysis results, and triggering corresponding response strategies. This center includes a risk assessment unit and a strategy execution unit.

[0085] The risk assessment unit quantifies the risk of detected abnormal events. Risk assessment comprehensively considers multiple dimensions, including anomaly type, severity, number of affected devices, and duration. Anomalies are categorized into four levels based on their impact on system functionality: Level 1 is a functional failure anomaly, causing complete loss of device or system functionality; Level 2 is a performance degradation anomaly, where device or system functionality is partially impaired but still operational; Level 3 is a warning anomaly, where device or system exhibits abnormal signs but has not yet affected functionality; and Level 4 is a suggestive anomaly, where device or system parameters deviate from the normal range but the impact is minor.

[0086] The severity of the anomaly is quantified by the reconstruction error or deviation; the greater the deviation, the higher the severity. The number of affected devices is calculated using the correlation analysis module 4; the wider the impact, the greater the risk. The duration of the anomaly is determined by the duration of continuous monitoring of the anomaly state; the longer the duration, the greater the risk.

[0087] Risk Score The calculation formula is:

[0088] ,

[0089] in, To score risk, The exception type level is (1 to 4). The normalized value for the severity of the anomaly (from 0 to 1). To influence the normalized value of the number of devices (from 0 to 1). The normalized value for the duration of the anomaly (0 to 1). , , , These are the weighting coefficients for each dimension. In a preferred embodiment, The value is 0.4. The value is 0.3. The value is 0.2. A value of 0.1 reflects the dominant role of anomaly type in risk assessment. Risk Score The value range is 0 to 1. The system classifies abnormal events into four levels based on the risk score: low risk (0 to 0.3), medium risk (0.3 to 0.6), high risk (0.6 to 0.8), and extremely high risk (0.8 to 1.0).

[0090] The strategy execution unit triggers corresponding handling strategies based on the risk level. This unit includes an alarm push subunit, a device control subunit, and a log recording subunit. The alarm push subunit is responsible for sending early warning notifications to management personnel through various channels, including SMS, email, mobile application push notifications, and voice calls. For low-risk events, the system only sends email notifications; for medium-risk events, the system sends SMS and mobile application push notifications; for high-risk and extremely high-risk events, the system sends SMS and push notifications while simultaneously making a voice call to ensure that management personnel are informed of the anomaly as soon as possible.

[0091] The equipment control subunit executes automated control operations on malfunctioning equipment. For performance degradation anomalies, the system attempts to restart the equipment or adjust its parameters to restore normal operation. For functional failure anomalies, the system isolates the faulty equipment from the system and switches to backup equipment or a backup solution to prevent the fault from spreading and affecting the overall system. Equipment control operations employ a whitelist mechanism, allowing automatic control only on pre-authorized equipment. Control operations on unauthorized equipment require manual confirmation before execution to avoid more serious consequences from misoperation.

[0092] The logging subunit persistently stores complete information about abnormal events, including the time of occurrence, type, device, parameter values, context, risk score, handling measures, and results. Log data provides the foundation for subsequent root cause analysis, trend prediction, and system optimization. The system supports multi-dimensional querying and statistical analysis of historical logs, helping operations personnel identify potential systemic problems and areas for improvement.

[0093] In one embodiment, an industrial IoT system deployed the anomaly detection system of this invention to monitor the operating status of 200 production devices. During three months of continuous operation, the system detected 1253 anomaly events, including 891 low-risk events, 287 medium-risk events, 68 high-risk events, and 7 extremely high-risk events. The system automatically handled low-risk and medium-risk events, achieving a 92% success rate in restoring normal operation through device restarts or parameter adjustments. For high-risk and extremely high-risk events, the system promptly notified maintenance personnel for intervention, reducing the average response time from 25 minutes to 8 minutes. The system effectively reduced production downtime caused by equipment failures, resulting in significant economic benefits.

[0094] Reference Figure 5 The IoT data stream real-time processing and anomaly detection system of the present invention adopts a cloud-edge collaborative deployment architecture, deploying different functional modules on the edge side and the cloud side respectively, giving full play to the low latency advantage of edge computing and the strong computing power advantage of cloud computing.

[0095] The edge side deploys some functions of the distributed data access layer 1 and streaming processing engine 2, including data acquisition, protocol adaptation, preliminary cleaning, and simple feature extraction. Edge-side devices are typically industrial PCs or embedded gateways, configured with 4- to 8-core CPUs, 8GB to 16GB of RAM, and 256GB to 512GB of storage. Deploying edge-side devices close to the data source allows for local data acquisition and preliminary processing, reducing data transmission volume and latency.

[0096] The cloud-based deployment includes the core functions of the streaming processing engine 2, the anomaly detection platform 3, the correlation analysis module 4, and the early warning and response center 5. The cloud-based deployment employs a containerized approach, with all modules packaged as Docker container images and deployed on a Kubernetes cluster. Kubernetes provides automated container orchestration, load balancing, fault recovery, and elastic scaling capabilities, ensuring high availability and high performance. The size of the cloud cluster is dynamically adjusted based on the number of monitored devices and data traffic. For a medium-sized system monitoring 10,000 devices, the cloud cluster typically contains 32 to 64 compute nodes.

[0097] Data transmission between the edge and cloud sides is achieved through message queues. The system uses Apache Kafka as the message queue middleware, which provides high throughput, low latency, persistence, and fault tolerance in message delivery. Edge devices publish pre-processed data to Kafka topics, and cloud-side modules subscribe to the corresponding topics to retrieve data. The Kafka cluster is deployed in the cloud, containing 3 to 5 broker nodes, with a data redundancy factor set to 3 to ensure data reliability.

[0098] The system adopts a microservice architecture, with each functional module deployed as an independent microservice. Modules communicate with each other via REST API or gRPC interfaces. Microservice architecture offers several advantages: First, module decoupling means that a failure in one module does not affect the operation of other modules; second, independent deployment allows for individual module updates without redeploying the entire system; third, horizontal scaling allows for independent scaling of the number of instances of a module based on load; and fourth, flexible technology stacks allow different modules to be implemented using the most suitable technologies.

[0099] The system provides a unified management interface, allowing maintenance personnel to access the system via a web interface or mobile application to view information such as device status, anomaly alarms, and system performance metrics. The management interface adopts a responsive design, adapting to display devices of different sizes. The system also provides a RESTful API for third-party system integration, supporting data exchange and functional linkage with existing enterprise IT systems such as ERP, MES, and SCADA.

[0100] The system of this invention has been successfully deployed and applied in multiple real-world scenarios, achieving remarkable results.

[0101] Smart City Infrastructure Monitoring: A first-tier city has deployed over 50,000 sensors on urban roads, bridges, and tunnels to monitor parameters such as structural health, traffic flow, and environmental quality. By adopting the system of this invention, all-weather real-time monitoring of urban infrastructure has been achieved. The system successfully detected minor deformations in three bridge structures, providing early warnings and arranging reinforcement and maintenance to prevent potential safety accidents. The system also detected 12 instances of traffic congestion and 3 traffic accidents through abnormal traffic flow detection, with an average warning lead time of 8 minutes, saving valuable time for traffic management.

[0102] Power Internet of Things: A provincial power grid company has deployed over 200,000 smart sensors in substations and transmission lines to monitor parameters such as voltage, current, temperature, and vibration. After adopting the system of this invention, precise monitoring and fault early warning of power grid equipment were achieved. The system successfully predicted 7 transformer overheating faults and 15 insulator flashover faults, achieving a fault prediction accuracy rate of 93% and an average early warning time of 2.5 hours. Through proactive maintenance, economic losses caused by power outages were avoided, resulting in economic benefits exceeding 20 million yuan.

[0103] Large-scale sensor network: An environmental monitoring agency has deployed 100,000 air quality monitoring stations nationwide, collecting real-time data on pollutant concentrations such as PM2.5, PM10, SO2, NO2, O3, and CO. Using the system of this invention, real-time processing and anomaly identification of massive amounts of environmental data were achieved. The system detected 327 anomaly events related to pollutant concentrations, of which 93% were genuine pollution emission accidents or extreme weather events, and 7% were sensor malfunctions. Through anomaly correlation analysis, the system successfully traced the emission sources of 23 pollution events, providing strong support for environmental law enforcement.

[0104] Industrial Production Line Monitoring: An automobile manufacturing company deployed over 5,000 IoT devices on its production line to monitor the operational status of key equipment such as robotic arms, conveyor belts, welding equipment, and painting equipment. After adopting the system of this invention, the accuracy rate of equipment fault prediction reached 96%, with an average prediction lead time of 45 minutes. This allows maintenance personnel to complete equipment repairs during production breaks, reducing downtime. The average mean time between failures (MTBF) increased from 72 hours to 168 hours, production efficiency improved by 15%, and annual maintenance costs decreased by 28%.

[0105] These application cases fully demonstrate the superior performance of the technical solution of this invention in terms of real-time performance, accuracy, adaptability, and scalability, providing an efficient and reliable technical means for the intelligent operation and maintenance of IoT systems, and have broad application prospects and important economic and social value.

[0106] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A real-time data stream processing and anomaly detection system for the Internet of Things (IoT), characterized in that, include: A distributed data access layer is used to collect time-series data streams from multiple IoT devices. The distributed data access layer includes a data acquisition module and a protocol adaptation module. The data acquisition module is used to acquire raw data generated by IoT devices, and the protocol adaptation module is used to convert data from different communication protocols into a unified data format. A streaming processing engine, connected to the distributed data access layer, is used to process time-series data streams in real time. The streaming processing engine includes an adaptive buffer scheduling unit and a parallel computing unit. The adaptive buffer scheduling unit includes a traffic monitoring subunit, a load assessment subunit, and a buffer adjustment subunit. The traffic monitoring subunit is used to collect real-time statistics on the data inflow rate. The load assessment subunit is used to obtain the resource usage of each computing node. The buffer adjustment subunit is used to dynamically adjust the buffer size according to the data inflow rate and the resource usage. The parallel computing unit is used to distribute the data stream to multiple computing nodes for parallel processing. An anomaly detection platform, connected to the streaming engine, is used to identify abnormal patterns in time-series data. The platform includes a feature extraction unit, an online learning unit, and an anomaly determination unit. The feature extraction unit extracts multi-dimensional feature vectors from the time-series data. It includes a time-domain feature extraction subunit, a frequency-domain feature extraction subunit, and a spatiotemporal feature extraction subunit. The time-domain feature extraction subunit calculates the statistical properties of the time-series data, the frequency-domain feature extraction subunit obtains spectral features through transformation, and the spatiotemporal feature extraction subunit fuses time and spatial dimension information. The online learning unit is based on an autoencoder neural network architecture and includes an encoder and a decoder. The encoder maps high-dimensional feature vectors to a low-dimensional latent space, and the decoder reconstructs the original feature vectors from the latent representation. The online learning unit updates model parameters using incremental learning and includes a sample acquisition subunit, a gradient calculation subunit, and a parameter update subunit. The sample acquisition subunit extracts training samples from the data stream, and the gradient calculation subunit calculates the gradient values ​​of the model parameters based on the training samples. The gradient calculation subunit uses an adaptive loss function, which is: in, This is the total loss function value. The number of training samples. For the first The original feature vector of each sample, For the first The reconstructed feature vector of each sample, For the first The weight coefficients of each sample, This represents the square of the Euclidean distance. The regularization coefficient is . This represents the total number of model parameters. For the first Each model parameter Represents absolute value; The weighting coefficient Calculated using the time decay function: in, For the first The weight coefficients of each sample, The base of the natural logarithm, The time decay rate, For the first The time interval between each sample and the current time; The parameter update subunit is used to adjust the model weights according to the gradient value; the anomaly detection unit determines the anomaly state of the data points based on the reconstruction error, and the reconstruction error is calculated using the following formula: in, For the first Reconstruction error of each data point The original feature vector, To reconstruct the feature vector, This represents the square of the Euclidean distance; when the reconstruction error exceeds a dynamic threshold, the data point is considered abnormal. Adaptive calculation method is adopted: in, This is the threshold for anomaly detection. This represents the average reconstruction error of normal data within the most recent time window. This represents the standard deviation of the reconstruction error of normal data within the most recent time window. Sensitivity coefficient; The anomaly detection unit adopts a multi-level detection strategy, including a point anomaly detection subunit, a pattern anomaly detection subunit, and a set anomaly detection subunit. The point anomaly detection subunit is used to identify deviations of a single data point, the pattern anomaly detection subunit is used to identify anomaly patterns in time series, and the set anomaly detection subunit is used to identify anomaly combinations of multiple variables. The correlation analysis module, connected to the anomaly detection platform, is used to analyze the correlation relationships between abnormal data. The correlation analysis module includes a context construction unit and a correlation reasoning unit. The context construction unit includes a status query subunit, an environment perception subunit, and a historical correlation subunit. The status query subunit is used to obtain the operating status of the device corresponding to the abnormal data; the environment perception subunit is used to obtain the environmental parameters at the time of the anomaly; and the historical correlation subunit is used to query historical records of similar anomalies. The correlation reasoning unit uses a graph structure to represent the topological relationships between devices, traverses the device correlation graph using a breadth-first search algorithm to determine the scope of the anomaly's impact, and calculates the probability of the anomaly's impact. : in, In order to be in Time device Abnormal effects on equipment The probability, For equipment and equipment The strength of the correlation between them Let be the base of the natural logarithm. For the current moment, For equipment When an anomaly occurs, For abnormalities from the device Transmitted to device Average latency, The standard deviation of the delay time. Correction coefficients to account for anomaly types; The correlation strength Calculations based on historical data: in, For equipment and equipment The strength of the correlation between them For historical equipment After the equipment malfunctions There were also abnormal occurrences. For equipment Total number of anomalies; The early warning response center, connected to both the anomaly detection platform and the correlation analysis module, is used to generate early warning information and trigger response strategies. The early warning response center includes a risk assessment unit and a strategy execution unit. The risk assessment unit calculates a risk score based on the anomaly type, the number of affected devices, and the duration. : in, To score risk, For exception type level, This is a normalized value representing the severity of the anomaly. To influence the normalized value of the number of devices, This is the normalized value for the duration of the anomaly. , , , These are the weighting coefficients for each dimension; the strategy execution unit is used to execute corresponding disposal measures according to the risk level.

2. The IoT data stream real-time processing and anomaly detection system according to claim 1, characterized in that, The buffer capacity adjustment strategy of the buffer adjustment subunit is as follows: if the ratio of the current inflow rate to the processing capacity is greater than 1.5, the buffer capacity will be increased to twice the current capacity; if the ratio is less than 0.5 for 5 consecutive monitoring cycles, the buffer capacity will be reduced to 0.5 times the current capacity; the buffer capacity has upper and lower limits, with the upper limit being 30% of the total system memory and the lower limit being 5% of the total system memory.

3. The IoT data stream real-time processing and anomaly detection system according to claim 1, characterized in that, The strategy execution unit includes: an alarm push subunit for sending early warning notifications to administrators; a device control subunit for performing isolation or restart operations on abnormal devices; and a log recording subunit for storing complete information about abnormal events.

4. The IoT data stream real-time processing and anomaly detection system according to claim 1, characterized in that, The streaming processing engine adopts a microservice architecture, with each functional module deployed independently and supporting horizontal scaling.

Images