Flow timing anomaly detection and fault data extraction method and system

Abstract

The application discloses a stream timing anomaly detection and fault data extraction method and system, and through a distributed message queue, real-time collection of IoT equipment timing data and storage to a timing database; time window alignment, cleaning and feature engineering are carried out on stream data, window statistical features and inter-measuring-point relationship features based on expert knowledge are extracted; a layered anomaly detection strategy is adopted, and efficient and accurate anomaly identification is realized through threshold rules, expert rules and a lightweight machine learning model in sequence; when an anomaly is detected, according to a preconfigured fault correlation table, historical and real-time context data of related measuring points are automatically extracted from the timing database, and a fault scenario data package is generated, and the application relates to the technical field of data processing. The stream timing anomaly detection and fault data extraction method and system solve the problems of anomaly monitoring and fault rapid diagnosis of massive IoT timing data by combining stream processing, a high-performance timing database and a layered and knowledge-driven detection method.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for streaming time-series anomaly detection and fault data extraction. Background Technology

[0002] With the rapid development of IoT technology, various industries, such as industrial manufacturing, energy, transportation, and smart cities, have deployed a large number of sensors and smart devices, generating massive amounts of time-series data. Real-time or near-real-time analysis of this data, especially anomaly detection, is of great significance for ensuring the safe and stable operation of equipment, predicting potential failures, and reducing operation and maintenance costs.

[0003] Existing IoT time-series data anomaly detection technologies mainly include:

[0004] a) Threshold-based method: Set fixed or dynamic upper and lower limits for data, and trigger an alarm if the data exceeds the threshold.

[0005] b) Statistical analysis-based methods: Calculate the mean, variance, moving average, and other statistical measures of the data to determine whether the data deviates from the normal statistical distribution.

[0006] c) Machine learning-based methods: These methods utilize historical data to train models (such as clustering, classification, prediction models, outlier detection algorithms, etc.) to identify data points or sequences that do not conform to normal patterns. Commonly used algorithms include Isolation Forest, One-Class SVM, clustering algorithms (such as DBSCAN), and time series prediction models (such as ARIMA, LSTM, etc.).

[0007] d) Rule-based approach: Define a set of rules based on the experience of domain experts to identify abnormal situations.

[0008] To handle massive amounts of time-series data, existing data storage solutions often employ traditional relational databases or big data storage systems (such as HDFS). Some solutions are beginning to use time-series databases (such as InfluxDB and TimescaleDB) to optimize the storage and query performance of time-series data. Data processing may be implemented using batch processing frameworks (such as Apache Spark) or stream processing frameworks (such as Apache Flink and Apache Spark Streaming).

[0009] Problems with existing technology:

[0010] While existing technologies offer some methods for anomaly detection in IoT time-series data, the following significant problems and limitations still exist when dealing with massive, high-concurrency, and high-dimensional IoT time-series data:

[0011] a) Insufficient real-time performance: Traditional batch processing methods cannot meet the needs of real-time monitoring and fault early warning. Although some stream processing solutions achieve real-time processing, their anomaly detection algorithms may have excessively high computational complexity, making it difficult to cope with extremely high data rates.

[0012] b) Limited anomaly detection accuracy: Simple thresholding or statistical methods struggle to capture complex, multivariate correlation-based anomaly patterns. Detection based on a single measurement point or simple statistics is prone to false positives or false negatives. Furthermore, complex machine learning models (especially those requiring in-window data or sequences) are costly to infer in real-time on massive streaming data and struggle to effectively integrate domain expert knowledge.

[0013] c) Lack of effective utilization of correlations between measurement points and professional knowledge: Most existing methods focus on the analysis of single time series, ignoring the complex correlations and causal chains between different sensors under equipment failure or specific operating conditions. The rich experience of domain experts (e.g., "If signal A is high and signal B is low, component C may be malfunctioning") is difficult to directly and efficiently encode into real-time processing flows.

[0014] d) Inefficient Fault Diagnosis: When an anomaly is detected and an alarm is issued, maintenance or diagnostic personnel typically need to manually search for data (such as temperature, pressure, current, vibration, etc.) from different systems or databases related to the anomaly before and after the same time period. This process is time-consuming and labor-intensive, and it is easy to miss key information, seriously affecting the efficiency and timeliness of fault diagnosis. Existing solutions often only provide information about the anomaly point itself, lacking the proactive and automated provision of the "context" data required for the fault.

[0015] e) Storage and Query Efficiency Bottlenecks: Despite using a time-series database, if a large amount of historical data from relevant measurement points needs to be queried manually or through non-optimized methods when anomalies occur, especially in high-concurrency scenarios, query latency or excessive system load may still occur. Traditional databases are even less suitable for such high-concurrency time-series range queries.

[0016] f) High system complexity: Integrating various independent components (such as data acquisition, storage, anomaly detection, and alarms) may lead to a complex system architecture and difficult maintenance. Summary of the Invention

[0017] To address the shortcomings of existing technologies, this invention provides a method and system for streaming time-series anomaly detection and fault data extraction, which solves the problems of insufficient real-time performance, limited anomaly detection accuracy, lack of effective utilization of correlation between measurement points and professional knowledge, low fault diagnosis efficiency, bottlenecks in storage and query efficiency, and high system complexity.

[0018] To achieve the above objectives, the present invention provides a method for streaming time-series anomaly detection and fault data extraction, comprising the following steps:

[0019] S1. Based on the equipment type and potential failure modes, domain experts define the rules for the association between measurement points (for feature extraction) and the failure association configuration (anomaly type -> related measurement point list + time window), and load these configurations into the system;

[0020] S2, deploying Kafka clusters, InfluxDB clusters, stream processing jobs, fault correlation data rapid extraction services, and alarm and visualization systems;

[0021] S3, IoT devices begin generating data, which is then sent to Kafka via the gateway;

[0022] S4. The data is written to InfluxDB for persistent storage;

[0023] S5: The stream processing job reads data from Kafka in real time, performs time alignment, windowing, feature engineering, and hierarchical anomaly detection.

[0024] S6. Detected "abnormal events" are sent via Kafka;

[0025] S7: Quickly extract service subscription exception events from fault-related data and retrieve fault scenario data from InfluxDB according to the configuration;

[0026] S8. Fault scenario data packets are sent downstream, triggering alarms and displaying them in the visualization interface for maintenance personnel to quickly diagnose.

[0027] S9 allows maintenance personnel to perform diagnoses based on the complete context data provided, and to adjust rules or configurations to optimize system performance as needed.

[0028] This invention also discloses a streaming time-series anomaly detection and fault data extraction system, comprising:

[0029] Data ingestion and time-series database storage module:

[0030] It is responsible for collecting massive amounts of time-series data from various IoT devices in real time. Each data point includes a device ID, measurement point ID, timestamp, and value.

[0031] High-performance distributed message queues (such as Apache Kafka) are used as a data buffer layer to achieve high-throughput data reception and peak shaving.

[0032] One or more consumer applications are built to subscribe to data from a message queue and write the data at high speed to a time-series database (such as InfluxDB) specifically designed for processing time-series data. InfluxDB’s features (such as schema-less design, efficient write and query, automatic partitioning by time, etc.) are well-suited to the data storage requirements of this invention.

[0033] In InfluxDB, data is organized by tags such as device ID and measurement point ID, which facilitates efficient filtering and querying based on these tags.

[0034] Streaming preprocessing and feature engineering module:

[0035] Build a stream processing job;

[0036] (For example, using Apache Flink or Apache Spark Structured Streaming), the job receives real-time time-series data streams from a message queue or directly from the data ingestion module.

[0037] Time alignment and windowing: Using the windowing mechanism (such as sliding window or rolling window) provided by the stream processing framework, data from different measurement points with different acquisition frequencies are aligned and windowed according to a uniform time granularity. For example, a sliding window of fixed length (such as 10 seconds) can be defined, and processing can be performed once at certain intervals (such as 1 second).

[0038] Data cleaning and imputation: Clean the data within the window (e.g., filter invalid values) and handle missing values ​​(e.g., forward imputation, linear interpolation).

[0039] Window statistical characteristic calculation: Within each window, the statistical characteristics of each measurement point are calculated, such as mean, maximum, minimum, standard deviation, rate of change, percentiles, etc. These calculations are efficient and based solely on the data within the window.

[0040] Inter-measurement point relationship feature extraction: This is one of the key improvements of this invention. Based on pre-configured rules governing the association between measurement points using domain expert knowledge, new combined features are calculated within the aligned window data. For example, for a pump system, features such as "inlet pressure - outlet pressure" and "motor current / pump speed" can be calculated. These features directly reflect the mutual influence and operating status between various parts within the equipment. These rules and feature definitions are stored in a configuration management system.

[0041] The raw data (or a significant portion thereof) and the calculated features (including window statistical features and inter-point relationship features) are output as an enhanced data stream for use by the subsequent anomaly detection module.

[0042] Layered anomaly detection module:

[0043] This module can be the post-processing logic for a stream processing job, or it can be a standalone microservice that subscribes to the enhanced data stream output by the preprocessing module.

[0044] A layered detection strategy is adopted to achieve a combination of rapid filtering and accurate judgment:

[0045] The first layer: based on fast thresholding and simple rule detection. Preset thresholds or simple logical rules are applied to the current value of a single measurement point or simple window statistical features (such as mean, standard deviation) in the enhanced data stream. These thresholds can be calculated based on historical normal data statistics, and the rules can be very direct judgments (such as "temperature > 100 degrees"). This layer has minimal computational cost and can quickly filter out most obviously abnormal or normal data.

[0046] The second layer: Detection based on the relationship features between measurement points and complex rules. This is the second key improvement of this invention. Complex rules based on domain expert knowledge encoding are applied to the relationship features between measurement points extracted by the preprocessing module and the more complex window statistical features. These rule judgments combine the performance of multiple measurement points within the same time window and the correlation between them (e.g., "If inlet pressure - outlet pressure < threshold 1 and motor current > threshold 2, then alarm"). The definition and management of rules are implemented through a rule engine or custom logic. This layer can discover complex patterns where individual measurement points perform normally but their combinations are abnormal.

[0047] The third layer (optional): Detection based on lightweight machine learning models. For potentially complex anomaly patterns that cannot be described by explicit rules, pre-trained lightweight machine learning models (such as online One-Class SVM trained on normal data, pattern recognition based on simple neural networks, etc.) can be used to infer the augmented data stream. The model input can be a multivariate feature vector within a window. This layer provides the ability to detect more subtle anomalies.

[0048] When any layer detects a potential anomaly, it generates an "anomaly event" message, which includes the device ID, the timestamp of the anomaly occurrence, the type of anomaly detected (if categorizable), and the triggering rule or model information. These anomaly events are sent to another message queue (such as KafkaTopic) as triggers for fault alarms and subsequent processing.

[0049] Fault-related data rapid extraction module:

[0050] This is the third key improvement of this invention. A separate microservice or application is built to subscribe to an "abnormal event" message queue.

[0051] Maintain a "Fault Association Configuration Table". This table stores the fault modes associated with each "anomaly type" or device category, as well as which relevant measurement points need to be extracted from the time-series database when such anomalies are detected, and the extraction time window range (relative to the anomaly time point T, e.g., [T-5 minutes, T+1 minutes]). This configuration table reflects domain expert knowledge and directly guides the data extraction process.

[0052] When an “abnormal event” message is received, information such as device ID, abnormal timestamp (T), and abnormal type is parsed out.

[0053] Based on the anomaly type and device ID (or device type), query the "Fault Association Configuration Table" to obtain the "List of Related Measurement Points" and "Time Window Range" from which the data to be extracted.

[0054] Send a single or batch optimized query request to the time series database (InfluxDB). Leverage InfluxDB's query capabilities, optimized for time range and label filtering, to quickly retrieve all raw data points for a specified device and a specified list of measurement points within a specified time window. For example, use InfluxDB's Flux Query Language or InfluxQL to efficiently execute such batch range queries.

[0055] All relevant data points retrieved, along with the original abnormal event information, are encapsulated into a complete "fault scenario data package".

[0056] The "fault scenario data package" is sent to downstream systems (such as fault diagnosis platforms, historical data storage, alarm systems) or written to dedicated storage (such as object storage or another database table) for use by operations and maintenance personnel or automated diagnostic systems.

[0057] Alarm and visualization module:

[0058] Receive "fault scenario data packets" or "abnormal event" messages.

[0059] Based on the alarm policy, trigger the corresponding alarm notification (SMS, email, mobile application push, etc.).

[0060] It provides a user interface that displays an overview of anomalies in real time. When viewing a specific anomaly, it can immediately load and visualize the data curves of all relevant measurement points in the corresponding "fault scenario data package" before and after the anomaly time window, greatly assisting maintenance personnel in quickly and comprehensively locating and diagnosing faults.

[0061] Replaceable parts or similar technical solutions:

[0062] Kafka can be replaced by other distributed message queues such as Apache Pulsar or AWS Kinesis.

[0063] InfluxDB can be replaced by other high-performance time-series databases such as TimescaleDB, OpenTSDB, or Kx Systems kdb+.

[0064] Apache Flink can be replaced by Apache Spark Structured Streaming or other frameworks that support stateful stream processing.

[0065] The rules engine can be implemented using Drools or a custom one.

[0066] The third-layer machine learning model can select other lightweight algorithms, such as online methods based on statistical process control (SPC), depending on the actual anomaly patterns and computational resources. The core principle is to build a normal model using historical data and detect deviations between the current data and the model.

[0067] This invention provides a method and system for streaming time-series anomaly detection and fault data extraction. Compared with existing technologies, it has the following advantages:

[0068] 1. This streaming time-series anomaly detection and fault data extraction method and system boasts extremely high real-time performance: based on a streaming processing framework, data ingestion, preprocessing, and multi-layer anomaly detection are all completed within a very short time after data arrival.

[0069] It completes within a short timeframe (typically with a sub-second to second delay), meeting the needs for real-time monitoring and early warning.

[0070] 2. This streaming time-series anomaly detection and fault data extraction method and system offers higher anomaly detection accuracy and robustness: The hierarchical detection strategy combines the speed of simple methods with the accuracy of complex methods. It introduces and utilizes the relationship features between measurement points based on domain expert knowledge, which can effectively identify complex fault modes where a single measurement point is normal but the combination is abnormal, significantly reducing the false alarm rate and false negative rate. Combining historical data (used for thresholds, statistics, and model training) makes the detection more adaptable to the dynamic characteristics of the data.

[0071] 3. This streaming time-series anomaly detection and fault data extraction method and system revolutionizes fault diagnosis efficiency: When an anomaly is detected, the system can automatically and quickly extract complete data from all measurement points related to that anomaly type within the fault occurrence time window. This eliminates the tedious process of manual data querying and correlation by maintenance personnel, reducing data preparation work that might have taken minutes or even hours to seconds or even milliseconds. This allows fault diagnosis to begin almost simultaneously with the alarm, significantly shortening the time to process faults (MTTR).

[0072] 4. This streaming time-series anomaly detection and fault data extraction method and system optimizes the efficiency of massive data storage and retrieval: It adopts high-performance time-series databases such as InfluxDB, combined with a batch query method based on anomaly events and optimized by database indexes, ensuring that the system can respond quickly when a large amount of historical context data needs to be extracted, avoiding excessive pressure on the database.

[0073] 5. This streaming time-series anomaly detection and fault data extraction method and system makes full use of domain expert knowledge: it transforms expert experience into executable rules, feature definitions and fault association configurations, so that this valuable tacit knowledge can be systematically and automatically applied to the real-time monitoring and fault diagnosis process.

[0074] 6. The streaming time-series anomaly detection and fault data extraction method and system have good scalability and reliability: built on distributed components such as Kafka, Flink / Spark, and InfluxDB, the system can be easily horizontally scaled to cope with the ever-increasing amount of data and the number of devices, while also having high fault tolerance.

[0075] 7. This streaming time-series anomaly detection and fault data extraction method and system reduces operation and maintenance costs: Automated anomaly detection and fault correlation data provision reduce the workload of manual monitoring and data retrieval, and improve operation and maintenance efficiency. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of the system of the present invention.

[0077] The diagram shows: 1. Data ingestion and time-series database storage module; 2. Streaming preprocessing and feature engineering module; 3. Hierarchical anomaly detection module; 4. Fault correlation data rapid extraction module; 5. Alarm and visualization module. Detailed Implementation

[0078] The technical solutions in the embodiments of the present invention have been clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Please see Figure 1 The present invention provides a technical solution: a method and system for streaming time-series anomaly detection and fault data extraction.

[0080] This invention provides a technical solution: a streaming time-series anomaly detection and fault data extraction system, including a data ingestion and time-series database storage module 1, a streaming preprocessing and feature engineering module 2, a hierarchical anomaly detection module 3, a fault-related data rapid extraction module 4, and an alarm and visualization module 5.

[0081] The workflow for each module is as follows:

[0082] Data Ingestion and Time-Series Database Storage Module 1:

[0083] 1. Data Acquisition and Kafka Integration:

[0084] Each IoT device or gateway packages the collected measurement data (e.g., temperature, pressure, current, voltage, vibration, switch status, etc.) into a message. The message format includes a unique device identifier (e.g., deviceId), a unique measurement point identifier (e.g., sensorId), a precise timestamp (Unix Nano or Millis), and the measurement point value.

[0085] These messages are sent to the Kafka cluster in the cloud via the MQTT protocol or HTTP POST. Kafka is configured with multiple partitions to ensure high-throughput data writes.

[0086] 2. Write data to InfluxDB:

[0087] Deploy an InfluxDB cluster and configure appropriate replication factors and sharding strategies to support massive data storage and query loads.

[0088] Develop one or more Java / Scala consumer applications that use the Kafka Consumer API to subscribe to raw data in Kafka Topics.

[0089] Consumer applications read messages from Kafka, parse them, and construct InfluxDB Point objects. Point Measurement names can be standardized (e.g., iot_measurements), tags include deviceId, sensorId, etc., field contains the value, and the timestamp uses the timestamp from the message.

[0090] Using libraries such as InfluxDBJavaClient, Point objects can be written to InfluxDB in batches, which can significantly improve write throughput.

[0091] Configure InfluxDB's Retention Policy to set different storage periods based on data value in order to manage storage space.

[0092] Streaming preprocessing and feature engineering module 2 (based on Apache Flink):

[0093] Develop a Flink Streaming Job.

[0094] Job subscribes to raw data topics from Kafka.

[0095] Use Flink's WatermarkStrategy to handle time and implement event time processing.

[0096] Use KeyBy(deviceId) to distribute data from the same device to the same parallel instance. Handle windowing and alignment: Apply a sliding window to the data stream grouped by deviceId.

[0097] For example;

[0098] .window(SlidingEventTimeWindows.of(Time.minutes(1),Time.seconds(10))) represents a window that is one minute long and slides once every 10 seconds.

[0099] Within the window, collect data points for all sensorIds. To handle different measurement point frequencies, a custom ProcessWindowFunction can be used to aggregate or resample the data at the end of the window, ensuring that data for all relevant measurement points are available within the same time window, even if some measurement points do not have new data within the window (strategies such as forward filling can be used).

[0100] Feature calculation:

[0101] In ProcessWindowFunction or subsequent FlatMap / ProcessFunction, based on the data within the window, calculate the window statistics (mean, StdDev, etc.) for each sensorId.

[0102] Simultaneously, it reads the configured relationship rules between measurement points (e.g., stored in ZooKeeper or a database) and calculates new features for specific sensorId combinations based on the rules.

[0103] For example, the average value of pressure_in and pressure_out can be obtained from the window, and pressure_drop = avg(pressure_in) - avg(pressure_out) can be calculated as a new feature.

[0104] An enhanced data stream (which can be represented as a structured object or JSON) containing raw values, window statistical features, and features relating between measurement points.

[0105] Send it to another KafkaTopic (such as processed_iot_data).

[0106] Layered anomaly detection module 3 (based on Flink and rule engine):

[0107] Within the same Flink Job, or in another Job, subscribe to the processed_iot_data Topic.

[0108] First layer (fast): Use a Filter or a simple ProcessFunction to check whether the data points or basic features violate preset static or dynamic thresholds (dynamic thresholds can be based on sliding window statistics of historical data, such as EWMA standard deviation).

[0109] For example,

[0110] if(data.getSensorValue("temperature")>temperature_threshold)

[0111] {emit_anomaly_event(...)}.

[0112] The second layer (complex rules): uses an integrated rule engine library (such as the open-source Drools embedded engine, or lightweight custom judgment logic). The ProcessFunction receives enhanced data objects and transforms them into fact objects that the rule engine can process. Rule execution is triggered; the rule set contains judgments based on the relationship characteristics between measurement points and complex logic.

[0113] For example, the Drools rule can be written as:

[0114] rule "PossiblePumpBlockage"

[0115] when

[0116] $data:ProcessedIoTData(

[0117] deviceId=="Pump001",

[0118] feature("pressure_drop")<-1.0, / / pressure_in-pressure_outislow

[0119] feature("motor_current_avg")>50.0 / / Motorcurrentishigh)

[0120] then

[0121] insertLogical(newAnomalyEvent($data.getDeviceId(),$data.getTimestamp(),"PumpBlockage","Rule:PumpBlockage"));

[0122] end

[0123] If the rules are violated, an "exception event" object is generated.

[0124] The third layer (ML): Within the same ProcessFunction, a pre-trained lightweight ML model is loaded. For each augmented data object, model inference is performed, and anomaly scores are calculated. If the score exceeds a threshold, an "anomaly event" is generated. Model parameters or model files can be stored in shared storage or a configuration service.

[0125] The generated "Exception Event" object contains;

[0126] The event information includes deviceId, timestamp, anomalyType, and triggerInfo (such as the triggered rule name or model score). These events are then serialized and sent to the anomaly_events Kafka Topic.

[0127] Fault-related data rapid extraction module 4:

[0128] Develop a standalone microservice (e.g., using Spring Boot).

[0129] Subscribe to anomaly_events Kafka Topic.

[0130] When the service starts, it loads the "fault association configuration table" from the database or configuration file and stores it in memory. The structure is as follows:

[0131] Map <Pair<String,String> Pair <List <string>,Pair<Duration,Duration>>>

[0132] (key: <deviceid> , <anomalytype>,value: <relatedsensorids>,<[BeforeDuration,AfterDuration]>).

[0133] When an exception event message is received, look up the corresponding configuration.

[0134] Constructing InfluxDB queries: Use the InfluxDB Java Client or Flux Client. Flux query example:

[0135] from(bucket:"iot_bucket")

[0136] |>range(start:<anomaly_timestamp-BeforeDuration> stop:<anomaly_timestamp+AfterDuration> )

[0137] |>filter(fn:(r)=>r._measurement=="iot_measurements")|>filter(fn:(r)=>r.deviceId==" <deviceid>")

[0138] |>filter(fn:(r)=>contains(value:r.sensorId,set:["<sensorId_1> ","<sensorId_2> ",...]))

[0139] When executing a query, InfluxDB leverages its time-series indexes and internal optimizations to efficiently return data from a specified time range, a specified device, and a specified measurement point.

[0140] The query results and exception event information are packaged together into a "fault scenario data package" in JSON or Protobuf format.

[0141] Send the data packets to downstream Kafka topics (such as fault_context_data) or store them directly in object storage.

[0142] Alarm and Visualization Module 5:

[0143] Alarm systems (such as Prometheus Alertmanager combined with Kafka Exporter) subscribe to anomaly_events topics and trigger notifications.

[0144] The visualization platform (such as Grafana) is configured with InfluxDB as the data source. A backend service is also developed to consume the fault_context_data topic. When a user views the details of a specific anomaly alert in Grafana, the backend service retrieves the corresponding fault context data package from storage based on the alert information and provides it to the Grafana frontend for display via API. Alternatively, Grafana can be directly configured to connect to the database / object storage storing the fault context data and perform a query based on the transmitted anomaly ID. The display interface simultaneously presents graphs of the faulty measurement point and all related measurement points within the fault time window, and can also display detailed numerical tables.

[0145] Precautions during operation:

[0146] Time synchronization: Ensure high time synchronization across all IoT devices, gateways, message queues, and processing systems, typically using the NTP protocol. Accurate timestamps are fundamental to time-series data processing.

[0147] Data quality: Handling issues such as noise, spikes, drift, and out-of-order data during the data acquisition process. The cleaning and filling steps in stream processing are crucial.

[0148] Configuration management: Relationship rules between measurement points, fault association configurations, etc. should be centrally managed and support hot loading or smooth updates to avoid system downtime.

[0149] Performance tuning: The performance of distributed components such as Kafka, Flink / Spark, and InfluxDB needs to be optimized based on the actual data volume, concurrency, and...

[0150] The processing logic is fine-tuned (e.g., number of partitions, parallelism, memory usage, GC, write batch size, index design, etc.).

[0151] False alarm / false negative balance: The thresholds and rules for anomaly detection need to be continuously optimized based on actual operation and maintenance feedback to balance false alarms (too many invalid alarms) and false negatives (missing real faults).

[0152] Fault scenario data packet size: The extraction time window and the number of relevant measurement points should be set reasonably to avoid the impact of excessively large single data packets on transmission and processing efficiency.

[0153] Security: Ensure data security and access control during data acquisition, transmission, storage, and processing.

[0154] This specific implementation provides a feasible and efficient technical approach. By combining streaming processing, high-performance time-series databases, and hierarchical, knowledge-driven detection methods, it can effectively solve the problems of anomaly monitoring and rapid fault diagnosis of massive IoT time-series data. At the same time, all contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0155] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0156] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.< / deviceid> < / relatedsensorids> < / anomalytype> < / deviceid> < / string>

Claims

1. A method for streaming time-series anomaly detection and fault data extraction, characterized in that, Includes the following steps: S1. Based on the equipment type and potential failure modes, domain experts define the rules for the relationship between measurement points and the failure association configuration, and load these configurations into the system. S2, deploying Kafka clusters, InfluxDB clusters, stream processing jobs, fault correlation data rapid extraction services, and alarm and visualization systems; S3, IoT devices begin generating data, which is then sent to Kafka via the gateway; S4. The data is written to InfluxDB for persistent storage; S5. The stream processing job reads data from Kafka in real time, performs time alignment, windowing, feature engineering, and hierarchical anomaly detection; the hierarchical anomaly detection adopts a three-layer detection strategy, including: the first layer, based on fast threshold and simple rule detection; The second layer is based on the relationship features between measurement points and complex rules for detection; the third layer is based on a lightweight machine learning model for detection. S6. Detected "abnormal events" are sent via Kafka; S7: Quickly extract service subscription exception events from fault association data, and retrieve fault scenario data from InfluxDB based on predefined fault association configurations; The fault association configuration defines the mapping relationship between the anomaly type and the relevant measurement point list and time window; The service queries the fault association configuration based on the anomaly type and device information in the received abnormal event, obtains the relevant measurement point list and time window range, and initiates a query to InfluxDB to obtain the data of the relevant measurement points of the device within the corresponding time window; S8. Fault scenario data packets are sent downstream, triggering alarms and displaying them in the visualization interface for maintenance personnel to quickly diagnose. S9. Based on the complete context data provided, the operations and maintenance personnel can perform diagnostics and adjust rules or configurations according to the actual situation to optimize system performance.

2. The streaming time-series anomaly detection and fault data extraction method according to claim 1, characterized in that: In step S1, the rules governing the relationship between measurement points are used for feature extraction, and the fault association is configured as anomaly type, a list of related measurement points, and a time window.

3. The streaming time-series anomaly detection and fault data extraction method according to claim 1, characterized in that: Step S1 also employs a high-performance distributed message queue as a data buffer layer to achieve high-throughput data reception and peak shaving.

4. The method for streaming time-series anomaly detection and fault data extraction according to claim 1, characterized in that: In step S2, InfluxDB is a time series database used to process time series data. In InfluxDB, data is organized by device ID and measurement point ID tags, and filtered and queried by tags.

5. The method for streaming time-series anomaly detection and fault data extraction according to claim 1, characterized in that: The Kafka is configured with multiple partitions to ensure high-throughput data writing.

6. A streaming time-series anomaly detection and fault data extraction system applied to the method described in any one of claims 1 to 5, characterized in that, include: The data acquisition and time-series database storage module is used to collect and store massive amounts of time-series data from various IoT devices in real time. The streaming preprocessing and feature engineering module is used to build a streaming processing job that receives real-time time-series data streams from a message queue or directly from the data ingestion module. A hierarchical detection strategy is adopted, which includes a first layer based on fast threshold and simple rule detection; The second layer is based on the relationship features between measurement points and complex rules for detection; the third layer is based on a lightweight machine learning model for detection, achieving a combination of fast filtering and accurate judgment. The fault-related data rapid extraction module is used to build an independent microservice or application that subscribes to an exception event message queue. When such an exception is detected, it needs to extract data of relevant measurement points from the time series database. The alarm and visualization module is used to receive fault scenario data packets and abnormal event messages; The fault association data rapid extraction module is also used to maintain a fault association configuration table, which defines the mapping relationship between anomaly types, related measurement point lists, and time windows. When an anomaly event is received, the module queries the fault association configuration table based on the anomaly type and equipment information to obtain the related measurement point list and time window range, and then initiates a query to the time series database.

7. The streaming time-series anomaly detection and fault data extraction system according to claim 6, characterized in that, The hierarchical detection strategy includes: First layer: Detection based on fast thresholding and simple rules; The second layer: detection based on the relationship features between measurement points and complex rules; The third layer: detection based on a lightweight machine learning model.

Images