Method and device for identifying abnormal equipment in streaming data transmission

Abstract

The invention relates to the technical field of Internet of Things, and discloses a method for identifying abnormal equipment through streaming data transmission, which comprises the following steps of: 1, acquiring equipment information, continuously acquiring monitoring data reported by equipment and reporting time, extracting target data from the monitoring data, and transmitting the target data to a server; integrating the equipment information, the reporting time and the target data into triple data; 2, processing the triple data by adopting a SimHash algorithm to generate a real-time data fingerprint; and 3, calculating the Hamming distance between two adjacent real-time data fingerprints by adopting the Hamming distance, when the Hamming distance is smaller than a Hamming distance threshold value, triggering a counting mechanism to obtain continuous triggering times, and when the continuous triggering times are larger than preset times, judging that the equipment has an abnormal condition. According to the method, the streaming data can be effectively processed, and the abnormal equipment can be accurately and quickly identified. Meanwhile, the invention also discloses a device for identifying the abnormal equipment in streaming data transmission.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a method and apparatus for identifying abnormal devices through streaming data transmission. Background Technology

[0002] Current common equipment monitoring methods are mostly based on fixed thresholds or simple rules, which are difficult to adapt to the real-time and dynamic characteristics of streaming data. For some complex equipment anomalies, such as subtle abnormal change patterns of equipment at different times and under different environments, these traditional methods are often not accurate or timely enough in identifying them, and are prone to missed or false alarms, failing to meet the actual needs for efficient and accurate monitoring of equipment operating status.

[0003] CN201611215695 discloses a device parameter anomaly detection system and method based on threshold adaptive setting, including a power transformer, a concentrator, a transmission line, and a detection system. The power transformer includes N monitoring points, and the detection system includes a receiver and a processor. The N monitoring points are connected to the concentrator, and the concentrator is connected to the receiver in the detection system via the transmission line. The receiver is connected to the processor, and N is a positive integer greater than 2. The N monitoring points are used to detect the operating status of the power transformer in real time, acquire multi-dimensional real-time measurement point data, and transmit it to the concentrator. The concentrator, after receiving the measurement point data sent by the N monitoring points, sends the measurement point data received at different times in batches to the receiver via the transmission line according to the receiving time. The receiver is used to receive data from the concentrator. The transmitted measurement point data, after preprocessing, is sent to the processor. The processor includes: a regularization module for normalizing the preprocessed measurement point data; a dimensionality reduction module for performing multi-dimensional dimensionality reduction on the normalized data sequence of substation equipment data; a prediction module for predicting the real-time dimensionality-reduced data sequence of substation equipment operating status to generate a predicted data sequence of substation equipment operating status; a score calculation module for calculating the score of the substation equipment operating status at the current moment using the actual data and predicted data; a threshold calculation module for determining the threshold range of the substation equipment operating status, including the upper threshold boundary and the lower threshold boundary, and calculating the normal threshold range; and an anomaly detection module for determining whether the score at the current moment is within the threshold range to obtain an anomaly result.

[0004] This monitoring system uses a prediction module to predict equipment status, obtaining a predicted data sequence. It then calculates a status score by comparing the predicted data sequence with actual data, and determines whether the score falls within a threshold range to identify any equipment malfunctions. However, this method, which relies on prediction and comparison with actual data before threshold determination, struggles to meet the real-time and dynamic requirements of streaming data. Furthermore, it may fail to effectively capture subtle changes in the equipment, thus overlooking potential anomalies.

[0005] The technical problem that this invention aims to solve is: how to effectively process streaming data and accurately and quickly identify abnormal devices. Summary of the Invention

[0006] The main objective of this invention is to provide a method for identifying abnormal devices using streaming data transmission. This method integrates device information, reporting time, and target data into triplet data. The triplet data is then processed using the SimHash algorithm to obtain a real-time data fingerprint. Finally, the Hamming distance of the real-time data is calculated to determine whether the device is abnormal. This approach effectively captures subtle changes in device behavior patterns and accurately identifies abnormal devices. Compared to traditional methods based on fixed thresholds, this significantly improves the accuracy of anomaly identification and reduces false positives and false negatives.

[0007] In addition, a device for identifying abnormal devices through streaming data transmission is also provided.

[0008] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0009] A method for identifying abnormal devices through streaming data transmission includes the following steps:

[0010] Step 1: Obtain device information and continuously collect monitoring data and reporting time reported by the device. Extract target data from the monitoring data and integrate device information, reporting time and target data into triplet data.

[0011] Step 2: Process the triplet data using the SimHash algorithm to generate real-time data fingerprints;

[0012] Step 3: Calculate the Hamming distance between two adjacent real-time data fingerprints using Hamming distance. When the Hamming distance is less than the Hamming distance threshold, a counting mechanism is triggered to obtain the number of consecutive triggers. When the number of consecutive triggers is greater than the preset number, it is determined that the device has an abnormal situation.

[0013] Preferably, step 1 includes the following sub-steps:

[0014] Step A1: Obtain device information;

[0015] Step A2: Collect monitoring data reported by the device at preset time intervals, and extract target data from the monitoring data based on the device anomaly association information. The preset time interval is 1 to 60 seconds.

[0016] Step A3: Collect the reporting time. Based on the reporting time, integrate the device information, target data, and corresponding data reporting time into multiple triplet data.

[0017] Preferably, step 2 specifically involves: using the SimHash algorithm to split the triplet data, extracting device information, reporting time, and target data from the triplet data, converting the device information, reporting time, and target data into binary vectors to obtain device information vector, reporting time vector, and target data vector, and setting the weights of the device information vector, reporting time vector, and target data vector to generate a real-time data fingerprint.

[0018] Preferably, in step 3, the counting mechanism is as follows: when the Hamming distance is less than the Hamming distance threshold, the real-time data fingerprint is recorded, the device is marked as a suspected abnormal device, and the number of times the Hamming distance is less than the Hamming distance threshold is recorded consecutively, and the number of consecutive triggers is output.

[0019] Preferably, in step 3, the Hamming distance threshold is 5 to 13, and the preset number of attempts is 3 to 8.

[0020] Preferably, the device information is a unique identifier; the reporting time is the corresponding time of the monitoring data reported by the device; and the monitoring data consists of various types of parameters of the device.

[0021] Preferably, the triplet data is in JSON format.

[0022] It should be noted that:

[0023] SimHash Algorithm: The SimHash algorithm is a hash algorithm used for text similarity detection. It was originally proposed by Charles LADorso et al. at Google for approximate nearest neighbor search on large-scale datasets. The core idea of ​​the SimHash algorithm is to convert text into fixed-length binary signatures, which can be used to quickly compare the similarity between texts.

[0024] Hamming distance: The number of bits whose corresponding values ​​differ between two codewords is called the Hamming distance between the two codewords. In an effective code set, the minimum Hamming distance between any two codewords is called the Hamming distance of that code set.

[0025] Streaming data: Streaming data refers to data that is generated and transmitted continuously and in real time, rather than being returned as a complete dataset all at once. Unlike the traditional "request-full response" model, streaming data divides content into multiple chunks, which are transmitted to the front end gradually over the network. The front end can process and display the data in real time during the reception process, without waiting for all the data to load completely.

[0026] Meanwhile, an apparatus for identifying abnormal devices in streaming data transmission is also provided, which implements the above-mentioned method for identifying abnormal devices in streaming data transmission, and includes the following units:

[0027] Data acquisition unit: used to acquire equipment information, continuously collect monitoring data reported by the equipment and the reporting time, extract target data from the monitoring data, and integrate equipment information, reporting time and target data into triplet data;

[0028] Fingerprint generation unit: Used to process triplet data using the SimHash algorithm to generate real-time data fingerprints;

[0029] Anomaly detection unit: Used to calculate the Hamming distance between two adjacent real-time data fingerprints using Hamming distance. When the Hamming distance is less than the Hamming distance threshold, a counting mechanism is triggered to obtain the number of consecutive triggers. When the number of consecutive triggers is greater than the preset number, it is determined that the device has an anomaly.

[0030] Compared with existing technologies, this solution has the following advantages:

[0031] This solution identifies anomalous devices by integrating device information, reporting time, and target data into triplet data. This triplet data can extract necessary information from large amounts of streaming data for device identification and anomalous device data. The triplet data is processed using the SimHash algorithm to obtain a real-time data fingerprint. Then, the Hamming distance of the real-time data is calculated to determine whether the device is anomalous. This method effectively captures subtle changes in device behavior patterns, accurately identifies anomalous devices, significantly improves the accuracy of anomaly identification, and reduces false positives and false negatives.

[0032] Furthermore, the SimHash algorithm and Hamming distance are used to determine whether a device is abnormal. When a device's data first shows a problem, it is immediately marked as a suspected anomaly. Only when the device experiences continuous problems is it judged as an abnormal device. When a device is abnormal, it can be repaired immediately. When a device is marked as a suspected anomaly, staff can focus on monitoring that device during the maintenance cycle. This improves the performance and reliability of the device monitoring system, ensuring that abnormalities can be detected and dealt with in a timely manner, reducing potential safety risks and the impact of device failures. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method for identifying abnormal devices through streaming data transmission in Example 1;

[0034] Figure 2 This is a flowchart of the device for identifying abnormal devices through streaming data transmission in Example 1. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of this application implemented as described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0036] Example 1

[0037] refer to Figure 1 A method for identifying abnormal devices through streaming data transmission includes the following steps:

[0038] Step 1: Obtain device information and continuously collect monitoring data and reporting time reported by the device. Extract target data from the monitoring data and integrate device information, reporting time and target data into triplet data.

[0039] In this embodiment, the specific usage process of this application is illustrated by taking the identification of abnormal repeated opening and closing events of elevator doors as an example.

[0040] Preferably, the device information is a unique identifier; the reporting time is the corresponding time of the monitoring data reported by the device; and the monitoring data consists of various types of parameters of the device.

[0041] Equipment information is a unique identifier, also known as the equipment ID. Monitoring data includes various parameters of the equipment. Taking an elevator as an example, this includes floor, movement status, car door status, speed, car temperature, lighting, and tilt angle. The reporting time is the specific time when each piece of monitoring data is reported by the equipment. Each report of monitoring data is recorded with a corresponding reporting time.

[0042] Preferably, step 1 includes the following sub-steps:

[0043] Step A1: Obtain device information;

[0044] In this embodiment, it may be necessary to monitor multiple elevators. Therefore, the device IDs of different devices are obtained to distinguish the monitoring data obtained by different devices.

[0045] Step A2: Collect monitoring data reported by the device at preset time intervals, and extract target data from the monitoring data based on the device anomaly association information. The preset time interval is 1 to 60 seconds.

[0046] In this embodiment, the monitoring data reported by the device is collected at a frequency of 2 seconds. Based on the device abnormal association information, in identifying the abnormal repeated opening and closing events of the elevator, the association information is the floor, motion status and car door status. Therefore, the target data is the floor, motion status and car door status. The relevant data of the floor, motion status and car door status are extracted from the monitoring data.

[0047] Step A3: Collect the reporting time. Based on the reporting time, integrate the device information, target data, and corresponding reporting time into multiple triplet data.

[0048] In this embodiment, when collecting monitoring data, the reporting time corresponding to the monitoring data is also collected. For example, if a monitoring data is recorded at 15:40:42 on July 3, 2025, then the reporting time of this monitoring data is recorded as 15:40:42 on July 3, 2025. And so on, the reporting time corresponding to each monitoring data is recorded.

[0049] Device information, target data, and corresponding reporting time are integrated into triplet data to obtain multiple triplet data sets. The triplet data sets are in JSON format.

[0050] Example of triplet data code:

[0051] device:001, day:20250703, time:154042, data:5,0,1 (Stayed floor, movement status, car door status)

[0052] In this data, device:001 represents the device ID as 001, day:20250703,time:154042 indicates that the data was recorded at 15:40:42 on July 3, 2025, and in data:5,0,1, the first digit represents the floor where the device is located, the second digit represents the movement status, and the third digit represents the car door status. In this data, the specific floor where the device is located is the 5th floor, the movement status is stopped, and the car door status is open.

[0053] This method ensures that each triplet data set only includes data that identifies device malfunctions, reducing the amount of data. Furthermore, by adding device information, triplet data from different devices can be better differentiated.

[0054] Step 2: Process the triplet data using the SimHash algorithm to generate real-time data fingerprints;

[0055] Preferably, step 2 specifically involves: using the SimHash algorithm to split the triplet data, extracting device information, reporting time, and target data from the triplet data, converting the device information, reporting time, and target data into binary vectors to obtain device information vector, reporting time vector, and target data vector, and setting the weights of the device information vector, reporting time vector, and target data vector to generate a real-time data fingerprint.

[0056] In this embodiment, during the continuous acquisition of monitoring data and reporting time to form multiple triplet data, the triplet data is continuously input into the SimHash algorithm. The SimHash algorithm first splits the triplet data, extracts device information, reporting time, and target data from the triplet data, and then converts the device information, reporting time, and target data into 64-bit binary vectors respectively. The staff sets corresponding weights for the device information vector, reporting time vector, and target data vector respectively. In this embodiment, the weights of the device information vector, reporting time vector, and target data vector are all equal. Then, the device information vector, reporting time vector, and target data vector are integrated together according to their weights to generate a 64-bit real-time data fingerprint, thereby generating multiple 64-bit real-time data fingerprints.

[0057] This method yields a fingerprint value that represents the characteristics of the data, and it has the feature of generating similar fingerprints from similar data, providing a basis for subsequent similarity comparisons.

[0058] Here is a code example for generating real-time data fingerprints:

[0059] def generate_simhash(triplet):

[0060] features = [

[0061] triplet["device_id"],

[0062] f"{triplet['time']}-{triplet['metrics']['temp']}",

[0063] f"{triplet['metrics']['humidity']}-{hash(triplet['metrics'])}" ]

[0065] return SimHash(features,num_bits=64)

[0066] Step 3: Calculate the Hamming distance between two adjacent real-time data fingerprints using Hamming distance. When the Hamming distance is less than the Hamming distance threshold, a counting mechanism is triggered to obtain the number of consecutive triggers. When the number of consecutive triggers is greater than the preset number, it is determined that the device has an abnormal situation.

[0067] In this embodiment, Hamming distance is continuously calculated for adjacent real-time data fingerprints. When the Hamming distance is less than 8, a counting mechanism is triggered, marking the device as a suspected abnormal device. Marking a device as suspected abnormal indicates a minor anomaly that does not affect normal operation. During maintenance cycles, staff can identify which devices are suspected of being abnormal and conduct targeted inspections to address potential issues promptly, ensuring the system's normal operation. Furthermore, if the Hamming distance is continuously checked more than three times consecutively, it indicates the device is stuck in a certain state with no further changes, meaning it's in an abnormal or frozen state. In this case, staff immediately conduct further inspection and processing to resolve the problem promptly.

[0068] The counting mechanism is as follows: when the Hamming distance is less than the Hamming distance threshold, the real-time data fingerprint is recorded, the device is marked as a suspected abnormal device, and the number of consecutive times the Hamming distance is less than the Hamming distance threshold is recorded, and the number of consecutive triggers is output.

[0069] Code example for calculating Hamming distance:

[0070] int hamming=popcount(fingerprint1^fingerprint2);

[0071] The advantages of this application are:

[0072] 1. Accurate identification: By using SimHash fingerprints and Hamming distance calculations, it can effectively capture subtle changes in device behavior patterns and accurately identify abnormal and suspected abnormal devices. Compared with traditional methods based on fixed thresholds, it greatly improves the accuracy of anomaly identification and reduces false alarms and missed detections.

[0073] 2. Strong real-time performance: Targeting the characteristics of streaming data transmission, it enables real-time data acquisition, processing, and analysis, which can promptly detect equipment anomalies and trigger alarms, allowing staff to respond quickly, handle equipment failures in a timely manner, reduce the impact of failures on equipment system operation, and improve overall operational efficiency and safety.

[0074] 3. Wide applicability: This application can be applied to monitoring scenarios of various equipment, such as elevators and industrial equipment. As long as the relevant monitoring index data of the equipment can be collected, abnormal equipment can be identified by constructing a triplet and applying the method of this application. For different types of equipment, only the corresponding target data needs to be set, without modifying other parts. It has good versatility and scalability, and provides effective technical support for the intelligent monitoring of various types of equipment.

[0075] Example 2

[0076] refer to Figure 2 An apparatus for identifying abnormal streaming data transmission devices, and a method for implementing the above-mentioned method for identifying abnormal streaming data transmission devices, comprising the following units:

[0077] Data acquisition unit: used to acquire equipment information, continuously collect monitoring data reported by the equipment and the reporting time, extract target data from the monitoring data, and integrate equipment information, reporting time and target data into triplet data;

[0078] Fingerprint generation unit: Used to process triplet data using the SimHash algorithm to generate real-time data fingerprints;

[0079] Anomaly detection unit: Used to calculate the Hamming distance between two adjacent real-time data fingerprints using Hamming distance. When the Hamming distance is less than the Hamming distance threshold, a counting mechanism is triggered to obtain the number of consecutive triggers. When the number of consecutive triggers is greater than the preset number, it is determined that the device has an anomaly.

[0080] The specific process of this device is as follows: the data acquisition unit collects the device information, monitoring data and reporting time of the device, extracts the target data from the monitoring data, continuously integrates the device information, reporting time and target data into triplet data, obtains multiple triplet data, and then sends the multiple triplet data to the fingerprint generation unit.

[0081] After receiving multiple triplet data, the fingerprint generation unit processes the triplet data using the SimHash algorithm, converting each part of the triplet data into a 64-bit binary vector. Then, according to the set weights, the 64-bit binary vectors of each part of the triplet data are integrated together to generate a real-time data fingerprint. In this way, multiple real-time data fingerprints are continuously generated at the same time intervals, and the multiple real-time data fingerprints are sent to the anomaly detection unit.

[0082] After receiving multiple real-time data fingerprints, the anomaly detection unit continuously calculates the Hamming distance between two adjacent real-time data fingerprints using Hamming distance and determines whether the Hamming distance between two adjacent real-time data fingerprints is greater than the Hamming distance threshold. When it is greater than the Hamming distance threshold, a counting mechanism is triggered, marking the device as a suspected abnormal device and outputting the number of consecutive triggers. When the number of consecutive triggers is greater than 3, the device is determined to be an abnormal device, and staff are notified to handle it.

[0083] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for identifying abnormal devices through streaming data transmission, characterized in that, Includes the following steps: Step 1: Obtain device information and continuously collect monitoring data and reporting time reported by the device. Extract target data from the monitoring data and integrate device information, reporting time and target data into triplet data. Step 2: Process the triplet data using the SimHash algorithm to generate real-time data fingerprints; Step 3: Calculate the Hamming distance between two adjacent real-time data fingerprints using Hamming distance. When the Hamming distance is less than the Hamming distance threshold, a counting mechanism is triggered to obtain the number of consecutive triggers. When the number of consecutive triggers is greater than the preset number, it is determined that the device has an abnormal situation.

2. The method for identifying abnormal devices through streaming data transmission according to claim 1, characterized in that, Step 1 includes the following sub-steps: Step A1: Obtain device information; Step A2: Collect monitoring data reported by the device at preset time intervals, and extract target data from the monitoring data based on the device anomaly association information. The preset time interval is 1 to 60 seconds. Step A3: Collect the reporting time. Based on the reporting time, integrate the device information, target data, and corresponding data reporting time into multiple triplet data.

3. The method for identifying abnormal devices using streaming data transmission according to claim 2, characterized in that, Step 2 specifically involves: using the SimHash algorithm to split the triplet data, extracting device information, reporting time, and target data from the triplet data, converting the device information, reporting time, and target data into binary vectors to obtain device information vector, reporting time vector, and target data vector, and setting the weights of the device information vector, reporting time vector, and target data vector to generate a real-time data fingerprint.

4. The method for identifying abnormal devices through streaming data transmission according to claim 1, characterized in that, In step 3, the counting mechanism is as follows: when the Hamming distance is less than the Hamming distance threshold, the real-time data fingerprint is recorded, the device is marked as a suspected abnormal device, and the number of times the Hamming distance is less than the Hamming distance threshold is recorded consecutively, and the number of consecutive triggers is output.

5. The method for identifying abnormal devices through streaming data transmission according to claim 1, characterized in that, In step 3, the Hamming distance threshold is 5 to 13, and the preset number of attempts is 3 to 8.

6. The method for identifying abnormal devices through streaming data transmission according to claim 1, characterized in that, The device information is a unique identifier; the reporting time is the corresponding time of the monitoring data reported by the device; the monitoring data consists of various types of parameters of the device.

7. The method for identifying abnormal devices through streaming data transmission according to claim 1, characterized in that, The triplet data is in JSON format.

8. A device for identifying abnormal devices through streaming data transmission, characterized in that, A method for implementing the streaming data transmission anomaly identification device according to any one of claims 1-7, comprising the following units: Data acquisition unit: used to acquire equipment information, continuously collect monitoring data reported by the equipment and the reporting time, extract target data from the monitoring data, and integrate equipment information, reporting time and target data into triplet data; Fingerprint generation unit: Used to process triplet data using the SimHash algorithm to generate real-time data fingerprints; Anomaly detection unit: Used to calculate the Hamming distance between two adjacent real-time data fingerprints using Hamming distance. When the Hamming distance is less than the Hamming distance threshold, a counting mechanism is triggered to obtain the number of consecutive triggers. When the number of consecutive triggers is greater than the preset number, it is determined that the device has an anomaly.

Images