A method and system for real-time monitoring and alarming of abnormal data

Abstract

The application discloses a kind of real-time monitoring and alarm method and system to abnormal data, belong to industrial internet system edge end equipment exception monitoring technical field, the implementation of the method includes data receiving and filtering module, data processing module, alarm grading module, alarm result publishing module and log information module, data receiving and filtering module is used to collect the data collected by all the equipment of current acquisition end, and all data is sorted and classified;Data processing module allows user to define alarm rule according to alarm rule template, after the data is divided according to specific time window, comparison calculation is carried out according to the alarm rule defined by user.The application filters the data collected in real time to ensure that the whole system is in normal operation state, provides real-time detection and timely feedback method for the problem that equipment error cannot be found in time in project operation, and feedback is given to operation and maintenance personnel for processing according to the abnormal information of each module.

Description

Technical Field

[0001] This invention relates to the field of abnormal monitoring technology for edge devices in industrial internet systems, specifically a method and system for real-time monitoring and alarming of abnormal data. Background Technology

[0002] In recent years, "Industrial Internet" has become a hot topic in the technology field, and more and more companies and projects are moving in this direction. The edge layer devices of the entire Industrial Internet system act as data collection terminals. The data they collect supports the normal operation of the entire Industrial Internet system. Therefore, if a device malfunctions and the collected data is not within the normal range, it will affect the subsequent model building and operation, and even affect the normal operation of industrial software.

[0003] Traditionally, to determine whether a device is still operating normally, maintenance personnel need to check the device's own operation logs, operating system logs, and the logs of the entire project. This not only requires a lot of transmission time, but also requires maintenance personnel to check a large number of useless logs because they do not know when the device malfunctioned. This is very time-consuming and labor-intensive for maintenance personnel. Summary of the Invention

[0004] The technical objective of this invention is to address the above-mentioned shortcomings by providing a method and system for real-time monitoring and alarming of abnormal data. This method provides a way to detect and promptly report equipment errors that cannot be detected in a timely manner during project operation and maintenance, and provides feedback on abnormal information from each module to the operation and maintenance personnel for processing.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] A method for real-time monitoring and alarming of abnormal data is disclosed. The method includes a data receiving and filtering module, a data processing module, an alarm classification module, an alarm result publishing module, and a log information module.

[0007] The data receiving and filtering module is used to collect data from all devices at the current acquisition end and to organize and classify all the data;

[0008] The data processing module allows users to customize alarm rules according to alarm rule templates. After the data is segmented according to a specific time window, it is compared and calculated according to the user-defined alarm rules.

[0009] The alarm classification module maintains a map of the alarm levels of all devices at the current acquisition end. This map caches the latest alarm levels of all devices at the current acquisition end. When the alarm level of a device reaches a set level, the alarm classification module will send a signal to the device to stop collecting data, so as to prevent the device from malfunctioning and causing the entire program to fail.

[0010] The alarm result publishing module is responsible for distributing new device alarms to users who have subscribed to the device's alarm information and maintenance personnel, so that relevant personnel can obtain the latest status of the device in a timely manner.

[0011] The log information module analyzes the program's running status and process based on the log information from each module, and clearly feeds back error information to the operations and maintenance personnel to facilitate their analysis.

[0012] This method adds a data processing module after the data collection at the edge layer. It is responsible for filtering and analyzing all the data and promptly returning all abnormal data to users and maintenance personnel. Maintenance personnel can then check whether the data abnormality is caused by equipment problems and repair or replace the equipment in a timely manner to ensure the normal operation of the entire system.

[0013] Preferably, the data receiving and filtering module summarizes all data collected by all devices at the acquisition end and classifies them according to device name; then, according to the monitoring rules set by the user, it sends the data of the monitored device to the data processing module.

[0014] Furthermore, the implementation of the data receiving and filtering module includes the following steps:

[0015] The receiving submodule receives information collected from various devices;

[0016] The filtering submodule filters the information, saves and packages the information of the points that are monitored by the user, and discards the data of devices that are not monitored by the user.

[0017] The filtering submodule sends the packaged data to the data processing module.

[0018] Preferably, the data processing module copies the data sent by the filtering submodule according to the time window specified by the user, and copies it into an upgrade window and a downgrade window;

[0019] The data processing module calculates the data in the specified window according to the user-defined alarm rules. If the user defines that when the error data reaches 60% of the total data, it is judged that the device is in an alarm state. Then, if there are ten data entries and more than six error data entries, the device is judged to be in an alarm state. The alarm level of the current device and the alarm data are then sent to the alarm classification module.

[0020] Furthermore, the implementation of the data processing module includes the following steps:

[0021] The data processing module copies the single data entries packaged by the data receiving and filtering module, dividing them into data required for alarm escalation and data required for alarm downgrade.

[0022] The data processing module places the data required for alarm escalation into an alarm escalation time window of a specified size according to user-defined rules, and places the data required for alarm downgrade into an alarm downgrade time window of a specified size according to user-defined rules.

[0023] A batch of data will pop up when the window period of the data processing module expires;

[0024] The data processing module determines the alarm level of the batch of information according to the user-defined alarm rules. If multiple alarm levels are met, the alarm downgrade and alarm escalation processes will prioritize the highest alarm level. The alarm rule template includes data range, data trend, and commonly used aggregate functions.

[0025] The data processing module packages the data according to whether the alarm level is generated by the alarm escalation window or the alarm degrade window, specifying whether the level is required for alarm degrade or alarm escalation;

[0026] The data processing module sends the packaged data to the alarm classification module.

[0027] The alarm rule template includes data range, data trend, and commonly used aggregation functions. The alarm rule template allows users to set alarm levels based on data range, data trend, and commonly used aggregation functions. The specific alarm level to be matched is determined by the alarm information defined by the user in the alarm template. If the message in the window matches multiple alarm levels, the alarm with the highest matching alarm level will be sent to the alarm classification module.

[0028] Preferably, the data processing module uses the following default comparison rule:

[0029] An alarm is triggered when the percentage of user-defined alarm data multiplied by the total number of alarm data is less than the total number of alarm data.

[0030] Preferably, when new alarm data and alarm levels are sent, the alarm classification module compares them with the alarm levels stored in the map. The upgrade window of the data processing module only performs the logic of alarm upgrade, and the downgrade window only performs the logic of alarm downgrade.

[0031] If there is a change in alarm level, the new alarm level and alarm data will be sent to the alarm result publishing module. If the alarm level reaches the maximum limit, the alarm rating module will send a signal to the collection program to terminate the collection of data from the device.

[0032] Furthermore, the implementation of the alarm classification module includes the following steps:

[0033] The alarm classification module parses the data sent by the data processing module, which is divided into alarm escalation data and alarm downgrade data;

[0034] Alarm escalation process: If the alarm level of the data is greater than the current alarm level, save the level and send the batch of data to the alarm result publishing module; if the alarm level reaches the highest level, stop the data collection behavior of the device and wait for maintenance personnel to maintain it.

[0035] Alarm degradation process: If the alarm level of the data is lower than the current alarm level, save the level and send the batch of data to the alarm result publishing module.

[0036] Furthermore, the log information module includes the following functions:

[0037] Based on the preset parameters such as info, debug, error, and others in the log information module, the corresponding anomaly detection records are output.

[0038] If there is an error message, the operations and maintenance personnel need to check why the data is showing an error and how to resolve it;

[0039] The info and debug information are designed to help operations and maintenance personnel view the project's running status, workflow, and the logic behind error messages.

[0040] This invention also claims a system for real-time monitoring and alarming of abnormal data, including a data receiving and filtering module, a data processing module, an alarm classification module, an alarm result publishing module, and a log information module.

[0041] The data receiving and filtering module is used to receive status data from the acquisition devices in the entire system, filter the data, and only package the data of the monitored devices according to a specific time window and send it to the data processing module.

[0042] The data processing module is used to calculate the data sent by the data receiving module according to the user-defined alarm rules to determine whether the monitored device is in an abnormal state during this period, and to send the data of the device in an abnormal state to the alarm classification module.

[0043] The alarm classification module compares the alarm information of each device with the existing level to determine whether to downgrade or upgrade, thereby determining the current alarm level of the device, and sending the latest alarm level and alarm data of the device to the alarm result publishing module.

[0044] The alarm result publishing module publishes the new alarm level of the device to all users and maintenance personnel who have subscribed to this message, so as to facilitate timely discovery of the problem and avoid the disaster that may be caused by prolonged equipment failure.

[0045] The log information module determines whether the error is due to a client-sent request error or a component error based on the log exception information from each module, and returns the error information to the user for alerting and handling.

[0046] This system implements the aforementioned methods for real-time monitoring and alarming of abnormal data.

[0047] Compared with the prior art, the method and system for real-time monitoring and alarming of abnormal data of the present invention have the following advantages:

[0048] 1. Since all modules are always on, the cause and timing of device malfunctions can be known promptly.

[0049] 2. When the alarm level reaches a critical level, the alarm classification module will pause all data acquisition programs of the device to prevent erroneous data from affecting the subsequent model building and the normal operation of the entire system.

[0050] 3. The log information module can effectively help operations and maintenance personnel view the system status and the cause of errors by outputting logs for each part of the logic contained in the log.

[0051] 4. No need for maintenance personnel to manually monitor the program for extended periods. If an error occurs, there will be log alerts and warnings, reducing the maintenance time and costs for maintenance personnel.

[0052] 5. The alarm classification module maintains the alarm status of all devices in a unified manner, ensuring the atomicity of alarm information throughout the entire system.

[0053] 6. The data processing module segments the data according to a specific time window and sets an alarm percentage to effectively avoid random values.

[0054] 7. The data processing module uses a large number of coroutines to meet the concurrency of large amounts of data and avoid program blocking due to excessive data. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the implementation of a method for real-time monitoring and alarming of abnormal data according to an embodiment of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to specific embodiments.

[0057] This invention provides a method for real-time monitoring and alarming of abnormal data. This method ensures the entire system operates normally by filtering the collected data. For example, in the Industrial Internet, photovoltaic panels continuously generate electricity based on sunlight. Our program collects data such as the status and power generation of each photovoltaic panel. If a photovoltaic panel suddenly malfunctions—for instance, if its normal power generation is between 30-50 kilowatts, but the program suddenly collects data of 0 or 10 kilowatts—it indicates that the panel is not functioning correctly, allowing users and maintenance personnel to promptly identify and resolve the problem. The method includes a data receiving and filtering module, a data processing module, an alarm classification module, an alarm result publishing module, and a log information module.

[0058] The data receiving and filtering module is primarily responsible for aggregating all data collected by all devices at the acquisition end and categorizing it by device name. Then, according to the monitoring rules set by the user, it sends the data from the monitored devices to the data processing module.

[0059] The data processing module allows users to customize alarm rules according to alarm rule templates. After segmenting data into pieces based on specific time windows, the module compares and calculates data according to these user-defined rules. The default comparison rule is: an alarm is triggered when the user-defined percentage of alarm data × total number of data points < total number of alarm data points. Furthermore, the alarm rule templates allow users to set alarm levels based on data range, data trends, and commonly used aggregation functions. The specific alarm level matched is determined by the alarm information defined in the alarm template. If a message within a window matches multiple alarm levels, the highest matching alarm level is sent to the alarm classification module.

[0060] The data processing module copies the data sent by the filtering module according to the user-specified time window, creating an escalation window and a degrade window. To avoid randomness, this module calculates the data in the specified window according to user-defined alarm rules. For example, if the user defines an alarm as a device when 60% of the total data is incorrect, then if there are ten data entries and more than six are incorrect, the device is considered to be in an alarm state. The current alarm level and alarm data are then sent to the alarm classification module.

[0061] The alarm classification module maintains a map storing the alarm levels of all devices currently in the data acquisition system. This map caches the latest alarm levels for all devices. When new alarm data and alarm levels arrive, they are compared with the alarm levels stored in the map. The data processing module's upgrade window only handles alarm upgrade logic, and the downgrade window only handles alarm downgrade logic. If there is a change in alarm level, the new alarm level and alarm data are sent to the alarm result publishing module. Furthermore, if the alarm level reaches the maximum limit, the alarm classification module will send a signal to the data collection program to terminate the collection of data from that device, preventing device errors from causing the entire program to malfunction.

[0062] The alarm result publishing module is responsible for distributing new device alarms to users who have subscribed to the device's alarm information and maintenance personnel, so that relevant personnel can obtain the latest status of the device in a timely manner.

[0063] The log information module analyzes the program's running status and process based on the log information from each module, and clearly feeds back error information to the operations and maintenance personnel to facilitate their analysis.

[0064] The implementation of the data receiving and filtering module includes the following steps:

[0065] The receiving submodule receives information collected from various devices;

[0066] The filtering submodule filters the information, saves and packages the information of the points that are monitored by the user, and discards the data of devices that are not monitored by the user.

[0067] The filtering submodule sends the packaged data to the data processing module.

[0068] The implementation of the data processing module includes the following steps:

[0069] The data processing module copies the single data entries packaged by the data receiving and filtering module, dividing them into data required for alarm escalation and data required for alarm downgrade.

[0070] The data processing module places the data required for alarm escalation into an alarm escalation time window of a specified size according to user-defined rules, and places the data required for alarm downgrade into an alarm downgrade time window of a specified size according to user-defined rules.

[0071] A batch of data will pop up when the window period of the data processing module expires;

[0072] The data processing module determines the alarm level of the batch of information according to the user-defined alarm rules. If multiple alarm levels are met, the alarm downgrade and alarm escalation processes will prioritize the highest alarm level. The alarm rule template includes data range, data trend, and commonly used aggregate functions.

[0073] The data processing module packages the data according to whether the alarm level is generated by the alarm escalation window or the alarm degrade window, specifying whether the level is required for alarm degrade or alarm escalation;

[0074] The data processing module sends the packaged data to the alarm classification module.

[0075] The implementation of the alarm classification module includes the following steps:

[0076] The alarm classification module parses the data sent by the data processing module, which is divided into alarm escalation data and alarm downgrade data;

[0077] Alarm escalation process: If the alarm level of the data is greater than the current alarm level, save the level and send the batch of data to the alarm result publishing module; if the alarm level reaches the highest level, stop the data collection behavior of the device and wait for maintenance personnel to maintain it.

[0078] Alarm degradation process: If the alarm level of the data is lower than the current alarm level, save the level and send the batch of data to the alarm result publishing module.

[0079] The log information module includes the following functions:

[0080] Based on the preset parameters such as info, debug, error, and others in the log information module, the corresponding anomaly detection records are output.

[0081] If there is an error message, the operations and maintenance personnel need to check why the data is showing an error and how to resolve it;

[0082] The info and debug information are designed to help operations and maintenance personnel view the project's running status, workflow, and the logic behind error messages.

[0083] This method ensures the entire system operates normally by filtering the real-time collected data. For issues where equipment malfunctions cannot be detected promptly during project maintenance, it provides a method for real-time detection and timely feedback, and relays anomalies from each module to maintenance personnel for handling.

[0084] This invention also provides a system for real-time monitoring and alarming of abnormal data, including a data receiving and filtering module, a data processing module, an alarm classification module, an alarm result publishing module, and a log information module.

[0085] The data receiving and filtering module is used to receive status data from the acquisition devices in the entire system, filter the data, and only package the data of the monitored devices according to a specific time window and send it to the data processing module.

[0086] The data processing module is used to calculate the data sent by the data receiving module according to the user-defined alarm rules to determine whether the monitored device is in an abnormal state during this period, and to send the data of the device in an abnormal state to the alarm classification module.

[0087] The alarm classification module compares the alarm information of each device with the existing level to determine whether to downgrade or upgrade, thereby determining the current alarm level of the device, and sending the latest alarm level and alarm data of the device to the alarm result publishing module.

[0088] The alarm result publishing module publishes the new alarm level of the device to all users and maintenance personnel who have subscribed to this message, so as to facilitate timely discovery of the problem and avoid the disaster that may be caused by prolonged equipment failure.

[0089] The log information module determines whether the error is due to a client-sent request error or a component error based on the log exception information from each module, and returns the error information to the user for alerting and handling.

[0090] This system implements the real-time monitoring and alarm method for abnormal data described in the above embodiments.

[0091] The implementation of the data receiving and filtering module includes the following steps:

[0092] The receiving submodule receives information collected from various devices;

[0093] The filtering submodule filters the information, saves and packages the information of the points that are monitored by the user, and discards the data of devices that are not monitored by the user.

[0094] The filtering submodule sends the packaged data to the data processing module.

[0095] The implementation of the data processing module includes the following steps:

[0096] The data processing module copies the single data entries packaged by the data receiving and filtering module, dividing them into data required for alarm escalation and data required for alarm downgrade.

[0097] The data processing module places the data required for alarm escalation into an alarm escalation time window of a specified size according to user-defined rules, and places the data required for alarm downgrade into an alarm downgrade time window of a specified size according to user-defined rules.

[0098] A batch of data will pop up when the window period of the data processing module expires;

[0099] The data processing module determines the alarm level of the batch of information according to the user-defined alarm rules. If multiple alarm levels are met, the alarm downgrade and alarm escalation processes will prioritize the highest alarm level. The alarm rule template includes data range, data trend, and commonly used aggregate functions.

[0100] The data processing module packages the data according to whether the alarm level is generated by the alarm escalation window or the alarm degrade window, specifying whether the level is required for alarm degrade or alarm escalation;

[0101] The data processing module sends the packaged data to the alarm classification module.

[0102] The implementation of the alarm classification module includes the following steps:

[0103] The alarm classification module parses the data sent by the data processing module, which is divided into alarm escalation data and alarm downgrade data;

[0104] Alarm escalation process: If the alarm level of the data is greater than the current alarm level, save the level and send the batch of data to the alarm result publishing module; if the alarm level reaches the highest level, stop the data collection behavior of the device and wait for maintenance personnel to maintain it.

[0105] Alarm degradation process: If the alarm level of the data is lower than the current alarm level, save the level and send the batch of data to the alarm result publishing module.

[0106] The log information module includes the following functions:

[0107] Based on the preset parameters such as info, debug, error, and others in the log information module, the corresponding anomaly detection records are output.

[0108] If there is an error message, the operations and maintenance personnel need to check why the data is showing an error and how to resolve it;

[0109] The info and debug information are designed to help operations and maintenance personnel view the project's running status, workflow, and the logic behind error messages.

[0110] Through the specific embodiments described above, those skilled in the art can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. Based on the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features to achieve different technical solutions.

[0111] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.

Claims

1. A method for real-time monitoring and alarming of abnormal data, characterized in that, The implementation of this method includes a data receiving and filtering module, a data processing module, an alarm classification module, an alarm result publishing module, and a log information module. The data receiving and filtering module is used to collect data from all devices at the current acquisition end and to organize and classify all the data; The data processing module allows users to customize alarm rules according to alarm rule templates. After the data is segmented according to time windows, it is compared and calculated according to the user-defined alarm rules. The alarm classification module maintains a map of the alarm levels of all devices at the current acquisition end. This map caches the latest alarm levels of all devices at the current acquisition end. When the alarm level of a device reaches a set level, the alarm classification module will send a signal to the device to stop collecting data. The alarm result publishing module is responsible for distributing new device alarms to users who have subscribed to the device's alarm information and maintenance personnel. The log information module analyzes the program's running status and process based on the log information from each module, and clearly feeds back error information to the operation and maintenance personnel. The implementation of the data processing module includes the following steps: The data processing module copies the single data entries packaged by the data receiving and filtering module, dividing them into data required for alarm escalation and data required for alarm downgrade. The data processing module places the data required for alarm escalation into an alarm escalation time window of a specified size according to user-defined rules, and places the data required for alarm downgrade into an alarm downgrade time window of a specified size according to user-defined rules. A batch of data will pop up when the window period of the data processing module expires; The data processing module determines the alarm level of the batch of data according to the user-defined alarm rules. If multiple alarm levels are met, the highest alarm level is used in both the alarm downgrade and alarm escalation processes. The alarm rule template includes data range, data trend, and commonly used aggregate functions. The data processing module packages the data according to whether the alarm level is generated by the alarm escalation window or the alarm degrade window, specifying whether the level is required for alarm degrade or alarm escalation; The data processing module sends the packaged data to the alarm classification module.

2. The method for real-time monitoring and alarming of abnormal data according to claim 1, characterized in that, The data receiving and filtering module aggregates all data collected by all devices at the acquisition end and categorizes them according to device name; then, according to the monitoring rules set by the user, it sends the data of the monitored devices to the data processing module.

3. A method for real-time monitoring and alarming of abnormal data according to claim 1 or 2, characterized in that, The implementation of the data receiving and filtering module includes the following steps: The receiving submodule receives information collected from various devices; The filtering submodule filters the information, saves and packages the information of the points that are monitored by the user, and discards the data of devices that are not monitored by the user. The filtering submodule sends the packaged data to the data processing module.

4. The method for real-time monitoring and alarming of abnormal data according to claim 3, characterized in that, The data processing module copies the data sent by the filtering submodule according to the time window specified by the user, and copies it into an upgrade window and a downgrade window; The data processing module calculates the data in the specified window according to the user-defined alarm rules. If the user defines that when the error data reaches 60% of the total data, it is judged that the device is in an alarm state. Then, if there are ten data entries and more than six error data entries, the device is judged to be in an alarm state. The alarm level of the current device and the alarm data are then sent to the alarm classification module.

5. The method for real-time monitoring and alarming of abnormal data according to claim 1, characterized in that, The default comparison rule for the data processing module is: An alarm is triggered when the percentage of user-defined alarm data multiplied by the total number of alarm data is less than the total number of alarm data.

6. The method for real-time monitoring and alarming of abnormal data according to claim 1, characterized in that, When new alarm data and alarm levels are sent, the alarm classification module compares them with the alarm levels stored in the map. The upgrade window of the data processing module only performs the logic of alarm upgrade, and the downgrade window only performs the logic of alarm downgrade. If there is a change in alarm level, the new alarm level and alarm data will be sent to the alarm result publishing module. If the alarm level reaches the maximum limit, the alarm rating module will send a signal to the collection program to terminate the collection of data from the device.

7. A method for real-time monitoring and alarming of abnormal data according to claim 1 or 6, characterized in that, The implementation of the alarm classification module includes the following steps: The alarm classification module parses the data sent by the data processing module, which is divided into alarm escalation data and alarm downgrade data; Alarm escalation process: If the alarm level of the data is greater than the current alarm level, save the level and send the batch of data to the alarm result publishing module; if the alarm level reaches the highest level, stop the data collection behavior of the device and wait for maintenance personnel to maintain it. Alarm degradation process: If the alarm level of the data is lower than the current alarm level, save the level and send the batch of data to the alarm result publishing module.

8. The method for real-time monitoring and alarming of abnormal data according to claim 1, characterized in that, The log information module includes the following functions: Based on the preset parameters such as info, debug, error, and others in the log information module, the corresponding anomaly detection records are output. If there is an error message, the operations and maintenance personnel need to check why the data is showing an error and how to resolve it; The info and debug information are designed to help operations and maintenance personnel view the project's running status, workflow, and the logic behind error messages.

9. A system for real-time monitoring and alarming of abnormal data, characterized in that, It includes a data receiving and filtering module, a data processing module, an alarm classification module, an alarm result publishing module, and a log information module. The data receiving and filtering module is used to receive the status data of the acquisition devices in the entire system, filter the data, and only package the data of the monitored devices according to the time window and send it to the data processing module. The data processing module is used to calculate the data sent by the data receiving module according to the user-defined alarm rules to determine whether the monitored device is in an abnormal state during this period, and to send the data of the device in an abnormal state to the alarm classification module. The alarm classification module compares the alarm information of each device with the existing level to determine whether to downgrade or upgrade, thereby determining the current alarm level of the device, and sending the latest alarm level and alarm data of the device to the alarm result publishing module. The alarm result publishing module publishes the new alarm level of the device to all users and maintenance personnel who have subscribed to this message, so as to facilitate timely discovery of the problem and avoid the disaster that may be caused by prolonged equipment failure. The log information module determines whether the error is due to a client-sent request error or a component error based on the log exception information from each module, and returns the error information to the user for alerting and processing. The system implements the method for real-time monitoring and alarm of abnormal data as described in any one of claims 1 to 8.

Images