Equipment exception handling method and system based on multi-modal data linkage

Abstract

The invention relates to the technical field of equipment exception handling, and discloses an equipment exception handling system based on multi-modal data linkage, which comprises a sensor, a multi-modal exception detection and trigger module, a lightweight AI model and the like, the invention further provides an equipment exception handling method based on multi-modal data linkage, and the method comprises the following steps: S1, parameter monitoring: collecting operation parameter data and position information data of equipment through a sensor; compared with a traditional system which needs to separately collect equipment data and video data from multiple sources for analysis and response, the method has the advantages that the whole process from abnormal triggering to data analysis to control completion is greatly accelerated, and meanwhile, the false alarm rate can be greatly reduced through multi-modal data fusion; the root cause analysis accuracy is greatly optimized compared with that of a traditional analysis method.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment anomaly handling, and particularly to a method and system for equipment anomaly handling based on multi-modal data linkage. Background Art

[0002] An equipment anomaly handling system is a technical system for monitoring the operating status of equipment, detecting abnormal situations and taking corresponding handling measures. Its core functions include anomaly detection, handling, recording and feedback, aiming to improve equipment reliability, reduce failure rates and maintenance costs. In existing factories, an equipment anomaly handling system is used to monitor the equipment in real time to ensure the accuracy and real-time update of all equipment information and related data.

[0003] Currently, the following problems usually exist in existing equipment anomaly handling systems:

[0004] Response lag: After an anomaly alarm is triggered, modules such as the reverse control module and video monitoring need to be manually intervened to be linked, resulting in too long response time and prone to the situation that the alarm cannot be processed in time.

[0005] Data isolation: Equipment data, video images, location information, etc. are scattered in different subsystems and cannot be analyzed in real-time association. It is difficult to determine equipment failures, and even if determined, there is usually a high false positive rate.

[0006] Static rule dependence: Equipment anomaly judgment is usually based on preset thresholds (such as exceeding the temperature threshold, electronic fence crossing, etc.), and it is impossible to adjust the control strategy in time by combining real-time video images and the dynamic situation of equipment status. Summary of the Invention

[0007] (1) Technical problems to be solved

[0008] Aiming at the deficiencies of the prior art, the present invention provides a method and system for equipment anomaly handling based on multi-modal data linkage, which solves the problems of response lag: after an anomaly alarm is triggered, modules such as the reverse control module and video monitoring need to be manually intervened to be linked, resulting in too long response time and prone to the situation that the alarm cannot be processed in time; data isolation: equipment data, video images, location information, etc. are scattered in different subsystems and cannot be analyzed in real-time association. It is difficult to determine equipment failures, and even if determined, there is usually a high false positive rate; static rule dependence: equipment anomaly judgment is usually based on preset thresholds (such as exceeding the temperature threshold, electronic fence crossing, etc.), and it is impossible to adjust the control strategy in time by combining real-time video images and the dynamic situation of equipment status.

[0009] (2) Technical solutions

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] An equipment anomaly handling system based on multi-modal data linkage, comprising a sensor, a multi-modal anomaly detection and triggering module, a lightweight AI model, an edge computing node verification module, a cross-system intelligent linkage module, a multi-source data-driven root cause analysis module, and a dynamic policy optimization module. The sensor collects the operation parameters of the equipment in real time. The sensor is connected to the multi-modal anomaly detection and triggering module for transmitting the operation parameters of the equipment collected by the sensor to the multi-modal anomaly detection and triggering module. The lightweight AI model is arranged on the side of the gateway. The lightweight AI model is connected to the multi-modal anomaly detection and triggering module for receiving the data of the multi-modal anomaly detection and triggering module and performing secondary verification on it. The lightweight AI model is connected to the edge computing node verification module. The cross-system intelligent linkage module is connected to an external camera for controlling the camera. The multi-source data-driven root cause analysis module is connected to the edge computing node verification module. The dynamic policy optimization module is connected to the multi-source data-driven root cause analysis module.

[0012] Further, the parameters collected by the sensor during the operation of the equipment include any one or more of temperature, vibration, pressure, current, and voltage.

[0013] On the basis of the foregoing solution, the cross-system intelligent linkage module is further connected to the multi-source data-driven root cause analysis and edge computing node verification module for transmitting the data of the camera to the multi-source data-driven root cause analysis and edge computing node verification module.

[0014] The present invention also proposes a method for handling equipment anomalies based on multi-modal data linkage, comprising the following steps:

[0015] S1: Parameter monitoring, collecting the operation parameter data and location information data of the equipment through the sensor;

[0016] S2: Anomaly detection and triggering, based on historical data and real-time environmental data, dynamically adjusting the anomaly threshold using the sliding window algorithm to define the range of the electronic fence rule, and then transmitting the operation parameter data and location data collected by the sensor in real time to the multi-modal anomaly detection and triggering module, triggering a primary anomaly event in combination with the electronic fence rule;

[0017] S3: Anomaly confirmation, deploying a lightweight AI model on the gateway side to perform secondary verification on the primary anomaly event, filtering out noise information. If the anomaly is confirmed, the anomaly data is transmitted to the edge computing node verification module, and then a standardized anomaly event message is generated by the edge computing node verification module and pushed to the system;

[0018] S4: Configure the camera, automatically retrieve the surrounding cameras through the cross-system intelligent linkage module according to the device location information, capture video segments, and intercept the video for 50 - 70 seconds when an anomaly is triggered. Analyze the video data through AI, use a video detection model based on YOLOv7 to detect key target elements in the picture, combine a behavior recognition model based on OpenPose to recognize human behaviors, and perform operations according to the analysis results;

[0019] S5: Build a model, input the device parameters, sensor data, video analysis results, and historical maintenance records into the root cause analysis module driven by multi-source data, and through the spatio-temporal graph neural network based on ST-GNN in the root cause analysis module driven by multi-source data, build an anomaly propagation path model for multi-source data correlation analysis, and locate the root cause through a causal reasoning engine based on Do-Calculus;

[0020] S6: Policy optimization, transmit the located root cause data to the dynamic policy optimization module, and the dynamic policy optimization module performs according to the analysis results: automatically update the control rule library and detection threshold or generate a maintenance work order for the device lacking maintenance and push it to the person in charge, either of the operations.

[0021] As a further solution of the present invention, the sliding window algorithm in S2 is to set a fixed-size and continuous subsequence window, maintain the window, and then, in a sliding window manner, continuously update the position and size of the window.

[0022] Further, the content of the standardized anomaly event message in S2 includes the device ID, anomaly type, and timestamp.

[0023] On the basis of the foregoing solution, the operations performed according to the analysis results in S4 include: fusing the analysis results with device data, generating a multi-modal anomaly report, marking the risk level, or initiating a reverse control action on the device according to the video analysis results.

[0024] As a further solution of the present invention, the key target elements in S5 include any one or two of the people and vehicles in the picture, and the content of the human behavior recognition includes any one or two of climbing the device and disassembly actions.

[0025] The beneficial effects of the present invention are:

[0026] 1. Compared with the traditional system that needs to separately collect device data and video data from multiple sources and then analyze and respond, the full process of this method from anomaly trigger to data analysis to completion of control is greatly accelerated.

[0027] 2. Through multi-modal data fusion, the false alarm rate of the present invention can be significantly reduced, and the accuracy of root cause analysis is also greatly optimized compared with traditional analysis methods.

[0028] 3. The system of the present invention can continuously and dynamically adjust and optimize rules and strategies through accurate root cause analysis of problems, enabling the system to have stronger resilience to various types of faults and problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the system framework of an equipment anomaly handling system based on multi-modal data linkage of the present invention;

[0030] Figure 2 It is a schematic diagram of the process structure of an equipment anomaly handling method based on multi-modal data linkage of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that unless otherwise clearly specified and limited, the terms "installation", "connection", and "setting" should be understood in a broad sense. For those of ordinary skill in the art, the specific meanings of the above terms in this patent can be understood according to specific circumstances.

[0032] Embodiment 1

[0033] Refer to Figure 1 - Figure 2 , an equipment anomaly handling system based on multi-modal data linkage, including a sensor, a multi-modal anomaly detection and triggering module, a lightweight AI model, an edge computing node verification module, a cross-system intelligent linkage module, a multi-source data-driven root cause analysis module, and a dynamic policy optimization module. The sensor collects the operating parameters of the equipment in real time. The sensor is connected to the multi-modal anomaly detection and triggering module for transmitting the equipment operating parameters collected by the sensor to the multi-modal anomaly detection and triggering module. The lightweight AI model is set on the side of the gateway. The lightweight AI model is connected to the multi-modal anomaly detection and triggering module for receiving the data from the multi-modal anomaly detection and triggering module and performing secondary verification on it. The lightweight AI model is connected to the edge computing node verification module. The cross-system intelligent linkage module is connected to an external camera for controlling the camera. The multi-source data-driven root cause analysis module is connected to the edge computing node verification module. The dynamic policy optimization module is connected to the multi-source data-driven root cause analysis module. This system can continuously and dynamically adjust and optimize rules and strategies through accurate root cause analysis of problems, enabling the system to have stronger resilience to various types of faults and problems.

[0034] In the present invention, the sensor is used to collect parameters during the operation of the device, including temperature and vibration. The cross-system intelligent linkage module is also connected to the multi-source data-driven root cause analysis and edge computing node verification module, and is used to transmit the data of the camera to the multi-source data-driven root cause analysis and edge computing node verification module.

[0035] The present invention also proposes a method for handling device anomalies based on multi-modal data linkage, including the following steps:

[0036] S1: Parameter monitoring, collecting the operation parameter data and location information data of the device through the sensor;

[0037] S2: Anomaly detection and triggering, based on historical data and real-time environmental data, using the sliding window algorithm to dynamically adjust the anomaly threshold to define the range of the electronic fence rule, and then transmitting the operation parameter data and location data collected by the sensor in real time to the multi-modal anomaly detection and triggering module, and triggering a primary anomaly event in combination with the electronic fence rule;

[0038] S3: Anomaly confirmation, by deploying a lightweight AI model on the gateway side, performing secondary verification on the primary anomaly event, filtering out noise information, if the anomaly is confirmed, transmitting the anomaly data to the edge computing node verification module, and then generating a standardized anomaly event message through the edge computing node verification module and pushing it to the system;

[0039] S4: Deploying the camera, automatically retrieving the surrounding cameras through the cross-system intelligent linkage module according to the device location information, capturing video segments, and intercepting 50s of video when the anomaly is triggered. Analyze the video data through AI, use a video detection model based on YOLOv7 to detect key target elements in the picture, and combine a behavior recognition model based on OpenPose to recognize the behavior of personnel, and perform operations according to the analysis results;

[0040] S5: Building a model, inputting device parameters, sensor data, video analysis results, and historical maintenance records into the multi-source data-driven root cause analysis module, and through the spatio-temporal graph neural network based on ST-GNN in the multi-source data-driven root cause analysis module, building an anomaly propagation path model for multi-source data correlation analysis, and locating the root cause through a causal reasoning engine based on Do-Calculus. The false alarm rate can be greatly reduced through multi-modal data fusion, and the root cause analysis accuracy is also greatly optimized compared with traditional analysis methods;

[0041] S6: Policy optimization. Transmit the located root cause data to the dynamic policy optimization module. The dynamic policy optimization module executes according to the root cause analysis result: generate a maintenance work order for the equipment lacking maintenance and push it to the person in charge. Compared with the traditional system that needs to separately collect equipment data and video data from multiple sources and then analyze and respond, the whole process from anomaly triggering to data analysis to completion of control is greatly accelerated in this method.

[0042] In particular, the sliding window algorithm in S2 sets a fixed-size and continuous subsequence window and maintains the window. Then, in the way of sliding the window, continuously update the position and size of the window. The content of the standardized anomaly event message in S2 includes the equipment ID, anomaly type, and timestamp. The operations performed according to the analysis result in S4 include: initiating a reverse control action on the equipment according to the result of video analysis. The key target elements in S5 include the personnel in the picture, and the content of personnel behavior recognition includes climbing the equipment.

[0043] Embodiment 2

[0044] Refer to Figure 1 - Figure 2 , a device anomaly handling system based on multi-modal data linkage, including sensors, a multi-modal anomaly detection and triggering module, a lightweight AI model, an edge computing node verification module, a cross-system intelligent linkage module, a multi-source data-driven root cause analysis module, and a dynamic policy optimization module. The sensors collect the operating parameters of the device in real time. The sensors are connected to the multi-modal anomaly detection and triggering module for transmitting the device operating parameters collected by the sensors to the multi-modal anomaly detection and triggering module. The lightweight AI model is set on the side of the gateway. The lightweight AI model is connected to the multi-modal anomaly detection and triggering module for receiving the data of the multi-modal anomaly detection and triggering module and performing secondary verification on it. The lightweight AI model is connected to the edge computing node verification module. The cross-system intelligent linkage module is connected to an external camera for controlling the camera. The multi-source data-driven root cause analysis module is connected to the edge computing node verification module. The dynamic policy optimization module is connected to the multi-source data-driven root cause analysis module. This system can continuously and dynamically adjust and optimize rules and policies through accurate problem root cause analysis, making the system more resilient to various types of faults and problems.

[0045] In the present invention, the sensors are used to collect parameters during the operation of the device, including temperature, vibration, pressure, current, and voltage. The cross-system intelligent linkage module is also connected to the multi-source data-driven root cause analysis and edge computing node verification module for transmitting the data of the camera to the multi-source data-driven root cause analysis and edge computing node verification module.

[0046] The present invention also proposes a device anomaly handling method based on multi-modal data linkage, including the following steps:

[0047] S1: Parameter monitoring, collecting the operation parameter data and location information data of the device through sensors;

[0048] S2: Anomaly detection and triggering, based on historical data and real-time environmental data, using the sliding window algorithm to dynamically adjust the anomaly threshold to define the range of the electronic fence rule, and then transmitting the operation parameter data and location data collected by the sensor in real time to the multi-modal anomaly detection and triggering module, and triggering the primary anomaly event in combination with the electronic fence rule;

[0049] S3: Anomaly confirmation, by deploying a lightweight AI model on the gateway side, performing secondary verification on the primary anomaly event, filtering out noise information, if the anomaly is confirmed, transmitting the anomaly data to the edge computing node verification module, and then generating a standardized anomaly event message through the edge computing node verification module and pushing it to the system;

[0050] S4: Deploying cameras, automatically retrieving the surrounding cameras through the cross-system intelligent linkage module according to the device location information, capturing video segments, and intercepting 70s of video at the time of anomaly triggering, analyzing the video data through AI, using the video detection model based on YOLOv7 to detect the key target elements in the picture, and combining the behavior recognition model based on OpenPose to recognize the human behavior, and performing operations according to the analysis results;

[0051] S5: Building a model, feeding the device parameters, sensor data, video analysis results, and historical maintenance records into the multi-source data-driven root cause analysis module, and through the spatio-temporal graph neural network based on ST-GNN in the multi-source data-driven root cause analysis module, building an anomaly propagation path model for multi-source data correlation analysis, and locating the root cause through the causal reasoning engine based on Do-Calculus. The false alarm rate can be greatly reduced through multi-modal data fusion, and the root cause analysis accuracy is also greatly optimized compared with traditional analysis methods;

[0052] S6: Policy optimization, transmitting the located root cause data to the dynamic policy optimization module, and the dynamic policy optimization module performs the following operations according to the root cause analysis result: automatically updating the control rule library and detection threshold. Compared with the traditional system that needs to separately collect device data and video data from multiple sources and analyze and respond, the whole process from anomaly triggering to data analysis to control completion in this method is greatly accelerated.

[0053] In particular, the sliding window algorithm in S2 sets a fixed-size and continuous subsequence window, maintains the window, and then, by sliding the window, continuously updates the position and size of the window. The content of the standardized abnormal event message in S2 includes the device ID, abnormal type, and timestamp. The operations performed according to the analysis results in S4 include: fusing the analysis results with device data, generating a multimodal abnormal report, and marking the risk level. The key target elements in S5 include the personnel and vehicles in the picture, and the content of personnel behavior recognition includes climbing equipment and disassembly actions

[0054] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A device exception handling system based on multi-modal data linkage, characterized in that: It includes a sensor, a multimodal anomaly detection and triggering module, a lightweight AI model, an edge computing node verification module, a cross-system intelligent linkage module, a multi-source data-driven root cause analysis module and a dynamic strategy optimization module. The sensor collects the operating parameters of the equipment in real time. The sensor is connected to the multimodal anomaly detection and triggering module to transmit the equipment operating parameters collected by the sensor to the multimodal anomaly detection and triggering module. The lightweight AI model is arranged on the side of the gateway. The lightweight AI model is connected to the multimodal anomaly detection and triggering module to receive the data of the multimodal anomaly detection and triggering module and perform secondary verification on it. The lightweight AI model is connected to the edge computing node verification module. The cross-system intelligent linkage module is connected to an external camera for controlling the camera. The multi-source data-driven root cause analysis module is connected to the edge computing node verification module, and the dynamic strategy optimization module is connected to the multi-source data-driven root cause analysis module.

2. According to claim 1, a device abnormality handling system based on multimodal data linkage is characterized in that: The sensor is used to collect parameters during the operation of the device, including any one or more of temperature, vibration, pressure, current and voltage.

3. The device abnormality handling system based on multimodal data linkage according to claim 2 is characterized in that: The cross-system intelligent linkage module is also connected to the multi-source data-driven root cause analysis and edge computing node verification module, and is used to transmit the camera data to the multi-source data-driven root cause analysis and edge computing node verification module.

4. A device exception handling method based on multi-modal data linkage, characterized in that: The following steps are involved: S1: Parameter monitoring, collecting equipment operating parameter data and equipment location information data through sensors; S2: Anomaly detection and triggering, based on historical data and real-time environmental data, the sliding window algorithm is used to dynamically adjust the anomaly threshold to define the scope of the electronic fence rule, and then the operating parameter data and location data collected by the sensor in real time are transmitted to the multimodal anomaly detection and triggering module, which triggers the primary anomaly event in combination with the electronic fence rule; S3: Abnormal confirmation, by deploying a lightweight AI model on the gateway side, performing secondary verification on primary abnormal events and filtering noise information. If the abnormality is confirmed, the abnormal data is transmitted to the edge computing node verification module, and then the edge computing node verification module generates a standardized abnormal event message and pushes it to the system; S4: Deploy cameras, automatically call surrounding cameras through the cross-system intelligent linkage module according to the device location information, capture video clips, and intercept 50-70 seconds of video when the abnormality is triggered. Analyze the video data through AI, use the YOLOv7-based video detection model to detect key target elements in the picture, and combine the OpenPose-based behavior recognition model to identify human behavior, and perform operations based on the analysis results; S5: Build a model to deliver equipment parameters, sensor data, video analysis results, and historical maintenance records to the root cause analysis module driven by multi-source data. Use the spatiotemporal graph neural network based on ST-GNN in the root cause analysis module driven by multi-source data to build an abnormal propagation path model for multi-source data association analysis, and locate the root cause through the causal reasoning engine based on Do-Calculus. S6: Strategy optimization, transmitting the located root cause data to the dynamic strategy optimization module. The dynamic strategy optimization module performs any operation based on the cause analysis results: automatically updating the control rule library and detection threshold or generating a maintenance work order for the equipment that lacks maintenance and pushing it to the person in charge.

5. The device abnormality handling method based on multi-modal data linkage according to claim 4 is characterized in that: The sliding window algorithm in S2 is to set a fixed-size and continuous subsequence window and maintain the window, and then continuously update the position and size of the window by sliding the window.

6. The device abnormality handling method based on multi-modal data linkage according to claim 5 is characterized in that: The content of the standardized abnormal event message in S2 includes device ID, abnormality type and timestamp.

7. The device abnormality handling method based on multimodal data linkage according to claim 4 is characterized in that: The operations performed according to the analysis results in S4 include: fusing the analysis results with the device data, generating a multimodal abnormality report, marking the risk level, or initiating a reverse control action on the device according to the results of the video analysis.

8. The device abnormality handling method based on multi-modal data linkage according to claim 7 is characterized in that: The key target elements in S5 include any one or both of the personnel and vehicles in the picture, and the content of the personnel behavior recognition includes any one or both of climbing equipment and disassembly actions.

Images