Multimodal data association analysis method for industrial control systems based on knowledge graph

Abstract

The present invention belongs to the technical field of data association analysis and processing, and discloses a multimodal data association analysis method for an industrial control system based on a knowledge graph; the method comprises: reading multimodal data of equipment in an industrial control network; obtaining an initial industrial multimodal sub-knowledge graph of each equipment at different time points through the multimodal data and the mutual influence relationship between the multimodal data, thereby obtaining an initial industrial multimodal knowledge graph; connecting corresponding nodes in the initial industrial multimodal knowledge graph according to the physical connection relationship between the equipment to obtain a final industrial multimodal knowledge graph; obtaining an abnormality processing priority evaluation coefficient of the equipment according to the connection relationship between the nodes in the final industrial multimodal knowledge graph, obtaining the abnormality processing priority of the equipment through the abnormality processing priority evaluation coefficient of the equipment, and performing abnormality processing on the equipment according to the abnormality processing priority of the equipment; further improving the real-time performance of the data association analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of data association analysis and processing, and more specifically, to a multimodal data association analysis method for industrial control systems based on a knowledge graph. Background Art

[0002] Patent application publication number CN109885700A discloses a method for analyzing unstructured data based on an industrial knowledge graph. This method obtains an industrial knowledge graph, which includes nodes and edges. Each node represents an industrial entity in the real world, and each edge represents a relationship between industrial entities. The method determines the degree and associated nodes of each node in the industrial knowledge graph. The method determines the association probability of the associated nodes based on the degree of each node. The method determines the fusion information of each node based on the association probability. The method first determines the degree and associated nodes of each node in the industrial knowledge graph, then determines the association probability of the associated nodes based on the degree of each node, and then determines the fusion information of each node based on the association probability, thereby realizing unstructured data analysis of the industrial knowledge graph.

[0003] However, in the process of constructing knowledge graphs, traditional methods construct knowledge graphs based on domain ontology or logical reasoning or statistical learning and knowledge extraction or specific rules (such as graph structure rules, semantic similarity rules, constraint rules, time evolution rules, etc.). Once the construction is completed, the connection relationship between each node will not change. However, in some cases, the connection relationship between each node will change with time. Therefore, the knowledge graph constructed by the traditional method is not real-time, and the connection relationship between each node is inaccurate, which leads to incorrect exception handling priority of the equipment obtained by analyzing the knowledge graph constructed by the traditional method, which in turn causes the order of exception handling to be disordered, causing huge losses to the factory.

[0004] In view of this, the present invention proposes a multimodal data association analysis method for industrial control systems based on knowledge graph to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned objectives, the present invention provides the following technical solution: a method for multimodal data association analysis of industrial control systems based on knowledge graphs, comprising:

[0006] Step S1: reading multimodal data of devices in the industrial control network;

[0007] Step S2: Determine whether there is a connection between the multimodal data based on the purpose of the read multimodal data, obtain the mutual influence relationship between the multimodal data based on the judgment result, and obtain the initial industrial multimodal sub-knowledge graph of each device at different time points through the mutual influence relationship between the multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph;

[0008] Step S3: Improve the initial industrial multimodal knowledge graph by connecting the corresponding nodes in the initial industrial multimodal knowledge graph based on the physical connection relationship between the devices to obtain the final industrial multimodal knowledge graph;

[0009] Step S4: Obtain the equipment's exception handling priority evaluation coefficient based on the connection relationship between the nodes in the final industrial multimodal knowledge graph, obtain the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and perform exception handling on the equipment based on the equipment's exception handling priority.

[0010] Furthermore, the multimodal data includes structured data, unstructured data, image data, video data and text information data;

[0011] The structured data includes sensor data, control parameter data, equipment operation data and alarm record data, and the unstructured data includes operation and maintenance log data, alarm description data and event record data;

[0012] The image data includes instrument reading image data, the video data includes equipment working screen video data, and the text information data includes operation and maintenance manual data, fault case library data, and training document data.

[0013] Furthermore, the method of determining whether there is a connection between multimodal data based on the usage of the read multimodal data includes:

[0014] The sensor data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, instrument reading image data, and equipment working screen video data are used for equipment anomaly analysis; the control parameter data are used for control strategy evaluation; the event record data are used for evolution trajectory analysis; the operation and maintenance manual data, fault case library data, and training document data are used for safe operation and maintenance guidance;

[0015] If the multimodal data have the same purpose, there is a connection between the multimodal data. If the multimodal data have different purposes, there is no connection between the multimodal data.

[0016] Furthermore, the method for obtaining the mutual influence relationship between multimodal data based on the judgment result includes:

[0017] Get the start and end time of reading the multimodal data of the equipment in the industrial control network, and divide the time period from the start time to the end time into time points, and obtain the multimodal data of the device corresponding to each time point;

[0018] Obtaining multimodal data that changes from a first time point to a second time point based on the multimodal data of the device corresponding to each time point. If there is a relationship between the multimodal data that changes from the first time point to the second time point, the relationship between the multimodal data that changes from the first time point to the second time point is recorded as a second mutual influence relationship.

[0019] The multimodal data that changes from the second time point to the third time point is obtained based on the multimodal data of the device corresponding to each time point. If there is a connection between the multimodal data that changes from the second time point to the third time point, the relationship between the multimodal data that changes from the second time point to the third time point is recorded as the third mutual influence relationship. This is deduced by analogy until the multimodal data that changes from the second-to-last time point to the first-to-last time point is obtained. If there is a connection between the multimodal data that changes from the second-to-last time point to the first-to-last time point, the relationship between the multimodal data that changes from the second-to-last time point to the first-to-last time point is recorded as the last mutual influence relationship.

[0020] Furthermore, the method of obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points through multimodal data and the mutual influence relationship between multimodal data includes:

[0021] The initial industrial multimodal knowledge graph is composed of time nodes and multimodal data nodes, and the multimodal data nodes include sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes, and training document data nodes;

[0022] Label the equipment as , , is a positive integer, store the second time point in the time node, and store the first The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the corresponding multimodal data node respectively;

[0023] The time nodes are connected to each multimodal data node respectively, the nodes corresponding to the multimodal data that have changed from the first time point to the second time point and are recorded as the second mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point are highlighted to obtain the " The initial industrial multimodal sub-knowledge graph of the device at the second time point is obtained by the same method, and the initial industrial multimodal sub-knowledge graphs of the remaining devices at the second time point are obtained;

[0024] The same method is used to obtain the initial industrial multimodal sub-knowledge graphs of all equipment at the third time point, the fourth time point, ..., and the last time point;

[0025] The initial industrial multimodal knowledge graphs of all devices at the second time point, the third time point, ..., the last time point constitute the initial industrial multimodal knowledge graph.

[0026] Furthermore, the method of storing in the corresponding multimodal data node includes:

[0027] The second time point corresponds to the The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the sensor data node, control parameter data node, equipment operation data node, alarm record data node, operation and maintenance log data node, alarm description data node, event record data node, instrument reading image data node, equipment working screen video data node, operation and maintenance manual data node, fault case library data node and training document data node respectively.

[0028] Furthermore, the method for obtaining the final industrial multimodal knowledge graph includes:

[0029] According to the initial industrial multimodal knowledge graph of all devices at the second time point, the alarm record data node highlighted in the initial industrial multimodal knowledge graph at the second time point is obtained and recorded as the second highlighted alarm record data node, and the physical connection relationship between the devices is obtained. If a physical connection relationship exists between the devices corresponding to the second highlighted alarm record data node, the second highlighted alarm record data nodes having a physical connection relationship between the corresponding devices are interconnected;

[0030] Obtain, based on the initial industrial multimodal sub-knowledge graph of all devices at the third time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the third time point, and record it as the third highlighted alarm record data node; if there is a physical connection relationship between the devices corresponding to the third highlighted alarm record data node, then interconnect the third highlighted alarm record data nodes that have a physical connection relationship between the corresponding devices;

[0031] And so on, until the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the last time point is obtained based on the initial industrial multimodal sub-knowledge graph of all devices at the last time point, and recorded as the last highlighted alarm record data node. If there is a physical connection relationship between the devices corresponding to the last highlighted alarm record data node, the last highlighted alarm record data nodes with a physical connection relationship between the corresponding devices are interconnected to obtain the final industrial multimodal knowledge graph.

[0032] Furthermore, the method for obtaining the equipment exception handling priority evaluation coefficient based on the connection relationship between nodes in the final industrial multimodal knowledge graph includes:

[0033] The number of remaining nodes connected to the highlighted alarm record data node at each time point in the final industrial multimodal knowledge graph is obtained, and the number of remaining nodes connected to the highlighted alarm record data node is used as the abnormality handling priority evaluation coefficient of the corresponding equipment at the corresponding time point.

[0034] Furthermore, the method for obtaining the exception handling priority of a device by using the exception handling priority evaluation coefficient of the device includes:

[0035] The exception handling priority includes a first-level exception handling priority, a second-level exception handling priority, and a third-level exception handling priority;

[0036] Set a threshold range for the exception handling priority evaluation coefficient. When the exception handling priority evaluation coefficient is less than the minimum value in the threshold range, the exception handling priority of the device at that time point is recorded as the first-level exception handling priority.

[0037] When the exception handling priority evaluation coefficient is greater than or equal to the minimum value in the exception handling priority evaluation coefficient threshold range and less than or equal to the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the second-level exception handling priority;

[0038] When the exception handling priority evaluation coefficient is greater than the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the third-level exception handling priority.

[0039] Furthermore, the method for performing exception handling on a device according to the exception handling priority of the device includes:

[0040] Obtain the location of the equipment with the third-level exception handling priority, dispatch relevant personnel to the location of the equipment with the third-level exception handling priority, and the relevant personnel repair the equipment with the third-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the third-level exception handling priority;

[0041] Obtain the location of the device with the second-level exception handling priority, dispatch relevant personnel to the location of the device with the second-level exception handling priority, and the relevant personnel repair the device with the second-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the device with the second-level exception handling priority;

[0042] Obtain the location of the equipment with the first-level exception handling priority, dispatch relevant personnel to the location of the equipment with the first-level exception handling priority, and the relevant personnel repair the equipment with the first-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the first-level exception handling priority.

[0043] The technical effects and advantages of the knowledge graph-based industrial control system multimodal data association analysis method of the present invention are as follows:

[0044] 1. The initial industrial multimodal sub-knowledge graph of each device at different time points is obtained through the mutual influence relationship between multimodal data and multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph. The initial industrial multimodal knowledge graph is improved to obtain the final industrial multimodal knowledge graph. The connection relationship of nodes at different time points changes. Industrial multimodal sub-knowledge graphs are constructed for different time points respectively, and are improved according to the physical connection relationship between devices to obtain the final industrial multimodal knowledge graph. This ensures that there are corresponding knowledge graphs at different time points, and thus makes the final industrial multimodal knowledge graph real-time, ensuring the accuracy of the connection relationship of nodes in the final industrial multimodal knowledge graph. As a result, the abnormal handling priority of the equipment obtained by analyzing the final industrial multimodal knowledge graph is correct, so that the abnormal handling of the equipment is carried out in an orderly manner, further reducing the losses caused by equipment maintenance in the factory;

[0045] 2. The equipment's exception handling priority evaluation coefficient is obtained based on the connection relationship between the nodes in the final industrial multimodal knowledge graph. The equipment's exception handling priority is obtained through the equipment's exception handling priority evaluation coefficient. The equipment's exception handling priority is handled based on the equipment's exception handling priority. The equipment's exception handling priority is obtained quickly and accurately, thereby speeding up the equipment's maintenance time and further reducing the factory's losses caused by equipment maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the multimodal data association analysis method for industrial control systems based on knowledge graph of the present invention;

[0047] Figure 2 Schematic diagram of the multimodal data association analysis system for industrial control systems based on knowledge graph of the present invention;

[0048] Figure 3 This is a flowchart of the exception handling priority of the acquisition device of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] Example 1

[0051] See also Figure 1 and Figure 3As shown, the method for multimodal data association analysis of industrial control systems based on knowledge graphs described in this embodiment includes:

[0052] Step S1: reading multimodal data of devices in the industrial control network;

[0053] Step S2: Determine whether there is a connection between the multimodal data based on the purpose of the read multimodal data, obtain the mutual influence relationship between the multimodal data based on the judgment result, and obtain the initial industrial multimodal sub-knowledge graph of each device at different time points through the mutual influence relationship between the multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph;

[0054] Step S3: Improve the initial industrial multimodal knowledge graph by connecting the corresponding nodes in the initial industrial multimodal knowledge graph based on the physical connection relationship between the devices to obtain the final industrial multimodal knowledge graph;

[0055] Step S4: Obtain the equipment's exception handling priority evaluation coefficient based on the connection relationship between the nodes in the final industrial multimodal knowledge graph, obtain the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and perform exception handling on the equipment based on the equipment's exception handling priority.

[0056] The process of reading multimodal data from devices in an industrial control network includes:

[0057] The multimodal data includes structured data, unstructured data, image data, video data and text information data;

[0058] The structured data includes sensor data, control parameter data, equipment operation data, and alarm record data. The sensor data includes temperature, pressure, current, voltage, etc. The control parameter data includes control thresholds, etc. The equipment operation data includes start and stop time, load information, etc. The alarm record data includes alarm ID, alarm time, alarm location, etc.

[0059] The unstructured data includes operation and maintenance log data, alarm description data, and event record data. The operation and maintenance log data includes information such as abnormal motor vibration at 2:13 PM on October 15, 2024. The alarm description data includes alarm number A017, which indicates excessive pressure. The event record data includes information such as sensor X being replaced and configuration Y being updated.

[0060] The image data includes instrument reading image data, etc.

[0061] The video data includes device working screen video data, etc.;

[0062] The text information data includes operation and maintenance manual data, fault case database data, and training document data. The contents of the operation and maintenance manual data include operating procedures, maintenance cycles, equipment structure diagrams, etc. The contents of the fault case database data include common equipment fault analysis and treatment methods, etc. The contents of the training document data include operating instructions, safety specifications, etc.

[0063] Setting a data reading terminal, wherein the data reading terminal is used to obtain structured data, unstructured data, image data, video data and text information data;

[0064] The server of the industrial control network is linked through the data reading terminal, and the structured data, unstructured data, image data, video data and text information data of the equipment in the industrial control network are read through the data reading terminal.

[0065] The process of obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points includes:

[0066] Get the start and end time of reading the multimodal data of the equipment in the industrial control network, and divide the time period from the start time to the end time into time points, and obtain the multimodal data of the device corresponding to each time point;

[0067] Obtain the multimodal data that changes from the first time point to the second time point based on the multimodal data of the device corresponding to each time point, and determine whether there is a connection between the multimodal data that changes from the first time point to the second time point; if there is a connection between the multimodal data that changes from the first time point to the second time point, the relationship between the multimodal data that changes from the first time point to the second time point is recorded as a second mutual influence relationship; obtain the multimodal data that changes from the second time point to the third time point based on the multimodal data of the device corresponding to each time point, and determine whether there is a connection between the multimodal data that changes from the second time point to the third time point; if there is a connection between the multimodal data that changes from the second time point to the third time point, the relationship between the multimodal data that changes from the second time point to the third time point is recorded as a third mutual influence relationship; and so on, until the determination of whether there is a connection between the multimodal data that changes from the second-to-last time point to the first-to-last time point is completed, and the relationship between the multimodal data that changes from the second-to-last time point to the first-to-last time point is recorded as the final mutual influence relationship;

[0068] It should be explained that the basis for determining whether the instrument reading image data has changed is to perform image recognition on the instrument reading image data to determine whether the instrument reading has changed. If the instrument reading has changed, the instrument reading image data has changed. If the instrument reading has not changed, the instrument reading image data has not changed. The basis for determining whether the equipment working screen video data has changed is to perform visual inspection on the equipment working screen video data to determine whether the equipment is working abnormally. If the equipment changes from a normal working state to an abnormal working state, the equipment working screen video data changes. If the equipment does not change from a normal working state to an abnormal working state, the equipment working screen video data does not change.

[0069] The process of determining whether there is a connection between multimodal data that changes from a first time point to a second time point includes:

[0070] The sensor data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, instrument reading image data, and equipment working screen video data are used for equipment anomaly analysis; the control parameter data are used for control strategy evaluation; the event record data are used for evolution trajectory analysis; the operation and maintenance manual data, fault case library data, and training document data are used for safe operation and maintenance guidance;

[0071] If the purposes of the changed multimodal data are the same, then there is a connection between the changed multimodal data; if the purposes of the changed multimodal data are different, then there is no connection between the changed multimodal data;

[0072] The initial industrial multimodal knowledge graph is composed of time nodes and multimodal data nodes, and the multimodal data nodes include sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes and training document data nodes. The time nodes are used to store time points, and the sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes and training document data nodes are used to store sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data respectively;

[0073] Label the equipment as , , is a positive integer, store the second time point in the time node, and store the first The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the sensor data node, control parameter data node, equipment operation data node, alarm record data node, operation and maintenance log data node, alarm description data node, event record data node, instrument reading image data node, equipment working screen video data node, operation and maintenance manual data node, fault case library data node and training document data node respectively, and The time node is connected with the sensor data node, the control parameter data node, the equipment operation data node, the alarm record data node, the operation and maintenance log data node, the alarm description data node, the event record data node, the instrument reading image data node, the equipment working screen video data node, the operation and maintenance manual data node, the fault case library data node and the training document data node respectively, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point and are recorded as the second mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point are highlighted to obtain the " The initial industrial multimodal sub-knowledge graph of the device at the second time point is obtained by the same method, and the initial industrial multimodal sub-knowledge graphs of the remaining devices at the second time point are obtained;

[0074] Store the third time point in the new time node, and store the third time point in the new time node. "The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in a new sensor data node, a new control parameter data node, a new equipment operation data node, a new alarm record data node, a new operation and maintenance log data node, a new alarm description data node, a new event record data node, a new instrument reading image data node, a new equipment working screen video data node, a new operation and maintenance manual data node, a new fault case library data node and a new training document data node, and the The new time node is respectively connected with the new sensor data node, the new control parameter data node, the new equipment operation data node, the new alarm record data node, the new operation and maintenance log data node, the new alarm description data node, the new event record data node, the new instrument reading image data node, the new equipment working screen video data node, the new operation and maintenance manual data node, the new fault case library data node and the new training document data node, and the nodes corresponding to the multimodal data that have changed from the second time point to the third time point and are recorded as the third mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the second time point to the third time point are highlighted to obtain the " "The initial industrial multimodal sub-knowledge graph of the device at the third time point is obtained by the same method. The initial industrial multimodal sub-knowledge graphs of the remaining devices at the third time point are obtained by the same method. The initial industrial multimodal sub-knowledge graphs of all devices at the fourth time point, the fifth time point, ..., and the last time point are obtained by the same method.

[0075] The initial industrial multimodal knowledge graphs of all devices at the second time point, the third time point, ..., the last time point constitute the initial industrial multimodal knowledge graph.

[0076] The initial industrial multimodal knowledge graph is improved. The corresponding nodes in the initial industrial multimodal knowledge graph are connected according to the physical connection relationship between devices. The process of obtaining the final industrial multimodal knowledge graph includes:

[0077] According to the initial industrial multimodal sub-knowledge graph of all devices at the second time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the second time point is obtained, and recorded as the second highlighted alarm record data node, and the physical connection relationship between the devices is obtained. If there is a physical connection relationship between the devices corresponding to the second highlighted alarm record data node, the second highlighted alarm record data nodes with a physical connection relationship between the corresponding devices are interconnected. According to the initial industrial multimodal sub-knowledge graph of all devices at the third time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the third time point is obtained, and recorded as the third highlighted alarm record data node. point, if there is a physical connection relationship between the devices corresponding to the third highlighted alarm record data node, then the third highlighted alarm record data nodes with physical connection relationships between the corresponding devices are interconnected, and so on, until the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the last time point is obtained according to the initial industrial multimodal sub-knowledge graph of all devices at the last time point, and recorded as the last highlighted alarm record data node, if there is a physical connection relationship between the devices corresponding to the last highlighted alarm record data node, then the last highlighted alarm record data nodes with physical connection relationships between the corresponding devices are interconnected to obtain the final industrial multimodal knowledge graph;

[0078] It should be explained that by connecting the highlighted alarm record data nodes corresponding to the devices with physical connection relationships, changes in the alarm record data of the devices indicate that the devices have abnormalities. The abnormal devices will affect the abnormalities of the devices with physical connection relationships. Therefore, connecting the highlighted alarm record data nodes corresponding to the devices with physical connection relationships indicates that the abnormalities and the physically connected devices affect each other.

[0079] It should be explained that traditional methods construct knowledge graphs based on logical reasoning or statistical learning and knowledge extraction or specific rules. Once the construction is completed, the connection relationship between each node will not change. However, in some cases, the connection relationship between each node will change over time. Therefore, the knowledge graph constructed by the traditional method is not real-time, and the connection relationship between each node is inaccurate, which leads to incorrect exception handling priorities of the equipment obtained by analyzing the knowledge graph constructed by the traditional method, and then causes the order of exception handling to be disordered, causing huge losses to the factory. Therefore, the present invention constructs an initial industrial multimodal knowledge graph by constructing initial industrial multimodal sub-knowledge graphs of all equipment at the second time point, the third time point, ..., and the last time point to form an initial industrial multimodal knowledge graph, and improves the initial industrial multimodal knowledge graph to obtain a final industrial multimodal knowledge graph, so that the construction of the final industrial multimodal knowledge graph is real-time, ensuring the accuracy of the connection relationship of the nodes in the final industrial multimodal knowledge graph, and thus making the exception handling priority of the equipment obtained by analyzing the final industrial multimodal knowledge graph correct, so that the exception handling of the equipment is carried out in an orderly manner, further reducing the losses caused by equipment maintenance in the factory.

[0080] The process of obtaining the equipment's exception handling priority evaluation coefficient based on the connection relationship between nodes in the final industrial multimodal knowledge graph and obtaining the equipment's exception handling priority evaluation coefficient includes:

[0081] The exception handling priority includes a first-level exception handling priority, a second-level exception handling priority, and a third-level exception handling priority;

[0082] It should be explained that the priority level of the third-level exception handling priority is higher than the priority level of the second-level exception handling priority, which is higher than the priority level of the first-level exception handling priority;

[0083] Obtain the number of remaining nodes connected to the highlighted alarm record data node at each time point in the final industrial multimodal knowledge graph, and use the number of remaining nodes connected to the highlighted alarm record data node as the abnormality processing priority evaluation coefficient of the corresponding device at the corresponding time point;

[0084] Set the threshold range of the exception handling priority evaluation coefficient. The threshold range of the exception handling priority evaluation coefficient can be set through experimental data analysis or experience. When the exception handling priority evaluation coefficient is less than the minimum value in the threshold range, the exception handling priority of the device at that time point is recorded as the first-level exception handling priority;

[0085] When the exception handling priority evaluation coefficient is greater than or equal to the minimum value in the exception handling priority evaluation coefficient threshold range and less than or equal to the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the second-level exception handling priority;

[0086] When the exception handling priority evaluation coefficient is greater than the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the third level exception handling priority;

[0087] It should be explained that the more nodes a highlighted alarm record data node is connected to, the greater the problem with the device, the higher the exception handling priority of the device, and the earlier the device is processed. For example, the highlighted alarm record data node of device one is connected to the sensor data node, the alarm description data node, and one other highlighted alarm record data node, and the highlighted alarm record data node of device two is connected to the sensor data node, the alarm description data node, the operation and maintenance log data node, and two other highlighted alarm record data nodes. Device two is connected to one more operation and maintenance log data node, so the problem with device two is more serious, resulting in changes in the operation and maintenance log data, that is, the problem with device two has caused the other problems of device two, that is, device two has more problems and a higher degree of abnormality. Device two is connected to one more highlighted alarm record data node, indicating that the problem with device two may have caused problems with the other two devices, while the problem with device one may have only caused problems with the other one device. Therefore, the exception handling priority of device two is higher than that of device one, and the highlighted alarm record data node of device two is connected to more nodes;

[0088] The process of handling device exceptions based on their priority level includes:

[0089] Obtain the location of the equipment with the third-level exception handling priority, dispatch relevant personnel to the location of the equipment with the third-level exception handling priority, and the relevant personnel repair the equipment with the third-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the third-level exception handling priority;

[0090] Obtain the location of the device with the second-level exception handling priority, dispatch relevant personnel to the location of the device with the second-level exception handling priority, and the relevant personnel repair the device with the second-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the device with the second-level exception handling priority;

[0091] Obtain the location of the equipment with the first-level exception handling priority, dispatch relevant personnel to the location of the equipment with the first-level exception handling priority, and the relevant personnel repair the equipment with the first-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the first-level exception handling priority.

[0092] In this embodiment, the initial industrial multimodal sub-knowledge graph of each device at different time points is obtained through the mutual influence relationship between multimodal data and multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph. The initial industrial multimodal knowledge graph is improved to obtain the final industrial multimodal knowledge graph. The connection relationship of nodes at different time points changes. Industrial multimodal sub-knowledge graphs are constructed for different time points respectively, and are improved according to the physical connection relationship between devices to obtain the final industrial multimodal knowledge graph, so that corresponding knowledge graphs are obtained at different time points, and the final industrial multimodal knowledge graph is real-time, ensuring the final industrial multimodal knowledge graph. The accuracy of the connection relationship between the nodes in the industrial multimodal knowledge graph makes the equipment exception handling priority obtained by analyzing the final industrial multimodal knowledge graph correct, so that the equipment exception handling is carried out in an orderly manner, further reducing the factory's losses caused by equipment maintenance; the equipment exception handling priority evaluation coefficient is obtained according to the connection relationship between the nodes in the final industrial multimodal knowledge graph, the equipment exception handling priority is obtained through the equipment exception handling priority evaluation coefficient, and the equipment exception is handled according to the equipment exception handling priority. The equipment exception handling priority is obtained quickly and accurately, thereby speeding up the equipment maintenance time and further reducing the factory's losses caused by equipment maintenance.

[0093] Example 2

[0094] See also Figure 2 As shown, for the parts not described in detail in this embodiment, please refer to the description of Example 1. A multimodal data association analysis system for an industrial control system based on a knowledge graph is provided, including:

[0095] Data reading component, responsible for reading multimodal data from devices in the industrial control network;

[0096] The graph construction component is responsible for obtaining the mutual influence relationship between multimodal data. The initial industrial multimodal sub-knowledge graph of each device at different time points is obtained through the mutual influence relationship between multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph;

[0097] The graph improvement component is responsible for improving the initial industrial multimodal knowledge graph to obtain the final industrial multimodal knowledge graph;

[0098] The exception handling component is responsible for obtaining the equipment's exception handling priority evaluation coefficient based on the connection relationship between nodes in the final industrial multimodal knowledge graph, obtaining the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and performing exception handling on the equipment based on the equipment's exception handling priority.

[0099] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0100] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only one type. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0101] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

[0102] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal data association analysis method for industrial control systems based on knowledge graph, characterized by: The multimodal data association analysis method for industrial control systems based on knowledge graph includes: Step S1: reading multimodal data of devices in the industrial control network; Step S2: Determine whether there is a connection between the multimodal data based on the purpose of the read multimodal data, obtain the mutual influence relationship between the multimodal data based on the judgment result, and obtain the initial industrial multimodal sub-knowledge graph of each device at different time points through the mutual influence relationship between the multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph; The process of obtaining the mutual influence relationship between multimodal data based on the judgment results includes: Get the start and end time of reading the multimodal data of the equipment in the industrial control network, and divide the time period from the start time to the end time into time points, and obtain the multimodal data of the device corresponding to each time point; Obtaining multimodal data that changes from a first time point to a second time point based on the multimodal data of the device corresponding to each time point. If there is a relationship between the multimodal data that changes from the first time point to the second time point, the relationship between the multimodal data that changes from the first time point to the second time point is recorded as a second mutual influence relationship. This process is repeated until the multimodal data that changes from the second-to-last time point to the first-to-last time point are obtained. If there is a relationship between the multimodal data that changes from the second-to-last time point to the first-to-last time point, the relationship between the multimodal data that changes from the second-to-last time point to the first-to-last time point is recorded as the final mutual influence relationship. The process of obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points includes: The initial industrial multimodal knowledge graph consists of time nodes and multimodal data nodes; Label the equipment as , , is a positive integer, the second time point is stored in the time node, and the first The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the corresponding multimodal data node respectively; The time nodes are connected to each multimodal data node respectively, the nodes corresponding to the multimodal data that have changed from the first time point to the second time point and are recorded as the second mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point are highlighted to obtain the " The initial industrial multimodal sub-knowledge graph of the device at the second time point is obtained by the same method, and the initial industrial multimodal sub-knowledge graphs of the remaining devices at the second time point are obtained; The same method is used to obtain the initial industrial multimodal sub-knowledge graphs of all equipment at the third time point, the fourth time point, ..., and the last time point; Step S3: Improve the initial industrial multimodal knowledge graph by connecting the corresponding nodes in the initial industrial multimodal knowledge graph based on the physical connection relationship between devices to obtain the final industrial multimodal knowledge graph, which specifically includes: According to the initial industrial multimodal knowledge graph of all devices at the second time point, the alarm record data node highlighted in the initial industrial multimodal knowledge graph at the second time point is obtained and recorded as the second highlighted alarm record data node, and the physical connection relationship between the devices is obtained. If a physical connection relationship exists between the devices corresponding to the second highlighted alarm record data node, the second highlighted alarm record data nodes having a physical connection relationship between the corresponding devices are interconnected; This process is repeated in this way until the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the last time point is obtained based on the initial industrial multimodal sub-knowledge graph of all devices at the last time point, and recorded as the last highlighted alarm record data node. If there is a physical connection relationship between the devices corresponding to the last highlighted alarm record data node, the last highlighted alarm record data nodes with physical connection relationships between the corresponding devices are interconnected to obtain the final industrial multimodal knowledge graph. Step S4: Obtain the equipment's exception handling priority evaluation coefficient based on the connection relationship between the nodes in the final industrial multimodal knowledge graph, obtain the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and perform exception handling on the equipment based on the equipment's exception handling priority; The process of obtaining the device's exception handling priority evaluation coefficient includes: The number of remaining nodes connected to the highlighted alarm record data node at each time point in the final industrial multimodal knowledge graph is obtained, and the number of remaining nodes connected to the highlighted alarm record data node is used as the abnormality handling priority evaluation coefficient of the corresponding equipment at the corresponding time point.

2. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 1 is characterized in that: The multimodal data includes structured data, unstructured data, image data, video data and text information data; The structured data includes sensor data, control parameter data, equipment operation data and alarm record data, and the unstructured data includes operation and maintenance log data, alarm description data and event record data; The image data includes instrument reading image data, the video data includes equipment working screen video data, and the text information data includes operation and maintenance manual data, fault case library data, and training document data.

3. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 2 is characterized in that: The method for determining whether there is a connection between multimodal data based on the usage of the read multimodal data includes: The sensor data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, instrument reading image data, and equipment working screen video data are used for equipment anomaly analysis; the control parameter data are used for control strategy evaluation; the event record data are used for evolution trajectory analysis; the operation and maintenance manual data, fault case library data, and training document data are used for safe operation and maintenance guidance; If the multimodal data have the same purpose, there is a connection between the multimodal data. If the multimodal data have different purposes, there is no connection between the multimodal data.

4. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 3 is characterized in that: The multimodal data nodes include sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes and training document data nodes.

5. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 4 is characterized in that: The method of storing in the corresponding multimodal data node includes: The second time point corresponds to the The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the sensor data node, control parameter data node, equipment operation data node, alarm record data node, operation and maintenance log data node, alarm description data node, event record data node, instrument reading image data node, equipment working screen video data node, operation and maintenance manual data node, fault case library data node and training document data node respectively.

6. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 5 is characterized in that: The method for obtaining the exception handling priority of a device by using the exception handling priority evaluation coefficient of the device includes: The exception handling priority includes a first-level exception handling priority, a second-level exception handling priority, and a third-level exception handling priority; Set a threshold range for the exception handling priority evaluation coefficient. When the exception handling priority evaluation coefficient is less than the minimum value in the threshold range, the exception handling priority of the device at that time point is recorded as the first-level exception handling priority. When the exception handling priority evaluation coefficient is greater than or equal to the minimum value in the exception handling priority evaluation coefficient threshold range and less than or equal to the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the second-level exception handling priority; When the exception handling priority evaluation coefficient is greater than the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the third-level exception handling priority.

7. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 6 is characterized in that: The method for performing exception handling on a device according to the exception handling priority of the device includes: Obtain the location of the equipment with the third-level exception handling priority, dispatch relevant personnel to the location of the equipment with the third-level exception handling priority, and the relevant personnel repair the equipment with the third-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the third-level exception handling priority; Obtain the location of the device with the second-level exception handling priority, dispatch relevant personnel to the location of the device with the second-level exception handling priority, and the relevant personnel repair the device with the second-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the device with the second-level exception handling priority; Obtain the location of the equipment with the first-level exception handling priority, dispatch relevant personnel to the location of the equipment with the first-level exception handling priority, and the relevant personnel repair the equipment with the first-level exception handling priority based on the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the first-level exception handling priority.

Images