Industrial control system multi-modal data association analysis method based on knowledge graph

Abstract

The invention belongs to the technical field of data association analysis processing, and discloses an industrial control system multi-modal data association analysis method based on a knowledge graph. Comprising the following steps: reading multi-modal data of equipment in an industrial control network; obtaining an initial industrial multi-modal sub-knowledge graph of each device at different time points through the multi-modal data and the mutual influence relationship between the multi-modal data, and further obtaining an initial industrial multi-modal knowledge graph; corresponding nodes in the initial industrial multi-mode knowledge graph are connected according to the physical connection relation between the devices, and a final industrial multi-mode knowledge graph is obtained; according to the connection relation between the nodes in the final industrial multi-mode knowledge graph, obtaining an exception handling priority judgment coefficient of the equipment, obtaining an exception handling priority of the equipment through the exception handling priority judgment coefficient of the equipment, and carrying out exception handling on the equipment according to the exception handling priority of the equipment; and the real-time performance of data association analysis is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data association analysis and processing, and more specifically, to a multimodal data association analysis method for an industrial control system based on a knowledge graph. Background Art

[0002] The patent application with the publication number CN109885700A discloses an unstructured data analysis method based on industrial knowledge graph. The method obtains the industrial knowledge graph, which includes nodes and edges. Each node represents an industrial entity existing in the real world, and each edge represents the relationship between industrial entities; determines the degree and associated nodes of each node in the industrial knowledge graph; determines the association probability of the associated nodes according to the degree of each node; and determines the fusion information of each node according to the association probability. The present invention first determines the degree and associated nodes of each node in the industrial knowledge graph, then determines the association probability of the associated nodes according to the degree of each node, and then determines the fusion information of each node according to the association probability, thereby realizing unstructured data analysis of the industrial knowledge graph.

[0003] However, in the process of constructing the knowledge graph, the traditional method constructs the knowledge graph according to the domain ontology or logical reasoning or statistical learning and knowledge extraction or specific rules (such as graph structure rules, semantic similarity rules, constraint rules, time evolution rules, etc.). Once the construction is completed, the connection relationship between the nodes will not change. However, in some cases, the connection relationship between the nodes will change with time. Therefore, the knowledge graph constructed by the traditional method is not real-time, and the connection relationship between the nodes is inaccurate, which leads to incorrect exception handling priority of the equipment obtained by analyzing the knowledge graph constructed by the traditional method, which in turn causes the order of exception handling to be disordered, causing huge losses to the factory.

[0004] In view of this, the present invention proposes a multimodal data association analysis method for industrial control systems based on knowledge graph to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a method for multimodal data association analysis of an industrial control system based on a knowledge graph, comprising: Step S1: reading multimodal data of devices in the industrial control network; Step S2: judging whether there is a connection between the multimodal data according to the purpose of the read multimodal data, obtaining the mutual influence relationship between the multimodal data according to the judgment result, and obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points through the mutual influence relationship between the multimodal data and the multimodal data, and the initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph; Step S3: improving the initial industrial multimodal knowledge graph, connecting the corresponding nodes in the initial industrial multimodal knowledge graph according to the physical connection relationship between the devices, and obtaining the final industrial multimodal knowledge graph; Step S4: Obtain the equipment's exception handling priority evaluation coefficient based on the connection relationship between nodes in the final industrial multimodal knowledge graph, obtain the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and perform exception handling on the equipment based on the equipment's exception handling priority.

[0006] Furthermore, the multimodal data includes structured data, unstructured data, image data, video data and text information data; The structured data includes sensor data, control parameter data, equipment operation data and alarm record data, and the unstructured data includes operation and maintenance log data, alarm description data and event record data; The image data includes instrument reading image data, the video data includes equipment working screen video data, and the text information data includes operation and maintenance manual data, fault case library data, and training document data.

[0007] Furthermore, the method of determining whether there is a connection between multimodal data according to the purpose of the read multimodal data includes: The sensor data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, instrument reading image data and equipment working screen video data are used for equipment abnormality analysis, the control parameter data are used for control strategy evaluation, the event record data are used for evolution trajectory analysis, and the operation and maintenance manual data, fault case library data and training document data are used for safe operation and maintenance guidance; If the purposes of multimodal data are the same, there is a connection between the multimodal data. If the purposes of multimodal data are different, there is no connection between the multimodal data.

[0008] Furthermore, the method for obtaining the mutual influence relationship between the multimodal data according to the judgment result includes: Get the start time and end time of reading the multimodal data of the device in the industrial control network, and divide the time period from the start time to the end time into time points, and obtain the multimodal data of the device corresponding to each time point; Obtaining multimodal data that changes from a first time point to a second time point based on the multimodal data of the device corresponding to each time point, and if there is a connection between the multimodal data that changes from the first time point to the second time point, recording the relationship between the multimodal data that changes from the first time point to the second time point as a second mutual influence relationship; The multimodal data that changes from the second time point to the third time point is obtained based on the multimodal data of the device corresponding to each time point. If there is a connection between the multimodal data that changes from the second time point to the third time point, the relationship between the multimodal data that changes from the second time point to the third time point is recorded as the third mutual influence relationship, and so on, until the multimodal data that changes from the second-to-last time point to the penultimate time point is obtained. If there is a connection between the multimodal data that changes from the second-to-last time point to the penultimate time point, the relationship between the multimodal data that changes from the second-to-last time point to the penultimate time point is recorded as the last mutual influence relationship.

[0009] Furthermore, the method of obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points through multimodal data and the mutual influence relationship between multimodal data includes: The initial industrial multimodal knowledge graph is composed of time nodes and multimodal data nodes, and the multimodal data nodes include sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes and training document data nodes; Label the equipment as , , is a positive integer, the second time point is stored in the time node, and the first " "The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the corresponding multimodal data node respectively; The time nodes are connected to each multimodal data node respectively, the nodes corresponding to the multimodal data that have changed from the first time point to the second time point and are recorded as the second mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point are highlighted to obtain the " "The initial industrial multimodal sub-knowledge graph of the device at the second time point is obtained, and the initial industrial multimodal sub-knowledge graph of the remaining devices at the second time point is obtained by the same method; The same method is used to obtain the initial industrial multimodal sub-knowledge graphs of all equipment at the third time point, the fourth time point, ..., the last time point; The initial industrial multimodal knowledge graphs of all devices at the second time point, the third time point, ..., and the last time point constitute the initial industrial multimodal knowledge graph.

[0010] Furthermore, the method of storing in the corresponding multimodal data node includes: The second time point corresponding to the The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the sensor data node, control parameter data node, equipment operation data node, alarm record data node, operation and maintenance log data node, alarm description data node, event record data node, instrument reading image data node, equipment working screen video data node, operation and maintenance manual data node, fault case library data node and training document data node respectively.

[0011] Furthermore, the method for obtaining the final industrial multimodal knowledge graph includes: According to the initial industrial multimodal sub-knowledge graph of all devices at the second time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the second time point is obtained and recorded as the second highlighted alarm record data node, and the physical connection relationship between the devices is obtained. If there is a physical connection relationship between the devices corresponding to the second highlighted alarm record data node, the second highlighted alarm record data nodes having a physical connection relationship between the corresponding devices are connected to each other; According to the initial industrial multimodal sub-knowledge graph of all devices at the third time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the third time point is obtained and recorded as the third highlighted alarm record data node; if there is a physical connection relationship between the devices corresponding to the third highlighted alarm record data node, the third highlighted alarm record data nodes having a physical connection relationship between the corresponding devices are connected to each other; And so on, until the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the last time point is obtained based on the initial industrial multimodal sub-knowledge graph of all devices at the last time point, and recorded as the last highlighted alarm record data node. If there is a physical connection relationship between the devices corresponding to the last highlighted alarm record data node, the last highlighted alarm record data nodes with a physical connection relationship between the corresponding devices are interconnected to obtain the final industrial multimodal knowledge graph.

[0012] Furthermore, the method for obtaining the abnormality handling priority evaluation coefficient of the equipment according to the connection relationship between the nodes in the final industrial multimodal knowledge graph includes: The number of remaining nodes connected to the highlighted alarm record data node at each time point in the final industrial multimodal knowledge graph is obtained, and the number of remaining nodes connected to the highlighted alarm record data node is used as the abnormality handling priority evaluation coefficient of the corresponding device at the corresponding time point.

[0013] Furthermore, the method for obtaining the exception handling priority of a device through the exception handling priority evaluation coefficient of the device includes: The exception handling priority includes a first-level exception handling priority, a second-level exception handling priority, and a third-level exception handling priority; Set an exception handling priority evaluation coefficient threshold range. When the exception handling priority evaluation coefficient is less than the minimum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the first-level exception handling priority. When the exception handling priority evaluation coefficient is greater than or equal to the minimum value in the exception handling priority evaluation coefficient threshold range and less than or equal to the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the secondary exception handling priority; When the exception handling priority evaluation coefficient is greater than the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the third-level exception handling priority.

[0014] Furthermore, the method for performing exception handling on a device according to the exception handling priority of the device includes: Obtain the location of the equipment with the third level of exception handling priority, dispatch relevant personnel to the location of the equipment with the third level of exception handling priority, and the relevant personnel repair the equipment with the third level of exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the third level of exception handling priority; Obtain the location of the equipment with the second-level exception handling priority, dispatch relevant personnel to the location of the equipment with the second-level exception handling priority, and the relevant personnel repair the equipment with the second-level exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the second-level exception handling priority; Obtain the location of the equipment with the first-level exception handling priority, dispatch relevant personnel to the location of the equipment with the first-level exception handling priority, and the relevant personnel repair the equipment with the first-level exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the first-level exception handling priority.

[0015] The technical effects and advantages of the multimodal data association analysis method for industrial control systems based on knowledge graphs of the present invention are as follows: 1. The initial industrial multimodal sub-knowledge graph of each device at different time points is obtained through the mutual influence relationship between multimodal data and multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph. The initial industrial multimodal knowledge graph is improved to obtain the final industrial multimodal knowledge graph. The connection relationship of nodes at different time points changes. Industrial multimodal sub-knowledge graphs are constructed for different time points respectively, and are improved according to the physical connection relationship between devices to obtain the final industrial multimodal knowledge graph, so that there are corresponding knowledge graphs at different time points, and the final industrial multimodal knowledge graph is real-time, ensuring the accuracy of the connection relationship of nodes in the final industrial multimodal knowledge graph, and then the abnormal handling priority of the equipment obtained by analyzing the final industrial multimodal knowledge graph is correct, so that the abnormal handling of the equipment is carried out in an orderly manner, further reducing the loss caused by equipment maintenance in the factory; 2. The equipment exception handling priority evaluation coefficient is obtained based on the connection relationship between the nodes in the final industrial multimodal knowledge graph. The equipment exception handling priority is obtained through the equipment exception handling priority evaluation coefficient. The equipment exception is handled according to the equipment exception handling priority. The equipment exception handling priority is obtained quickly and accurately, thereby speeding up the equipment maintenance time and further reducing the factory's losses caused by equipment maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the multimodal data association analysis method of the industrial control system based on the knowledge graph of the present invention; Figure 2 It is a schematic diagram of the multimodal data association analysis system of the industrial control system based on the knowledge graph of the present invention; Figure 3 This is a flowchart of the exception handling priority of the acquisition device of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1 See also Figure 1 and Figure 3 As shown, the method for multimodal data association analysis of industrial control systems based on knowledge graphs described in this embodiment includes: Step S1: reading multimodal data of devices in the industrial control network; Step S2: judging whether there is a connection between the multimodal data according to the purpose of the read multimodal data, obtaining the mutual influence relationship between the multimodal data according to the judgment result, and obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points through the mutual influence relationship between the multimodal data and the multimodal data, and the initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph; Step S3: improving the initial industrial multimodal knowledge graph, connecting the corresponding nodes in the initial industrial multimodal knowledge graph according to the physical connection relationship between the devices, and obtaining the final industrial multimodal knowledge graph; Step S4: Obtain the equipment's exception handling priority evaluation coefficient based on the connection relationship between nodes in the final industrial multimodal knowledge graph, obtain the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and perform exception handling on the equipment based on the equipment's exception handling priority.

[0019] The process of reading multimodal data from devices in an industrial control network includes: The multimodal data includes structured data, unstructured data, image data, video data and text information data; The structured data includes sensor data, control parameter data, equipment operation data and alarm record data. The sensor data includes temperature, pressure, current, voltage, etc. The control parameter data includes control threshold, etc. The equipment operation data includes start and stop time, load information, etc. The alarm record data includes alarm ID, alarm time, alarm location, etc.; The unstructured data includes operation and maintenance log data, alarm description data, and event record data. The content of the operation and maintenance log data includes abnormal motor vibration at 14:13 on October 15, 2024, etc. The content of the alarm description data includes alarm number A017, description: pressure is too high, etc. The content of the event record data includes replacement of sensor X, update of configuration Y, etc.; The image data includes instrument reading image data, etc.; The video data includes device working screen video data, etc.; The text information data includes operation and maintenance manual data, fault case library data and training document data. The contents of the operation and maintenance manual data include operating procedures, maintenance cycles, equipment structure diagrams, etc. The contents of the fault case library data include common equipment fault analysis and treatment methods, etc. The contents of the training document data include operating instructions, safety specifications, etc.; Setting a data reading terminal, wherein the data reading terminal is used to obtain structured data, unstructured data, image data, video data and text information data; The server of the industrial control network is linked via a data reading terminal, and the structured data, unstructured data, image data, video data and text information data of the equipment in the industrial control network are read via the data reading terminal.

[0020] The process of obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points includes: Get the start time and end time of reading the multimodal data of the device in the industrial control network, and divide the time period from the start time to the end time into time points, and obtain the multimodal data of the device corresponding to each time point; According to the multimodal data of the device corresponding to each time point, the multimodal data that changes from the first time point to the second time point is obtained, and it is determined whether there is a connection between the multimodal data that changes from the first time point to the second time point. If there is a connection between the multimodal data that changes from the first time point to the second time point, the relationship between the multimodal data that changes from the first time point to the second time point is recorded as the second mutual influence relationship. According to the multimodal data of the device corresponding to each time point, the multimodal data that changes from the second time point to the third time point is obtained, and it is determined whether there is a connection between the multimodal data that changes from the second time point to the third time point. If there is a connection between the multimodal data that changes from the second time point to the third time point, the relationship between the multimodal data that changes from the second time point to the third time point is recorded as the third mutual influence relationship, and so on, until the judgment of whether there is a connection between the multimodal data that changes from the second to last time point to the first to last time point is completed, and the relationship between the multimodal data that changes from the second to last time point to the first to last time point is recorded as the last mutual influence relationship; It should be explained that the basis for determining whether the meter reading image data has changed is to perform image recognition on the meter reading image data to determine whether the meter reading has changed. If the meter reading has changed, the meter reading image data has changed. If the meter reading has not changed, the meter reading image data has not changed. The basis for determining whether the equipment working screen video data has changed is to perform visual inspection on the equipment working screen video data to determine whether the equipment is working abnormally. If the equipment changes from a normal working state to an abnormal working state, the equipment working screen video data has changed. If the equipment has not changed from a normal working state to an abnormal working state, the equipment working screen video data has not changed. The process of determining whether there is a connection between multimodal data that changes from a first time point to a second time point includes: The sensor data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, instrument reading image data and equipment working screen video data are used for equipment abnormality analysis, the control parameter data are used for control strategy evaluation, the event record data are used for evolution trajectory analysis, and the operation and maintenance manual data, fault case library data and training document data are used for safe operation and maintenance guidance; If the purposes of the changed multimodal data are the same, then there is a connection between the changed multimodal data; if the purposes of the changed multimodal data are different, then there is no connection between the changed multimodal data; The initial industrial multimodal knowledge graph is composed of time nodes and multimodal data nodes, and the multimodal data nodes include sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes and training document data nodes. The time nodes are used to store time points, and the sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes and training document data nodes are used to store sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data respectively; Label the equipment as , , is a positive integer, the second time point is stored in the time node, and the first " The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the sensor data node, control parameter data node, equipment operation data node, alarm record data node, operation and maintenance log data node, alarm description data node, event record data node, instrument reading image data node, equipment working screen video data node, operation and maintenance manual data node, fault case library data node and training document data node respectively, and The time node is connected with the sensor data node, the control parameter data node, the equipment operation data node, the alarm record data node, the operation and maintenance log data node, the alarm description data node, the event record data node, the instrument reading image data node, the equipment working screen video data node, the operation and maintenance manual data node, the fault case library data node and the training document data node respectively, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point and are recorded as the second mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point are highlighted to obtain the " "The initial industrial multimodal sub-knowledge graph of the device at the second time point is obtained, and the initial industrial multimodal sub-knowledge graph of the remaining devices at the second time point is obtained by the same method; Store the third time point in the new time node, and store the third time point in the new time node. The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in a new sensor data node, a new control parameter data node, a new equipment operation data node, a new alarm record data node, a new operation and maintenance log data node, a new alarm description data node, a new event record data node, a new instrument reading image data node, a new equipment working screen video data node, a new operation and maintenance manual data node, a new fault case library data node and a new training document data node, and the The new time node is respectively connected with the new sensor data node, the new control parameter data node, the new equipment operation data node, the new alarm record data node, the new operation and maintenance log data node, the new alarm description data node, the new event record data node, the new instrument reading image data node, the new equipment working screen video data node, the new operation and maintenance manual data node, the new fault case library data node and the new training document data node, and the nodes corresponding to the multimodal data that have changed from the second time point to the third time point and are recorded as the third mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the second time point to the third time point are highlighted to obtain the first " "The initial industrial multimodal sub-knowledge graph of the device at the third time point is obtained by the same method, and the initial industrial multimodal sub-knowledge graphs of the remaining devices at the third time point are obtained by the same method, and the initial industrial multimodal sub-knowledge graphs of all devices at the fourth time point, the fifth time point, ..., and the last time point are obtained by the same method; The initial industrial multimodal knowledge graphs of all devices at the second time point, the third time point, ..., and the last time point constitute the initial industrial multimodal knowledge graph.

[0021] The initial industrial multimodal knowledge graph is improved, and the corresponding nodes in the initial industrial multimodal knowledge graph are connected according to the physical connection relationship between the devices. The process of obtaining the final industrial multimodal knowledge graph includes: According to the initial industrial multimodal sub-knowledge graph of all devices at the second time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the second time point is obtained, and recorded as the second highlighted alarm record data node, and the physical connection relationship between the devices is obtained. If there is a physical connection relationship between the devices corresponding to the second highlighted alarm record data node, the second highlighted alarm record data nodes having a physical connection relationship between the corresponding devices are interconnected, and according to the initial industrial multimodal sub-knowledge graph of all devices at the third time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the third time point is obtained, and recorded as the third highlighted alarm record data node. point, if there is a physical connection relationship between the devices corresponding to the third highlighted alarm record data node, then the third highlighted alarm record data nodes with physical connection relationship between the corresponding devices are interconnected, and so on, until the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the last time point is obtained according to the initial industrial multimodal sub-knowledge graph of all devices at the last time point, and recorded as the last highlighted alarm record data node, if there is a physical connection relationship between the devices corresponding to the last highlighted alarm record data node, then the last highlighted alarm record data nodes with physical connection relationship between the corresponding devices are interconnected to obtain the final industrial multimodal knowledge graph; It needs to be explained that by connecting the highlighted alarm record data nodes corresponding to the devices with physical connection relationship, the alarm record data of the devices changes, indicating that the devices are abnormal. The abnormal devices will affect the abnormalities of the devices with physical connection relationship. Therefore, connecting the highlighted alarm record data nodes corresponding to the devices with physical connection relationship indicates that the abnormal devices and the devices with physical connection relationship affect each other. It needs to be explained that the traditional method constructs the knowledge graph according to logical reasoning or statistical learning and knowledge extraction or specific rules. Once the construction is completed, the connection relationship between the nodes will not change. However, in some cases, the connection relationship between the nodes will change with time. Therefore, the knowledge graph constructed by the traditional method is not real-time, and the connection relationship between the nodes is inaccurate, which leads to incorrect abnormal handling priority of the equipment obtained by analyzing the knowledge graph constructed by the traditional method, and then causes the order of abnormal handling to be disordered, causing huge losses to the factory. Therefore, the present invention constructs the initial industrial multimodal knowledge graph by constructing the initial industrial multimodal sub-knowledge graph of all equipment at the second time point, the third time point, ..., and the last time point to form an initial industrial multimodal knowledge graph, and improves the initial industrial multimodal knowledge graph to obtain the final industrial multimodal knowledge graph, so that the construction of the final industrial multimodal knowledge graph is real-time, ensuring the accuracy of the connection relationship of the nodes in the final industrial multimodal knowledge graph, and then making the abnormal handling priority of the equipment obtained by analyzing the final industrial multimodal knowledge graph correct, so that the abnormal handling of the equipment is carried out in an orderly manner, further reducing the losses caused by equipment maintenance in the factory.

[0022] The process of obtaining the equipment's exception handling priority evaluation coefficient based on the connection relationship between the nodes in the final industrial multimodal knowledge graph and obtaining the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient includes: The exception handling priority includes a first-level exception handling priority, a second-level exception handling priority, and a third-level exception handling priority; It should be explained that the priority level of the third-level exception handling priority is higher than the priority level of the second-level exception handling priority, which is higher than the priority level of the first-level exception handling priority; Obtain the number of remaining nodes connected to the highlighted alarm record data node at each time point in the final industrial multimodal knowledge graph, and use the number of remaining nodes connected to the highlighted alarm record data node as the abnormality processing priority evaluation coefficient of the corresponding device at the corresponding time point; Set the threshold range of the exception handling priority evaluation coefficient. The threshold range of the exception handling priority evaluation coefficient can be set through experimental data analysis or experience. When the exception handling priority evaluation coefficient is less than the minimum value in the threshold range of the exception handling priority evaluation coefficient, the exception handling priority of the device at that time point is recorded as the first-level exception handling priority; When the exception handling priority evaluation coefficient is greater than or equal to the minimum value in the exception handling priority evaluation coefficient threshold range and less than or equal to the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the secondary exception handling priority; When the exception handling priority evaluation coefficient is greater than the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the third-level exception handling priority; It should be explained that the more the number of other nodes connected to the highlighted alarm record data node is, the greater the problem of the device is, the higher the exception handling priority of the device is, and the device is processed first. For example, the highlighted alarm record data node of device one is connected to the sensor data node, the alarm description data node and one other highlighted alarm record data node, and the highlighted alarm record data node of device two is connected to the sensor data node, the alarm description data node, the operation and maintenance log data node and two other highlighted alarm record data nodes. Device two is connected to one more operation and maintenance log data node, so the problem of device two is greater, resulting in changes in the operation and maintenance log data, that is, the problem of device two triggers the other problems of device two, that is, device two has more problems and a higher degree of abnormality, and device two is connected to one more other highlighted alarm record data node, indicating that the problem of device two may trigger problems of the other two devices, while the problem of device one may only trigger problems of the other one device, so the exception handling priority of device two is higher than that of device one, and the highlighted alarm record data node of device two is connected to more nodes; The process of handling device exceptions according to the device's exception handling priority includes: Obtain the location of the equipment with the third level of exception handling priority, dispatch relevant personnel to the location of the equipment with the third level of exception handling priority, and the relevant personnel repair the equipment with the third level of exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the third level of exception handling priority; Obtain the location of the equipment with the second-level exception handling priority, dispatch relevant personnel to the location of the equipment with the second-level exception handling priority, and the relevant personnel repair the equipment with the second-level exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the second-level exception handling priority; Obtain the location of the equipment with the first-level exception handling priority, dispatch relevant personnel to the location of the equipment with the first-level exception handling priority, and the relevant personnel repair the equipment with the first-level exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the first-level exception handling priority.

[0023] In this embodiment, the initial industrial multimodal sub-knowledge graph of each device at different time points is obtained through the mutual influence relationship between multimodal data and multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph. The initial industrial multimodal knowledge graph is improved to obtain the final industrial multimodal knowledge graph. The connection relationship of nodes at different time points changes. Industrial multimodal sub-knowledge graphs are constructed for different time points respectively, and are improved according to the physical connection relationship between devices to obtain the final industrial multimodal knowledge graph, so that different time points have corresponding knowledge graphs, and the final industrial multimodal knowledge graph is real-time, ensuring the final industrial The accuracy of the connection relationship between the nodes in the industrial multimodal knowledge graph makes the equipment exception handling priority obtained by analyzing the final industrial multimodal knowledge graph correct, so that the equipment exception handling is carried out in an orderly manner, further reducing the factory's losses caused by equipment maintenance; the equipment exception handling priority evaluation coefficient is obtained according to the connection relationship between the nodes in the final industrial multimodal knowledge graph, the equipment exception handling priority is obtained through the equipment exception handling priority evaluation coefficient, and the equipment exception is handled according to the equipment exception handling priority. The equipment exception handling priority is obtained quickly and accurately, thereby speeding up the equipment maintenance time and further reducing the factory's losses caused by equipment maintenance.

[0024] Example 2 See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, and a multimodal data association analysis system for an industrial control system based on a knowledge graph is provided, including: Data reading component, responsible for reading multimodal data of devices in industrial control networks; The graph construction component is responsible for obtaining the mutual influence relationship between multimodal data. The initial industrial multimodal sub-knowledge graph of each device at different time points is obtained through the mutual influence relationship between multimodal data and multimodal data. The initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph. The graph improvement component is responsible for improving the initial industrial multimodal knowledge graph to obtain the final industrial multimodal knowledge graph; The exception handling component is responsible for obtaining the equipment's exception handling priority evaluation coefficient based on the connection relationship between nodes in the final industrial multimodal knowledge graph, obtaining the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and performing exception handling on the equipment based on the equipment's exception handling priority.

[0025] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0026] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0027] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0028] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal data association analysis method for industrial control systems based on knowledge graph, characterized in that: The multimodal data association analysis method of the industrial control system based on the knowledge graph includes: Step S1: reading multimodal data of devices in the industrial control network; Step S2: judging whether there is a connection between the multimodal data according to the purpose of the read multimodal data, obtaining the mutual influence relationship between the multimodal data according to the judgment result, and obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points through the mutual influence relationship between the multimodal data and the multimodal data, and the initial industrial multimodal sub-knowledge graphs of all devices at different time points constitute the initial industrial multimodal knowledge graph; Step S3: improving the initial industrial multimodal knowledge graph, connecting the corresponding nodes in the initial industrial multimodal knowledge graph according to the physical connection relationship between the devices, and obtaining the final industrial multimodal knowledge graph; Step S4: Obtain the equipment's exception handling priority evaluation coefficient based on the connection relationship between nodes in the final industrial multimodal knowledge graph, obtain the equipment's exception handling priority through the equipment's exception handling priority evaluation coefficient, and perform exception handling on the equipment based on the equipment's exception handling priority.

2. According to claim 1, the method for multimodal data association analysis of industrial control systems based on knowledge graph is characterized in that: The multimodal data includes structured data, unstructured data, image data, video data and text information data; The structured data includes sensor data, control parameter data, equipment operation data and alarm record data, and the unstructured data includes operation and maintenance log data, alarm description data and event record data; The image data includes instrument reading image data, the video data includes equipment working screen video data, and the text information data includes operation and maintenance manual data, fault case library data, and training document data.

3. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 2 is characterized in that: The method for determining whether there is a connection between multimodal data according to the purpose of the read multimodal data includes: The sensor data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, instrument reading image data and equipment working screen video data are used for equipment abnormality analysis, the control parameter data are used for control strategy evaluation, the event record data are used for evolution trajectory analysis, and the operation and maintenance manual data, fault case library data and training document data are used for safe operation and maintenance guidance; If the purposes of multimodal data are the same, there is a connection between the multimodal data. If the purposes of multimodal data are different, there is no connection between the multimodal data.

4. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 3 is characterized in that: The method for obtaining the mutual influence relationship between multimodal data according to the judgment result includes: Get the start time and end time of reading multimodal data of devices in the industrial control network, and divide the time period from the start time to the end time into time points, and obtain the multimodal data of the device corresponding to each time point; Obtaining multimodal data that changes from a first time point to a second time point based on the multimodal data of the device corresponding to each time point, and if there is a connection between the multimodal data that changes from the first time point to the second time point, recording the relationship between the multimodal data that changes from the first time point to the second time point as a second mutual influence relationship; The multimodal data that changes from the second time point to the third time point is obtained based on the multimodal data of the device corresponding to each time point. If there is a connection between the multimodal data that changes from the second time point to the third time point, the relationship between the multimodal data that changes from the second time point to the third time point is recorded as the third mutual influence relationship, and so on, until the multimodal data that changes from the second-to-last time point to the penultimate time point is obtained. If there is a connection between the multimodal data that changes from the second-to-last time point to the penultimate time point, the relationship between the multimodal data that changes from the second-to-last time point to the penultimate time point is recorded as the last mutual influence relationship.

5. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 4 is characterized in that: The method for obtaining the initial industrial multimodal sub-knowledge graph of each device at different time points through multimodal data and the mutual influence relationship between multimodal data includes: The initial industrial multimodal knowledge graph is composed of time nodes and multimodal data nodes, and the multimodal data nodes include sensor data nodes, control parameter data nodes, equipment operation data nodes, alarm record data nodes, operation and maintenance log data nodes, alarm description data nodes, event record data nodes, instrument reading image data nodes, equipment working screen video data nodes, operation and maintenance manual data nodes, fault case library data nodes and training document data nodes; Label the equipment as , , is a positive integer, the second time point is stored in the time node, and the first "The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the corresponding multimodal data node respectively; The time nodes are connected to each multimodal data node respectively, the nodes corresponding to the multimodal data that have changed from the first time point to the second time point and are recorded as the second mutual influence relationship are connected to each other, and the nodes corresponding to the multimodal data that have changed from the first time point to the second time point are highlighted to obtain the " "The initial industrial multimodal sub-knowledge graph of the device at the second time point is obtained, and the initial industrial multimodal sub-knowledge graph of the remaining devices at the second time point is obtained by the same method; The same method is used to obtain the initial industrial multimodal sub-knowledge graphs of all equipment at the third time point, the fourth time point, ..., the last time point; The initial industrial multimodal knowledge graphs of all devices at the second time point, the third time point, ..., and the last time point constitute the initial industrial multimodal knowledge graph.

6. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 5 is characterized in that: The method of storing in the corresponding multimodal data node includes: The second time point corresponds to the The sensor data, control parameter data, equipment operation data, alarm record data, operation and maintenance log data, alarm description data, event record data, instrument reading image data, equipment working screen video data, operation and maintenance manual data, fault case library data and training document data of each device are stored in the sensor data node, control parameter data node, equipment operation data node, alarm record data node, operation and maintenance log data node, alarm description data node, event record data node, instrument reading image data node, equipment working screen video data node, operation and maintenance manual data node, fault case library data node and training document data node respectively.

7. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 6 is characterized in that: The method for obtaining the final industrial multimodal knowledge graph includes: According to the initial industrial multimodal sub-knowledge graph of all devices at the second time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the second time point is obtained and recorded as the second highlighted alarm record data node, and the physical connection relationship between the devices is obtained. If there is a physical connection relationship between the devices corresponding to the second highlighted alarm record data node, the second highlighted alarm record data nodes having a physical connection relationship between the corresponding devices are connected to each other; According to the initial industrial multimodal sub-knowledge graph of all devices at the third time point, the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the third time point is obtained and recorded as the third highlighted alarm record data node; if there is a physical connection relationship between the devices corresponding to the third highlighted alarm record data node, the third highlighted alarm record data nodes having a physical connection relationship between the corresponding devices are connected to each other; And so on, until the alarm record data node highlighted in the initial industrial multimodal sub-knowledge graph at the last time point is obtained based on the initial industrial multimodal sub-knowledge graph of all devices at the last time point, and recorded as the last highlighted alarm record data node. If there is a physical connection relationship between the devices corresponding to the last highlighted alarm record data node, the last highlighted alarm record data nodes with a physical connection relationship between the corresponding devices are interconnected to obtain the final industrial multimodal knowledge graph.

8. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 7 is characterized in that: The method for obtaining the abnormality handling priority evaluation coefficient of the equipment according to the connection relationship between the nodes in the final industrial multimodal knowledge graph includes: The number of remaining nodes connected to the highlighted alarm record data node at each time point in the final industrial multimodal knowledge graph is obtained, and the number of remaining nodes connected to the highlighted alarm record data node is used as the abnormality handling priority evaluation coefficient of the corresponding device at the corresponding time point.

9. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 8 is characterized in that: The method for obtaining the exception handling priority of a device through the exception handling priority evaluation coefficient of the device includes: The exception handling priority includes a first-level exception handling priority, a second-level exception handling priority, and a third-level exception handling priority; Set an exception handling priority evaluation coefficient threshold range. When the exception handling priority evaluation coefficient is less than the minimum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the first-level exception handling priority. When the exception handling priority evaluation coefficient is greater than or equal to the minimum value in the exception handling priority evaluation coefficient threshold range and less than or equal to the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the secondary exception handling priority; When the exception handling priority evaluation coefficient is greater than the maximum value in the exception handling priority evaluation coefficient threshold range, the exception handling priority of the device at that time point is recorded as the third-level exception handling priority.

10. The method for multimodal data association analysis of industrial control systems based on knowledge graph according to claim 9 is characterized in that: The method for performing exception handling on a device according to the exception handling priority of the device comprises: Obtain the location of the equipment with the third level of exception handling priority, dispatch relevant personnel to the location of the equipment with the third level of exception handling priority, and the relevant personnel repair the equipment with the third level of exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the third level of exception handling priority; Obtain the location of the equipment with the second-level exception handling priority, dispatch relevant personnel to the location of the equipment with the second-level exception handling priority, and the relevant personnel repair the equipment with the second-level exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the second-level exception handling priority; Obtain the location of the equipment with the first-level exception handling priority, dispatch relevant personnel to the location of the equipment with the first-level exception handling priority, and the relevant personnel repair the equipment with the first-level exception handling priority according to the data in the final industrial multimodal knowledge graph to eliminate the anomaly of the equipment with the first-level exception handling priority.

Images