Multi-modal fusion equipment characterization method based on knowledge graph

Abstract

The invention belongs to the technical field of intelligent operation and maintenance of nuclear power plants, and particularly relates to a multi-modal fusion equipment characterization method based on a knowledge graph. Comprising the following steps: step 1, constructing a knowledge graph; step 2, carrying out multi-modal fusion; and step 3, combining the knowledge graph and the multi-modal data. The knowledge graph embedding method has the beneficial effects that knowledge graph embedding is a technology for mapping entities and relationships in a knowledge graph to a low-dimensional vector space, and as a standard knowledge graph technology, the capability and convergence of the knowledge graph embedding method are proved by academic and industrial practices. According to the technology, complex knowledge graph information can be converted into a mathematical form easy to process, meanwhile, the richness and interpretability of data are kept, the technology can be fused with multi-modal data, unstructured data such as texts and images are combined with the knowledge graph, and the ability of the technology is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent operation and maintenance of nuclear power plants, and specifically relates to a multimodal fusion equipment characterization method based on knowledge graph. Background Art

[0002] Knowledge graph is a semantic network used to describe the relationship between entities. It is usually used to build artificial intelligence applications such as intelligent search engines, recommendation systems, and question-answering systems. Its core idea is to transform information in the real world into a graph, where nodes represent entities and edges represent the relationship between entities. Knowledge graph has multiple technical genes of artificial intelligence, big data, and the Internet. It is a comprehensive integration of technologies in multiple fields such as knowledge representation, representation learning, natural language processing, graph databases, and graph computing.

[0003] Multimodal fusion refers to the integration of information obtained from different perception modes (such as vision, speech, touch, etc.) to improve the performance and intelligence of the system. Multimodal fusion methods can be divided into model-independent methods and model-based methods.

[0004] As the core object in the nuclear power field, various studies on equipment have been going on for many years. The complex associated information of the equipment itself, including but not limited to the equipment itself, equipment-related documents, equipment operating status, equipment historical information, etc., makes it very difficult to establish a model to describe the equipment.

[0005] Most of the existing intelligent technologies for equipment are targeted at modeling a certain type of information or a few types of information of the equipment. They can neither effectively utilize all the equipment information nor effectively reuse and complement the accumulated equipment data in multiple intelligent models. In addition, for different equipment, the existing methods basically use targeted modeling methods, which cannot be generalized to other equipment or other fields, nor can they have cross-equipment robustness. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal fusion device characterization method based on knowledge graph. Through the multimodal fusion method based on knowledge graph, all device-related information is modeled as a whole, and many intelligent models are constructed across devices and even across fields, greatly enhancing the construction of related intelligent models.

[0007] The technical solution of the present invention is as follows: A multimodal fusion device characterization method based on knowledge graph comprises the following steps:

[0008] Step 1: Knowledge graph construction;

[0009] Step 2: Multimodal fusion;

[0010] Step 3: Combine knowledge graph and multimodal data.

[0011] The step 1 comprises the following steps:

[0012] Step 11: Graph relationship design

[0013] A five-layer system is designed around the equipment, as shown in Table 1

[0014] Table 1

[0015]

[0016] Step 12: Graph data processing

[0017] According to the collected data, secondary data processing is performed, including but not limited to: data filtering: filtering after invalid data is determined, data deduplication: removing duplicate data, data standardization: reasonably standardizing data of different dimensions and ranges;

[0018] Step 13: Graph Construction

[0019] According to the graph relationship design and the collected data in step 11, manual or automatic triple construction is performed, i.e., "entity-relationship-entity" construction;

[0020] Step 14: Graph Maintenance

[0021] For changes in equipment and related information, if new information is added, manual or automatic triples are constructed, and then the new triples are added to the original graph; if existing information is changed, modifications are made based on the node and edge information of the existing graph, including but not limited to the deletion, addition, and attribute modification of nodes and edges.

[0022] The step 2 comprises the following steps:

[0023] Step 21: Modeling equipment operating status and related information

[0024] First, the equipment operation status is disassembled into historical operation data of multiple measurement points and historical maintenance / inspection / evaluation reports. For the historical operation data of multiple measurement points, a recurrent neural network + attention mechanism is used to obtain feature information. For historical maintenance / inspection / evaluation reports, a word to vector model is used. Each type of vector is directly accumulated at the corresponding position after dimension compression, and then an attention layer is added to perform prediction.

[0025] Step 22: Information Fusion

[0026] After obtaining the operation data vector and historical maintenance / inspection / evaluation report vector and building the model structure, the fusion of each modality data is achieved by pre-training the equipment model.

[0027] The step 3 includes the following: the graph is trained using the optimized knowledge graph embedding method.

[0028] The optimized method in step 3 is to add the device operation status and other device information as an attribute of the device node to the knowledge graph embedding training, that is, h = h n +h r , where h n represents the graph node vector, h r Represents the vector obtained after multimodal pre-training of the device model.

[0029] The beneficial effects of the present invention are: Knowledge graph embedding is a technology that maps entities and relationships in a knowledge graph to a low-dimensional vector space. As a standard knowledge graph technology, its capabilities and convergence have been proven in practice by academia and industry. This technology can convert complex knowledge graph information into an easy-to-process mathematical form while maintaining the richness and interpretability of the data. It can be fused with multimodal data, combining unstructured data such as text and images with knowledge graphs to enhance its capabilities. Multimodal fusion is based on knowledge graph embedding, and additionally enhances the expression of its entity information, that is, in the scoring function (taking the rotateE method as an example):

[0030]

[0031] k represents the vector dimension, and h represents the entity vector. Here we can roughly think of it as:

[0032] h=h n +h r

[0033] where h n represents the graph node vector, h r It represents the vector after multimodal fusion. Here, the entity information expression is additionally enhanced by vector superposition. Under the premise that there are no large-scale errors in the data, it will only enhance the final effect of knowledge graph embedding. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Modeling flow charts for equipment operating status;

[0035] Figure 2 It is a schematic diagram of multi-information fusion;

[0036] Figure 3 A schematic diagram of the entire process. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] A multimodal fusion device characterization method based on knowledge graph includes the following steps:

[0039] Step 1: Knowledge graph construction

[0040] Step 11: Graph relationship design

[0041] From a general perspective, a five-layer system is designed around the device, as shown in Table 1.

[0042] Table 1

[0043]

[0044] Step 12: Graph data processing

[0045] Based on the collected data, reasonable data secondary processing is carried out, including but not limited to:

[0046] Data filtering: Filter after invalid data is determined.

[0047] Data deduplication: remove duplicate data.

[0048] Data standardization: Reasonably standardize data of different dimensions and ranges.

[0049] Step 13: Graph Construction

[0050] According to the graph relationship design and the collected data in step 11, manual or automatic triple construction is performed, that is, "entity-relationship-entity" construction. If there is clear association information in the data, it can be constructed directly. If not, it needs to be constructed manually or other model algorithms need to be used to speed up the construction.

[0051] Step 14: Graph Maintenance

[0052] For changes in equipment and related information, if it is completely new information added, such as new equipment or new environmental information, manual or automatic triplet construction can be performed, and then the new triplet information can be added to the original graph; if the existing information is changed, the node and edge information of the existing graph is modified, including but not limited to the deletion, addition, and attribute modification of nodes and edges.

[0053] Step 2: Multimodal Fusion

[0054] Step 21: Modeling equipment operating status and related information

[0055] Reference Figure 1First, the equipment operation status is disassembled into historical operation data of multiple measurement points, historical maintenance / inspection / evaluation reports, etc. For the historical operation data of multiple measurement points, a recurrent neural network (RNN) + attention mechanism is used to obtain feature information; for historical maintenance / inspection / evaluation reports, etc., a suitable word to vector model is used, such as a pre-trained large model to convert document information into vectors, where chart information needs to be converted into vectors using a suitable OCR or image model. After dimensional compression (i.e., all projected to a fixed dimension, such as n*k), each type of vector is directly accumulated at the corresponding position, and then the attention layer is added for prediction.

[0056] Step 22: Information Fusion

[0057] After obtaining the operation data vector and content vectors such as historical maintenance / inspection / evaluation reports and building the model structure, it is necessary to realize the fusion of data of each modality by pre-training the equipment model. The essential idea refers to the classic CLIP network, that is, using the data of modality A to predict the data of modality B, so that the model can learn the deep connection between different modalities. Therefore, the goal of pre-training the equipment model is to randomly select a certain equipment feature vector as the content to be predicted, and use other input vectors to let the model predict the feature vector to be predicted (for example, there are 10 input vectors from 1 to 10, select 7 as the content to be predicted, and use 1-6, 8-10 to predict 7).

[0058] Step 3: Combine knowledge graph and multimodal data

[0059] Reference Figure 2 In this method, the graph needs to be trained using the optimized knowledge graph embedding method. Taking the rotateE method as an example, its scoring function is: The optimized method is to take the device operation status and other device information as an attribute of the device node and add them to the knowledge graph embedding training. That is, h = h n +h r , where h n represents the graph node vector, h r Represents the vector obtained after multimodal pre-training of the device model.

[0060] Example:

[0061] like Figure 3 As shown, a multimodal fusion device characterization method based on knowledge graph includes the following steps:

[0062] Data collection: obtain existing equipment information, equipment-related measurement point data, equipment-related document information, equipment-related maintenance / inspection information, etc.; obtain the system where the equipment is located, the address of the equipment, equipment-related events, etc.

[0063] Graph structure design: From a general perspective, a 5-layer system is designed around the equipment, as shown in the following table.

[0064]

[0065] Language / vision model selection: According to the actual situation of the device, select the appropriate language or vision model, such as BERT and YOLO.

[0066] Data filtering: Filter after invalid data is determined.

[0067] Data deduplication: remove duplicate data.

[0068] Data standardization: Reasonably standardize data of different dimensions and ranges.

[0069] Data processing: data completion through manual or algorithmic means, etc.

[0070] Basic graph data: According to the graph design plan and the collected data, manual or automatic triples are constructed, that is, "entity-relationship-entity" construction.

[0071] Equipment operating status modeling: reference Figure 1 First, the equipment operation status is disassembled into historical operation data of multiple measurement points, historical maintenance / inspection / evaluation reports, etc. For the historical operation data of multiple measurement points, a recurrent neural network (RNN) + attention mechanism is used to obtain feature information; for historical maintenance / inspection / evaluation reports, etc., a suitable word to vector model is used, such as a pre-trained large model to convert document information into vectors, where chart information needs to be converted into vectors using a suitable OCR or image model. After dimensional compression (i.e., all projected to a fixed dimension, such as n*k), each type of vector is directly accumulated at the corresponding position, and then the attention layer is added to make predictions.

[0072] After obtaining the operation data vector and content vectors such as historical maintenance / inspection / evaluation reports and building the model structure, it is necessary to realize the fusion of data of each modality by pre-training the equipment model. The essential idea refers to the classic CLIP network, that is, using the data of modality A to predict the data of modality B, so that the model can learn the deep connection between different modalities. Therefore, the goal of pre-training the equipment model is to randomly select a certain equipment feature vector as the content to be predicted, and use other input vectors to let the model predict the feature vector to be predicted (for example, there are 10 input vectors from 1 to 10, select 7 as the content to be predicted, and use 1-6, 8-10 to predict 7).

[0073] Graph embedding learning: Algorithms such as transE, transR, rotateE can be used.

[0074] Graph storage and maintenance: Any graph database can be used for storage. If new information is added, such as new equipment or new environment information, manual or automatic triple construction can be performed, and then the new triple information can be added to the original graph; if existing information is changed, the nodes and edges of the existing graph are modified, including but not limited to the deletion, addition, and attribute modification of nodes and edges.

[0075] Device representation output: Design and develop APIs to obtain device embedding expressions from graph databases.

Claims

1. A multimodal fusion device characterization method based on knowledge graph, characterized in that: The steps include: Step 1: Knowledge graph construction; Step 2: Multimodal fusion; Step 3: Combine knowledge graph and multimodal data.

2. A method for characterizing a multimodal fusion device based on a knowledge graph as claimed in claim 1, characterized in that: The step 1 comprises the following steps: Step 11: Graph relationship design; Step 12: Spectral data processing; Step 13: Graph construction; According to the graph relationship design and the collected data in step 11, manual or automatic triple construction is performed, i.e. "entity-relationship-entity" construction; Step 14: Graph maintenance.

3. A method for characterizing a multimodal fusion device based on a knowledge graph as claimed in claim 2, characterized in that: The step 11 comprises: A five-layer system is designed around the equipment, as shown in Table 1 Table 1 4. The method for characterizing a multimodal fusion device based on a knowledge graph as claimed in claim 2, characterized in that: The step 12 comprises: Based on the collected data, secondary data processing is performed, including but not limited to: data filtering: filtering after invalid data is determined, data deduplication: removing duplicate data, data standardization: reasonable standardization of data of different dimensions and ranges.

5. The method for characterizing a multimodal fusion device based on a knowledge graph as claimed in claim 2, characterized in that: The step 14 comprises: For changes in equipment and related information, if new information is added, manual or automatic triples are constructed, and then the new triples are added to the original graph; if existing information is changed, modifications are made based on the node and edge information of the existing graph, including but not limited to the deletion, addition, and attribute modification of nodes and edges.

6. The method for characterizing a multimodal fusion device based on a knowledge graph according to claim 1, characterized in that: The step 2 comprises the following steps: Step 21: Modeling of equipment operation status and related information; Step 22: Information fusion.

7. A method for characterizing a multimodal fusion device based on a knowledge graph as claimed in claim 6, characterized in that: The step 21 includes first disassembling the equipment operation status into historical operation data of multiple measuring points and historical maintenance / inspection / evaluation reports. For the historical operation data of multiple measuring points, a recurrent neural network + attention mechanism is used to obtain feature information; for the historical maintenance / inspection / evaluation reports, a word to vector model is used, and each type of vector is directly accumulated at the corresponding position after dimension compression, and then an attention layer is added to perform prediction.

8. The method for characterizing a multimodal fusion device based on a knowledge graph according to claim 6, characterized in that: The step 22 includes obtaining the operation data vector and the historical maintenance / inspection / assessment report vector and building the model structure, and then realizing the fusion of each modality data by pre-training the equipment model.

9. The method for characterizing a multimodal fusion device based on a knowledge graph as claimed in claim 1, characterized in that: The step 3 includes the following: the graph is trained using the optimized knowledge graph embedding method.

10. A method for characterizing a multimodal fusion device based on a knowledge graph as claimed in claim 9, characterized in that: The optimized method in step 3 is to add the device operation status and other device information as an attribute of the device node to the knowledge graph embedding training, that is, h = h n +h r , where h n represents the graph node vector, h r Represents the vector obtained after multimodal pre-training of the device model.

Images