Multimodal power knowledge graph construction method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for constructing a multimodal electric power knowledge graph, including: determining the entity feature vectors of each entity involved in the electric power field in the same dimensional space based on the multimodal data in the electric power field; determining the neighbor entities of each entity through preset entity triples, and stratifying each neighbor entity according to the set rules; based on the entity relationship model formed after stratification, combined with the feature vectors of each entity, determining the comprehensive feature vector that characterizes the association relationship between each entity; based on the comprehensive feature vector, obtaining the electric power knowledge graph that characterizes the association relationship between each entity. The technical solution provided by the embodiment of the present invention characterizes the multimodal data in the electric power field with a unified comprehensive feature vector, providing data support for related intelligent applications and big data analysis in the electric power field.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing big data technology, and in particular, to a method, device, equipment and storage medium for constructing a multimodal power knowledge graph. Background Art

[0002] As the knowledge pillar in the field of artificial intelligence, knowledge graphs have attracted extensive attention from academia and industry for their powerful knowledge representation and reasoning capabilities. In recent years, knowledge graphs have been widely used in semantic search, question answering, knowledge management and other fields.

[0003] The representation learning of multimodal knowledge graphs is divided into intra-modal representation and inter-modal representation. The intra-modal representation is not much different from the representation learning of traditional knowledge graphs. The representation entities within the modality are in the same embedding space and can be learned directly. Due to the different data formats and encoding methods of multimodal data and different embedding spaces, there are still many challenges in inter-modal representation learning. For multimodal data in the power field, there are many types of knowledge and complex structures. In addition to relational data, there are many other forms of knowledge and data, such as time series data and multimedia data, which cannot break the barriers of multi-source heterogeneous knowledge. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, equipment and storage medium for constructing a multimodal power knowledge graph to achieve unified representation of multimodal data in the power field, and provide data support for related intelligent applications and big data analysis in the power field.

[0005] In a first aspect, an embodiment of the present invention provides a method for constructing a multimodal power knowledge graph, including:

[0006] Based on the multimodal data of the electric power field, determining the entity feature vectors of each entity involved in the electric power field in the same dimensional space;

[0007] Determine the neighbor entities of each entity through a preset entity triple, and stratify each neighbor entity according to a set rule;

[0008] Based on the entity relationship model formed after stratification, combined with the entity feature vectors, a comprehensive feature vector representing the association relationship between the entities is determined;

[0009] Based on the comprehensive feature vector, an electric power knowledge graph representing the association relationship between the entities is obtained.

[0010] In a second aspect, an embodiment of the present invention further provides a multimodal power knowledge graph construction device, including:

[0011] An entity feature vector determination module, used to determine the entity feature vectors of various entities involved in the power field in the same dimensional space based on multimodal data in the power field;

[0012] A stratification module, used to determine neighbor entities of each entity through a preset entity triple, and stratify each neighbor entity according to a set rule;

[0013] A comprehensive feature vector determination module, used to determine a comprehensive feature vector representing the association relationship between the entities based on the entity relationship model formed after stratification and in combination with the entity feature vectors;

[0014] The power knowledge graph acquisition module is used to obtain the power knowledge graph representing the association relationship between each of the entities based on the comprehensive feature vector.

[0015] In a third aspect, an embodiment of the present invention further provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multimodal power knowledge graph construction method as described in an embodiment of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processing device, implements the method for constructing a multimodal power knowledge graph as described in an embodiment of the present invention.

[0017] The embodiment of the present invention discloses a method, device, equipment and storage medium for constructing a multimodal electric power knowledge graph, including: determining the entity feature vectors of each entity involved in the electric power field in the same dimensional space based on the multimodal data in the electric power field; determining the neighbor entities of each entity through preset entity triples, and stratifying each neighbor entity according to the set rules; based on the entity relationship model formed after stratification, combined with the feature vectors of each entity, determining the comprehensive feature vector that characterizes the association relationship between each entity; based on the comprehensive feature vector, obtaining the electric power knowledge graph that characterizes the association relationship between each entity. The technical solution provided by the embodiment of the present invention characterizes the multimodal data in the electric power field with a unified comprehensive feature vector, providing data support for related intelligent applications and big data analysis in the electric power field. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a process of constructing a multimodal power knowledge graph provided in the first embodiment of the present invention;

[0019] Figure 2 An example diagram of entity relationships extracted from text data provided in the first embodiment of the present invention;

[0020] Figure 3 An example diagram of entity relationship extraction from image data provided in the first embodiment of the present invention;

[0021] Figure 4 An example diagram of the construction of a multimodal power knowledge graph provided in the first embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of a multimodal power knowledge graph construction device provided in the second embodiment of the present invention;

[0023] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0025] IMGpedia is a precedent for multimodal knowledge graphs. Combining semantic knowledge graphs with multimodal data faces challenges and opportunities in multiple tasks. IMGpedia uses four image descriptors for benchmarking, and the references and implementations of these descriptors are public. IMGpedia provides links to Wikidata. Since the classification in DBpedia is inconvenient for some visual semantic queries, IMGpedia aims to provide a better semantic query platform. IMGpedia is a good precedent in the multimodal direction, but there are also some problems, such as sparse relationship types, a small number of relationships, unclear image classification, etc., which are also problems that need to be solved in the future.

[0026] Representation learning of multimodal knowledge graphs is divided into intra-modal representation and inter-modal representation. Intra-modal representation is not much different from representation learning of traditional knowledge graphs. The representation entities within the modality are in the same embedding space and can be learned directly. Due to the different data formats and encoding methods of multimodal data and different embedding spaces, there are still many challenges in inter-modal representation learning.

[0027] In response to the above problems, embodiments of the present invention provide a method, device, equipment and storage medium for constructing a multimodal power knowledge graph.

[0028] Embodiment 1

[0029] Figure 1The present invention provides a flowchart of a method for constructing a multimodal electric power knowledge graph. This embodiment is applicable to the construction of a multimodal electric power knowledge graph. The method can be executed by a multimodal electric power knowledge graph construction device, which can be implemented by hardware / software and is generally integrated in a computer device. The method specifically includes the following steps:

[0030] S110. Based on the multimodal data in the electric power field, determine the entity feature vectors of each entity involved in the electric power field in the same dimensional space.

[0031] Among them, the data in the power field can include knowledge data in various forms, which can be in the form of text, image, etc. The data in the power field can be obtained from multiple databases, and there is no specific restriction here. In this embodiment, data containing image data and text data are defined as multimodal data, and data not containing image data are defined as unimodal data. Entity is a predefined data type, mainly referring to important information objects in the power field. The data in the power field contains various entities, such as transformers, generators, high-voltage lines and other entities.

[0032] In this embodiment, the entity feature vector can represent the relevant knowledge of the entity. In order to make the multimodal data uniformly represented, it is necessary to map the multimodal data to the same dimensional space for representation. The uniformly represented entity feature vector can contain the knowledge, image, sequence, text and image of the related entity.

[0033] Specifically, the acquired multimodal data in the power field, that is, different types of input data, such as image data, text data, etc., are encoded and mapped to the same dimensional space, and the entity feature vector of the unified representation of each entity can be obtained through cascade combination.

[0034] S120 . Determine neighbor entities of each entity through preset entity triples, and stratify each neighbor entity according to a set rule.

[0035] Among them, the entity triple represents the relationship between entities and can represent the relationship between entities and neighbor entities. Entity triples are obtained by performing entity extraction and entity relationship extraction on multimodal data in the power field. The setting rules can be to model each neighbor entity according to different relationships and different modes, and to model neighbor entities of different relationships and neighbor entities of different modes of each entity separately. According to the entities linked by multimodal data, they are divided into different layers according to data types. At the same time, for structured knowledge entity relationship data, they are divided into different layers according to different relationships.

[0036] Specifically, the preset entity triple contains the relationship between the entity and the neighbor entity. By obtaining the entity triple, multiple neighbor entities corresponding to the entity can be obtained. The neighbor entities of different relationships and neighbor entities of different modalities of each entity are modeled separately. According to the entities linked by multimodal data, they are divided into different layers according to the data type. At the same time, for structured knowledge entity relationship data, they are divided into different layers according to different relationships.

[0037] S130. Based on the entity relationship model formed after stratification, combined with the feature vectors of each entity, a comprehensive feature vector representing the association relationship between the entities is determined.

[0038] Among them, based on the entity relationship model formed after stratification, the graph convolution network (GCN) is used to aggregate the entity neighbor feature vectors of each layer, and then the information of different layers is processed for aggregation, and the entity feature vector of the entity is combined, that is, combined with its own information, to construct the comprehensive feature vector of the entity. The comprehensive feature vector contains the knowledge, image, sequence, text of the entity and the image, sequence text, knowledge and other information of the neighbor. The unified representation of the multimodal data knowledge graph is realized, providing important data support for other tasks.

[0039] S140. Based on the comprehensive feature vector, obtain an electric power knowledge graph that represents the association relationship between entities.

[0040] Among them, the power knowledge graph can be constructed by extracting information in the power field of various unstructured and structured data, expressing it as a professional knowledge structure to serve related applications in other fields. Specifically, the comprehensive feature vector is input into the decoder, and the decoder obtains the predicted links through internal setting operations, and the predicted links constitute the power knowledge graph that represents the association relationship between each entity.

[0041] The embodiment of the present invention discloses a method, device, equipment and storage medium for constructing a multimodal electric power knowledge graph, including: determining the entity feature vectors of each entity involved in the electric power field in the same dimensional space based on the multimodal data in the electric power field; determining the neighbor entities of each entity through preset entity triples, and stratifying each neighbor entity according to the set rules; based on the entity relationship model formed after stratification, combined with the feature vectors of each entity, determining the comprehensive feature vector that characterizes the association relationship between each entity; based on the comprehensive feature vector, obtaining the electric power knowledge graph that characterizes the association relationship between each entity. The technical solution provided by the embodiment of the present invention characterizes the multimodal data in the electric power field with a unified comprehensive feature vector, providing data support for related intelligent applications and big data analysis in the electric power field.

[0042] As an optional embodiment of the embodiment of the present invention, based on the above embodiment, before determining the entity feature vectors of each entity involved in the power field in the same dimensional space based on the multimodal data in the power field, it also includes:

[0043] a1) Define the data types of multimodal data in the power sector.

[0044] In this embodiment, the electric power knowledge graph can be understood as a resource description framework (RDF). Here, it is necessary to obtain entities and entity relationships in the multimodal data in the electric power field, design entity categories and relationship attributes for the multimodal data in the electric power field at the schema layer, and convert them into triples, that is, to obtain nodes and connection relationships in RDF.

[0045] In this step, the following basic data types are defined for professional power field data: entity, attribute, concept, relationship, picture link, text link, and sequence link. Among them, entity mainly refers to important information objects in the power field, including transformers, generators, high-voltage lines and other entities. Concepts refer to some basic professional concepts in the power field, including high voltage electricity, low voltage electricity, alternating current and other concepts. Relationships can be specifically understood as defining professional knowledge information between entities, between entities and concepts, and between concepts. Entity triples can be constructed through the relationship between entities. Picture links are entity pictures associated with entities, which define entity link relationships. Through the picture links of entities, intuitive information about the real appearance of entities can be obtained.

[0046] b1) Based on different entity extraction models, entity extraction is performed on the multimodal data in the power field to obtain various entities.

[0047] In this embodiment, the data in the electric power field includes multiple modal data, which can be text data or image data. Different entity extraction models are used for the multi-modal data to extract entities from text data, image data, etc.

[0048] Entity extraction from images can obtain physical objects contained in image data, such as transformers, power strips and other common entities in the power field. The association between actual entities and images can be constructed to build image links. Text data uses a set model to extract the actual entity annotation data contained in the text and build a description link from the entity to the text.

[0049] Optionally, based on different entity extraction models, entity extraction is performed on multimodal data in the power field respectively, and the steps of obtaining each entity can be expressed as:

[0050] If the modal data is text data, the modal data is input into a set deep learning model to obtain an entity corresponding to the modal data.

[0051] Specifically, if the modal data is text data, the named entity technology CRF+LSTM model is used to extract the actual annotation data contained in the text and construct a description link from the entity to the text.

[0052] If the modal data is image data, the modal data is input into a convolutional neural network model to obtain an entity corresponding to the modal data.

[0053] Specifically, if the modal data is image data, the Convolutional Neural Network (CNN) framework is used to extract entities from the image. After the image data is input into the CNN, the physical objects contained in the image will be output, such as transformers, power strips and other common entities in the power field. The association relationship between the actual entity and the image is constructed, and the image link is constructed.

[0054] c1) For unimodal data and multimodal data, different set data models are combined to obtain the entity relationships between each pair of entities.

[0055] For unstructured multimodal data, including text data and image data, it is necessary to extract the entity content, use the multimodal data of entity links, build a multimodal entity relationship extraction model, and completely transform the multimodal unstructured data into structured entity triple data. The single-modal relationship extraction based on the distance model + inter-modal relationship extraction can be used as the extraction method.

[0056] d1) Based on the indexing mechanism and the caching mechanism, the entities and entity relationships are stored in the form of entity triples.

[0057] In this step, the distributed database HBase is used to store the knowledge data of the cognitive graph in the power field. First, the existing standardized triple knowledge data is stored in the HBASE database using the RDF data format. At the same time, in order to store the text, sequence, image and other data information in the structure of the cognitive knowledge graph, the massive small file storage system based on HBASE realizes the low-latency indexing of the unstructured data of the cognitive knowledge graph in the power field. In order to be able to quickly retrieve structured data, Elasticsearch is used as the data retrieval index. At the same time, in view of the characteristics of data in the power field, this solution is based on the Redis caching mechanism. The program operation uses the memory database Redis to cache important and commonly used data information, which can realize the rapid retrieval of knowledge structured information and unstructured data information. The storage of multimodal data adopts the AbutionDB multimodal database, which supports the storage of dynamic and static knowledge graphs, time series event data, spatial data, machine learning features, relational data, and text data.

[0058] As an optional embodiment of the embodiment of the present invention, on the basis of the above embodiment, for unimodal data and multimodal data, respectively, in combination with different set data models, the steps of obtaining entity relationships between each pair of entities can be specifically expressed as follows:

[0059] If the modal data is text data, the modal data is input into a set derivative model, the modal data contains two entities whose relationship is to be predicted, and the entities are identified by symbols; based on the set derivative model, a relationship representation vector between the two entities is obtained; based on a multilayer perceptron, a relationship category between the two entities is obtained.

[0060] Specifically, the R-BERT model is used for relation extraction for unimodal data (i.e., text data). It is proposed to add special symbols between two entities for identification, such as separating two entities with ",". The R-BERT model outputs the relationship between entities based on the semantic information of the entity description text, and then constructs the entity relationship triples of sequence data.

[0061] Figure 2 This is an example diagram of entity relationships extracted from text data provided in the first embodiment of the present invention, such as Figure 2 As shown in the figure, text data is input into the BERT model, which can output the relationship representation vector between entities, and then pass through MLP to output the entity relationship between entities.

[0062] If the modal data is image data, the modal data is input into a convolutional neural network, where the modal data contains two entities whose relationship is to be predicted; the image semantic information of the entities is obtained and determined as a relationship representation vector between the two entities; the image semantic information of the entities is input into a multi-layer perceptron to obtain the relationship category between the two entities.

[0063] Specifically, for image data, this solution uses the CNN+MLP classification model. Figure 2 As shown, first, the model inputs the image data associated with the entity into CNN (convolutional neural network) to extract the image description semantics of the entity. Then, the image description semantics is used as the vector representation of the entity, and a multilayer perceptron (MLP) is trained as the decision network. The semantic information of the two entity images is input and the relationship information of the two entities is output.

[0064] Figure 3 This is an example diagram of extracting entity relationships from image data provided by the first embodiment of the present invention, such as Figure 3 As shown, the image data is input into CNN (convolutional neural network) to extract the image representation of the entity. Then, the image representation is used as the vector representation of the entity, and the MLP (multi-layer perceptron) is trained as the decision network. The two entity image representations are input and the entity relationship between the two entities is output.

[0065] Optionally, based on the multimodal data in the electric power field, the entity feature vectors of each entity involved in the electric power field in the same dimensional space are determined, including:

[0066] a2) Encode the multimodal data in the power field, obtain the first eigenvector and map it into the same dimensional space.

[0067] Specifically, different types of input data (such as image data, text data, etc.) are encoded into high-order latent vectors and mapped into the same dimensional space.

[0068] b2) According to the cascade relationship, the first feature vectors related to each entity are combined to obtain the entity feature vector corresponding to each entity.

[0069] Specifically, the entity feature vector corresponding to each entity is obtained through cascade combination.

[0070] Optionally, based on the comprehensive feature vector, the step of obtaining the electric power knowledge graph representing the association relationship between each entity can be specifically described as including: inputting the comprehensive feature vector into a trained decoder to obtain the electric power knowledge graph representing the association relationship between each entity.

[0071] Specifically, the comprehensive feature vector is input into the decoder to obtain the link prediction, that is, the entity relationship. The TransE model of traditional representation learning can be used as a decoder to output the link prediction result, and the link prediction result is used as the loss function for training.

[0072] In order to more clearly describe the embodiments of the present invention, illustratively, Figure 4This is an example diagram of the construction of a multimodal power knowledge graph provided in the first embodiment of the present invention. Figure 4 As shown in the figure, the steps of constructing the multimodal power knowledge graph include:

[0073] a3) Encode different types of input data (images, texts, etc.) into high-order latent vectors, map them to the same dimensional space, and obtain the entity feature vector of each entity in the same dimensional space through cascade combination.

[0074] b3) Determine the neighbor entities of each entity through preset entity triples, and layer each neighbor entity according to the set rules, model the neighbor entities of different relationships and neighbor entities of different modalities of each entity separately, and divide the entities linked by multimodal data into different layers according to data type. At the same time, for structured knowledge entity relationship data, divide it into different layers according to different relationships.

[0075] c3) Based on the entity relationship model formed after stratification, combined with the feature vectors of each entity, determine the comprehensive feature vector that characterizes the association relationship between the entities, use GCN to aggregate the entity neighbor vector representation of each layer, and then process the information of different layers for aggregation, and combine the entity’s own information to construct the final representation vector of the entity. Based on the multimodal data in the power field, determine the entity feature vectors of each entity involved in the power field in the same dimensional space.

[0076] d3) Based on the comprehensive feature vector, a link prediction between each of the entities is obtained through an encoder, and the link prediction result is used as a loss function for training, so as to form an electric power knowledge graph based on the link prediction.

[0077] As an optional embodiment, this embodiment concretizes the steps of obtaining preset entity triples for different modal data, and concretizes the steps of determining the entity feature vectors of each entity involved in the power field in the same dimensional space based on multimodal data in the power field and training and correcting parameters of the decoder, thereby realizing the unified representation of the multimodal data knowledge graph and providing important data support for other tasks.

[0078] Embodiment 2

[0079] Figure 5 A schematic diagram of the structure of a multimodal power knowledge graph construction device provided in the second embodiment of the present invention, the device comprises: an entity feature vector determination module 21, a stratification module 22, a comprehensive feature vector determination module 23 and a power knowledge graph acquisition module 24. Among them:

[0080] An entity feature vector determination module 21 is used to determine the entity feature vectors of various entities involved in the power field in the same dimensional space based on multimodal data in the power field;

[0081] A stratification module 22, used to determine neighbor entities of each entity through a preset entity triple, and stratify each neighbor entity according to a set rule;

[0082] A comprehensive feature vector determination module 23 is used to determine a comprehensive feature vector representing the association relationship between entities based on the entity relationship model formed after stratification and in combination with the feature vectors of each entity;

[0083] The power knowledge graph acquisition module 24 is used to obtain the power knowledge graph representing the association relationship between entities based on the comprehensive feature vector.

[0084] Optionally, the device further includes:

[0085] The data type definition module is used to define the data types of multimodal data in the power field;

[0086] The entity acquisition module is used to extract entities from multimodal data in the power field based on different entity extraction models to obtain various entities;

[0087] The entity relationship acquisition module is used to obtain the entity relationship between each pair of entities in combination with different set data models for single-modal data and multi-modal data;

[0088] The storage module is used to store entities and entity relationships in the form of entity triples based on an indexing mechanism and a caching mechanism.

[0089] Optionally, the entity acquisition module is specifically used to:

[0090] If the modal data is text data, the modal data is input into a set deep learning model to obtain an entity corresponding to the modal data;

[0091] If the modal data is image data, the modal data is input into a convolutional neural network model to obtain an entity corresponding to the modal data.

[0092] Optionally, the entity relationship acquisition module is specifically used for:

[0093] If the modal data is text data, the modal data is input into the set derivative model, the modal data contains two entities of the relationship to be predicted, and the entities are marked with symbols;

[0094] Based on the set derivative model, obtain the relationship representation vector between two entities;

[0095] Based on multi-layer perceptron, the relationship category between two entities is obtained.

[0096] Optionally, the entity relationship acquisition module is specifically used for:

[0097] If the modal data is image data, the modal data is input into the convolutional neural network, and the modal data includes two entities whose relationship is to be predicted;

[0098] Obtaining the image semantic information of the entity and determining it as a relationship representation vector between two entities;

[0099] The image semantic information of the entity is input into the multi-layer perceptron to obtain the relationship category between the two entities.

[0100] Optionally, the entity feature vector determination module 21 is specifically used for:

[0101] Encode the multimodal data in the power field, obtain the first eigenvector and map it into the same dimensional space;

[0102] According to the cascade relationship, the first feature vectors related to each entity are combined to obtain the entity feature vector corresponding to each entity.

[0103] The power knowledge graph acquisition module 24 is specifically used for:

[0104] The comprehensive feature vector is input into the trained decoder to obtain the power knowledge graph that represents the relationship between entities.

[0105] The above device can execute the method provided by the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0106] Embodiment 3

[0107] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown in FIG. Figure 6 As shown, the computer device includes a processor 31, a memory 32, an input device 33 and an output device 34; the number of processors 31 in the device can be one or more. Figure 6 A processor 31 is taken as an example; the processor 31, memory 32, input device 33 and output device 34 in the device can be connected by a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0108] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the modules corresponding to the multimodal power knowledge graph construction method in the embodiment of the present invention (for example, the entity feature vector determination module 21, the stratification module 22, the comprehensive feature vector determination module 23 and the power knowledge graph acquisition module 24). The processor 31 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 32, that is, realizes the above-mentioned multimodal power knowledge graph construction method.

[0109] The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0110] The input device 33 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device. The output device 34 may include a display device such as a display screen.

[0111] Embodiment 4

[0112] Embodiment 4 of the present invention further provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute a method for constructing a multimodal power knowledge graph when executed by a computer processor, the method comprising:

[0113] Based on the multimodal data of the electric power field, determining the entity feature vectors of each entity involved in the electric power field in the same dimensional space;

[0114] Determine the neighbor entities of each entity through a preset entity triple, and stratify each neighbor entity according to a set rule;

[0115] Based on the entity relationship model formed after stratification, combined with the entity feature vectors, a comprehensive feature vector representing the association relationship between the entities is determined;

[0116] Based on the comprehensive feature vector, an electric power knowledge graph representing the association relationship between the entities is obtained.

[0117] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the multimodal power knowledge graph construction method provided in any embodiment of the present invention.

[0118] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0119] It is worth noting that in the embodiment of the above-mentioned multimodal power knowledge graph construction device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0120] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for constructing a multimodal power knowledge graph, characterized in that: include: Based on the multimodal data in the electric power field, determine the entity feature vectors of various entities involved in the electric power field in the same dimensional space, wherein the data types of the multimodal data in the electric power field include: entities, attributes, concepts, relationships, picture links, text links and sequence links, entities refer to important information objects in the electric power field, including at least transformers, generators and high-voltage lines, concepts refer to basic professional concepts in the electric power field, including at least high voltage concepts, low voltage concepts and alternating current concepts, relationships refer to professional knowledge information between entities, between entities and concepts, and between concepts, and picture links refer to entity pictures associated with entities; Determine the neighbor entities of each entity through a preset entity triple, and stratify each neighbor entity according to a set rule; Based on the entity relationship model formed after stratification, combined with the entity feature vectors, a comprehensive feature vector representing the association relationship between the entities is determined; Based on the comprehensive feature vector, an electric power knowledge graph representing the association relationship between the entities is obtained; The method of determining the entity feature vectors of each entity involved in the power field in the same dimensional space based on the multimodal data in the power field includes: Encode the multimodal data in the power field, obtain the first eigenvector and map it into the same dimensional space; According to the cascade relationship, each of the entity-related first feature vectors is combined to obtain an entity feature vector corresponding to each entity; Wherein, the neighbor entities of each entity are determined by a preset entity triple, and each neighbor entity is layered according to a set rule, including: Obtain entity triples, and obtain multiple neighbor entities corresponding to each of the entities; Separately model neighbor entities of different relationships and neighbor entities of different modalities of each entity, and divide entities linked by multimodal data into different layers according to data types, and divide structured knowledge entity relationship data into different layers according to different relationships; The entity relationship model formed after stratification is combined with the entity feature vectors to determine a comprehensive feature vector representing the association relationship between the entities, including: Based on the entity relationship model formed after stratification, a graph neural network is used to aggregate the entity neighbor feature vectors of each layer, and then the aggregated entity neighbor feature vectors of different layers are processed for aggregation, and the entity feature vectors of the entity are combined to construct a comprehensive feature vector of the entity.

2. The method according to claim 1, characterized in that Before determining the entity feature vectors of each entity involved in the power field in the same dimensional space based on the multimodal data in the power field, the method further includes: Defining data types of multimodal data in the power field; Based on different entity extraction models, entity extraction is performed on the multimodal data in the power field to obtain various entities; For unimodal data and multimodal data, different set data models are combined to obtain the entity relationships between each pair of entities; Based on an indexing mechanism and a caching mechanism, the entities and the entity relationships are stored in the form of entity triples.

3. The method according to claim 2, characterized in that Based on different entity extraction models, entity extraction is performed on the multimodal data in the power field to obtain various entities, including: If the modal data is text data, the modal data is input into a set deep learning model to obtain an entity corresponding to the modal data; If the modal data is image data, the modal data is input into a convolutional neural network model to obtain an entity corresponding to the modal data.

4. The method according to claim 2, characterized in that: For the unimodal data and the multimodal data, different set data models are respectively combined to obtain the entity relationship between each pair of entities, including: If the modal data is text data, the modal data is input into a set derivative model, wherein the modal data includes two entities of the relationship to be predicted, and the entities are identified by symbols; Based on the set derivative model, obtaining a relationship representation vector between the two entities; Based on a multi-layer perceptron, the relationship category between the two entities is obtained.

5. The method according to claim 2, characterized in that: For the unimodal data and the multimodal data, different set data models are respectively combined to obtain the entity relationship between each pair of entities, including: If the modal data is image data, inputting the modal data into a convolutional neural network, wherein the modal data includes the two entities whose relationship is to be predicted; Acquire image semantic information of the entity and determine it as a relationship representation vector between the two entities; The image semantic information of the entity is input into a multi-layer perceptron to obtain the relationship category between the two entities.

6. The method according to claim 1, characterized in that The step of obtaining a power knowledge graph representing the association relationship between the entities based on the comprehensive feature vector includes: The comprehensive feature vector is input into a trained decoder to obtain an electric power knowledge graph that characterizes the association relationship between the entities.

7. A multimodal power knowledge graph construction device, characterized in that: include: An entity feature vector determination module is used to determine the entity feature vectors of various entities involved in the power field in the same dimensional space based on multimodal data in the power field, wherein the data types of the multimodal data in the power field include: entities, attributes, concepts, relationships, picture links, text links, and sequence links. Entities refer to important information objects in the power field, including at least transformers, generators, and high-voltage lines. Concepts refer to basic professional concepts in the power field, including at least high voltage concepts, low voltage concepts, and alternating current concepts. Relationships refer to professional knowledge information between entities, between entities and concepts, and between concepts. Picture links refer to entity pictures associated with entities. A stratification module, used to determine neighbor entities of each entity through a preset entity triple, and stratify each neighbor entity according to a set rule; A comprehensive feature vector determination module, used to determine a comprehensive feature vector representing the association relationship between the entities based on the entity relationship model formed after stratification and in combination with the entity feature vectors; An electric power knowledge graph acquisition module, used to obtain an electric power knowledge graph representing the association relationship between the entities based on the comprehensive feature vector; The entity feature vector determination module is specifically used for: Encode the multimodal data in the power field, obtain the first eigenvector and map it into the same dimensional space; According to the cascade relationship, the first feature vectors related to each entity are combined to obtain the entity feature vector corresponding to each entity; Layered modules, specifically for: Obtain entity triples, and obtain multiple neighbor entities corresponding to each of the entities; Separately model neighbor entities of different relationships and neighbor entities of different modalities of each entity, and divide entities linked by multimodal data into different layers according to data types, and divide structured knowledge entity relationship data into different layers according to different relationships; The comprehensive feature vector determination module is specifically used for: Based on the entity relationship model formed after stratification, a graph neural network is used to aggregate the entity neighbor feature vectors of each layer, and then the aggregated entity neighbor feature vectors of different layers are processed for aggregation, and the entity feature vectors of the entity are combined to construct a comprehensive feature vector of the entity.

8. A computer device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal power knowledge graph construction method as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, it implements the multimodal power knowledge graph construction method as described in any one of claims 1-6.

Images