Knowledge graph construction method and device, equipment, storage medium and computer product

Abstract

The invention discloses a knowledge graph construction method and device, equipment, a storage medium and a computer product, relates to the technical field of knowledge graphs, and discloses a knowledge graph construction method which comprises the following steps: constructing a plurality of single-mode knowledge graphs according to multi-mode information to be arranged; linking each node extracted from each single-mode knowledge graph to an external knowledge graph for alignment to obtain a multi-mode synthesis graph; and performing information completion on the multi-modal synthesis graph according to the external knowledge graph. By means of the mode, the external knowledge graph is introduced, the nodes extracted from the information of the multiple modes are linked to the same external knowledge graph, the constructed graph is expanded and supplemented through the information existing in the external knowledge graph, complex and diverse data sources in the open source intelligence field are effectively utilized, and the information extraction efficiency is improved. And graph construction is expanded by using external knowledge.

Description

Technical Field

[0001] This application relates to the field of knowledge graph technology, and in particular to knowledge graph construction methods, apparatus, devices, storage media and computer products. Background Technology

[0002] In the field of open-source intelligence, information sources are typically complex, including multiple modalities such as text, video, audio, and tables. When constructing knowledge graphs, node extraction techniques are usually used to build nodes, and node relationship extraction techniques are used to build edges between nodes. However, the graphs constructed using only a single modality extraction technique are very limited, and the information from each modality is not fully utilized or verified.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device, storage medium and computer product for constructing a knowledge graph, which aims to solve the technical problem that the information in the graph constructed by single-modality extraction technology is limited and the information of each modality is not fully utilized and verified.

[0005] To achieve the above objectives, this application proposes a knowledge graph construction method, which includes:

[0006] Multiple unimodal knowledge graphs were constructed based on the multimodal information to be processed;

[0007] Each node extracted from each unimodal knowledge graph is linked to an external knowledge graph for alignment, resulting in a multimodal composite graph.

[0008] Information is supplemented into the multimodal synthetic graph based on the external knowledge graph.

[0009] Optionally, the step of constructing multiple unimodal knowledge graphs based on the multimodal information to be processed includes:

[0010] Obtain the multimodal information to be processed, and obtain multiple single-modal information based on the multimodal information;

[0011] Multiple unimodal knowledge graphs are constructed based on the unimodal information.

[0012] Optionally, when the single-modal information is text modal information;

[0013] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0014] Name entities based on the text modality information to obtain an entity list.

[0015] Based on the entity list information, relationship extraction is performed to obtain the entity relationship list information;

[0016] Based on the entity relationship list information, attribute extraction is performed to obtain the attribute list information;

[0017] A text modal knowledge graph is constructed based on the attribute list information.

[0018] Optionally, when the single-modal information is image modal information;

[0019] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0020] The image entity is determined based on the image modal information;

[0021] Co-occurrence relationship information is determined based on the image recognition entities;

[0022] A knowledge graph of image modalities is constructed based on the image recognition entities and the co-occurrence relationship information.

[0023] Optionally, when the single-modal information is video modal information;

[0024] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0025] The video recognition entity is determined based on the video modal information;

[0026] Behavior detection and video event detection are performed on the video recognition entities to obtain video entity relationship information;

[0027] A video modal knowledge graph is constructed based on the video-identified entities and the relationship information between the video entities.

[0028] Optionally, when the single-modal information is audio modal information;

[0029] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0030] The audio modal information is converted to obtain the recognized text information;

[0031] A text modality knowledge graph is constructed based on the identified text information.

[0032] Optionally, the step of linking each node extracted from each unimodal knowledge graph to an external knowledge graph for alignment to obtain a multimodal synthetic graph includes:

[0033] By using a preset text processing strategy, nodes extracted from the text modality knowledge graph and / or audio modality knowledge graph in the unimodal knowledge graph are linked to an external knowledge graph for alignment, and / or by using a preset image processing strategy, nodes extracted from the image modality knowledge graph and / or video modality knowledge graph in the unimodal knowledge graph are linked to the external knowledge graph for alignment, thus obtaining a multimodal composite graph.

[0034] Optionally, the step of linking the nodes extracted from the text modal knowledge graph and / or audio modal knowledge graph in the unimodal knowledge graph to an external knowledge graph for alignment using a preset text processing strategy includes:

[0035] The first encoder and the second encoder are determined according to a preset text processing strategy;

[0036] The nodes extracted from the text modality knowledge graph are linked to an external knowledge graph for alignment based on the first encoder and the second encoder.

[0037] and / or;

[0038] A text transformation knowledge graph is determined based on the audio modality knowledge graph;

[0039] The nodes extracted from the text-to-knowledge graph are linked to an external knowledge graph for alignment using the first encoder and the second encoder.

[0040] Optionally, the step of linking each node extracted from the text modality knowledge graph to an external knowledge graph for alignment based on the first encoder and the second encoder includes:

[0041] The first encoder pre-encodes each node in the external knowledge graph to obtain external text representation information;

[0042] The nodes in the text modality knowledge graph are encoded by the second encoder to obtain internal text node representation information;

[0043] The dot product of each node representation in the internal text node representation information and the external text representation information is calculated to obtain the numerical result.

[0044] The numerical result is compared with the target threshold, and the link result is determined from the numerical result based on the comparison result;

[0045] Based on the linking results, each node extracted from the text modality knowledge graph is linked to an external knowledge graph for alignment.

[0046] Optionally, the step of linking the nodes extracted from the image modal knowledge graph and / or video modal knowledge graph in the unimodal knowledge graph to the external knowledge graph for alignment using a preset image processing strategy includes:

[0047] The face recognition strategy is determined based on the preset image processing strategy;

[0048] The face recognition strategy is used to identify image modal knowledge graphs and / or video modal knowledge graphs to obtain person recognition results;

[0049] Based on the person recognition results, the nodes extracted from the image modal knowledge graph and / or video modal knowledge graph are linked to the external knowledge graph for alignment.

[0050] Optionally, the step of completing the information in the multimodal synthetic graph based on the external knowledge graph includes:

[0051] Obtain the synthetic entity information of the multimodal synthetic map;

[0052] Based on the synthesized entity information, the external knowledge graph is queried to obtain supplementary relationship information and supplementary attribute information;

[0053] The multimodal synthetic map is updated based on the supplementary relationship information and the supplementary attribute information.

[0054] Furthermore, to achieve the above objectives, this application also proposes a knowledge graph construction apparatus, which includes:

[0055] The graph construction module is used to construct multiple unimodal knowledge graphs based on the multimodal information to be organized;

[0056] The graph synthesis module is used to link the nodes extracted from each single-modal knowledge graph to an external knowledge graph for alignment, thereby obtaining a multimodal synthesized graph.

[0057] The information completion module is used to complete the information of the multimodal synthetic graph based on the external knowledge graph.

[0058] Optionally, the graph construction module is further configured to acquire multimodal information to be organized, and obtain multiple unimodal information based on the multimodal information; and construct multiple unimodal knowledge graphs based on the unimodal information.

[0059] Optionally, the graph construction module is further configured to name entities based on the text modality information to obtain an entity list; extract relationships based on the entity list information to obtain an entity relationship list; extract attributes based on the entity relationship list information to obtain an attribute list; and construct a text modality knowledge graph based on the attribute list information.

[0060] Optionally, the graph construction module is further configured to determine image recognition entities based on the image modality information; determine co-occurrence relationship information based on the image recognition entities; and construct an image modality knowledge graph based on the image recognition entities and the co-occurrence relationship information.

[0061] Optionally, the graph construction module is further configured to determine video recognition entities based on the video modal information; perform behavior detection and video event detection on the video recognition entities to obtain video entity relationship information; and construct a video modal knowledge graph based on the video recognition entities and the video entity relationship information.

[0062] Optionally, the graph construction module is further configured to convert the audio modal information into recognized text information; and construct a text modal knowledge graph based on the recognized text information.

[0063] In addition, to achieve the above objectives, this application also proposes a knowledge graph construction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the knowledge graph construction method as described above.

[0064] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the knowledge graph construction method described above.

[0065] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the knowledge graph construction method described above.

[0066] One or more technical solutions proposed in this application have at least the following technical effects:

[0067] This application constructs multiple unimodal knowledge graphs based on multimodal information to be processed; links the nodes extracted from each unimodal knowledge graph to an external knowledge graph for alignment, resulting in a multimodal composite graph; and completes the multimodal composite graph with information from the external knowledge graph. In this way, by introducing an external knowledge graph, nodes extracted from multiple modalities are linked to the same external knowledge graph, and the information existing in the external knowledge graph is used to expand and supplement the constructed graph. This effectively utilizes the complex and diverse data sources in the open-source intelligence field and expands the graph construction using external knowledge. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart illustrating an embodiment of the knowledge graph construction method of this application.

[0071] Figure 2 This is a schematic diagram of knowledge graph node links provided in an embodiment of the knowledge graph construction method of this application;

[0072] Figure 3 This is a flowchart illustrating Embodiment 2 of the knowledge graph construction method of this application.

[0073] Figure 4 This is a schematic diagram of the module structure of the knowledge graph construction device according to an embodiment of this application;

[0074] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the knowledge graph construction method in the embodiments of this application.

[0075] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0076] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0077] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0078] The main solution of this application embodiment is: to construct multiple unimodal knowledge graphs based on the multimodal information to be organized; to link each node extracted from each unimodal knowledge graph to an external knowledge graph for alignment, thereby obtaining a multimodal composite graph; and to complete the information of the multimodal composite graph based on the external knowledge graph.

[0079] In this embodiment, for ease of description, the following description will focus on the knowledge graph construction device as the execution subject.

[0080] In the field of open-source intelligence, existing technologies typically involve complex information sources, including multiple modalities such as text, video, audio, and tables. When constructing knowledge graphs, node extraction techniques are usually used to build nodes, and node relationship extraction techniques are used to build edges between nodes. However, the graphs constructed using only a single modality extraction technique are very limited, and the information from each modality is not fully utilized or verified.

[0081] This application provides a solution that links nodes extracted from multiple modalities of information to the same external knowledge graph by introducing external knowledge graphs, and then uses the information in the external knowledge graphs to expand and supplement the constructed graph. This effectively utilizes the complex and diverse data sources in the open-source intelligence field and expands the graph construction using external knowledge.

[0082] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or server capable of performing the above functions. The following description uses a knowledge graph construction device as an example to illustrate this embodiment and the subsequent embodiments.

[0083] Based on this, embodiments of this application provide a knowledge graph construction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the knowledge graph construction method of this application.

[0084] In this embodiment, the knowledge graph construction method includes steps S10 to S40:

[0085] Step S10: Construct multiple unimodal knowledge graphs based on the multimodal information to be organized;

[0086] It's important to note that in the open-source intelligence field, information sources are typically complex, including multiple modalities such as text, video, audio, and tables. When constructing knowledge graphs, node extraction techniques are usually used to build nodes, and node relationship extraction techniques are used to build edges between nodes. However, graphs constructed using only a single modality extraction technique are very limited, and the information from each modality is not fully utilized or verified. The problems with existing methods are: 1. Graphs constructed directly from information source data may lack important information due to limited data. 2. Information sources in the open-source intelligence field usually include multiple modalities. Directly constructing a graph based on a single modality can easily lead to missing information. If attempts are made to align graphs constructed from multiple modalities, the missing information can cause inaccurate alignment. For example, in a graph constructed from modality 1, node 1 and node 2 in a graph constructed from modality 2 are actually the same node, but due to inaccurate alignment, nodes 1 and 2 are not merged in the aligned graph.

[0087] It should be understood that the multimodal information to be organized refers to the different information and data of multiple modalities prepared in advance for knowledge graph construction, including images, text, video, audio, etc.

[0088] In practice, the first step is to organize the multimodal information, that is, to classify it according to different modalities. Then, the classified individual modal information is used to construct individual knowledge graphs to obtain multiple single-modal knowledge graphs for subsequent merging.

[0089] Step S20: Link each node extracted from each single-modal knowledge graph to an external knowledge graph for alignment to obtain a multimodal composite graph;

[0090] It should be noted that after obtaining the unimodal knowledge graph, node extraction is then performed on it, which involves extracting the link points and knowledge points from the knowledge graph. For example... Figure 2 As shown, the extracted nodes are then linked to an external knowledge graph and aligned to obtain a multimodal synthetic graph.

[0091] In a feasible implementation, in order to accurately synthesize a multimodal synthesis graph, step S20 includes: linking each node extracted from the text modality knowledge graph and / or audio modality knowledge graph in the unimodal knowledge graph to an external knowledge graph for alignment using a preset text processing strategy, and / or linking each node extracted from the image modality knowledge graph and / or video modality knowledge graph in the unimodal knowledge graph to the external knowledge graph for alignment using a preset image processing strategy, thereby obtaining a multimodal synthesis graph.

[0092] It should be understood that before obtaining the final multimodal synthetic graph, each node in each single-modal knowledge graph needs to be processed differently based on different modes before alignment can be performed.

[0093] In specific implementation, since the number and type of single-modal knowledge graphs are not limited, in this embodiment, at least one of the text, image, audio and video modalities exists, and there may be four modalities but one to three are missing. Therefore, for the single-modal graphs corresponding to the existing modalities, only the graphs are operated accordingly.

[0094] It should be noted that in the scheme of this embodiment, the general processing methods of text modality and audio modality are the same. The knowledge graph of text modality is a preset text processing strategy, while the knowledge graph of audio modality first converts audio into text and then processes it according to the preset text processing strategy.

[0095] It should be understood that in this embodiment, the processing methods for image modalities and video modalities are the same, so the same operations are performed for node linking and knowledge graph alignment.

[0096] In one feasible implementation, to link and align nodes in a text modality knowledge graph and an audio modality knowledge graph, the step of linking nodes extracted from the text modality knowledge graph and / or audio modality knowledge graph in the single-modality knowledge graph to an external knowledge graph for alignment using a preset text processing strategy includes: determining a first encoder and a second encoder according to the preset text processing strategy; linking nodes extracted from the text modality knowledge graph to an external knowledge graph for alignment using the first encoder and the second encoder; and / or; determining a text transformation knowledge graph based on the audio modality knowledge graph; and linking nodes extracted from the text transformation knowledge graph to an external knowledge graph for alignment using the first encoder and the second encoder.

[0097] In practice, the first encoder and the second encoder are first determined. The first encoder is usually a pre-trained language model, and the second encoder can be a language model or a graph coding model.

[0098] It should be noted that after determining the first encoder and the second encoder, the nodes of the text modality knowledge graph are aligned based on the first encoder and the second encoder.

[0099] It should be understood that, in order to more conveniently process the audio modality knowledge graph, it is first transformed into a text-to-text knowledge graph, that is, the audio is transformed to obtain the converted text. Thus, it can be processed based on the same logic.

[0100] In a feasible implementation, to specifically process text modalities according to a first encoder and a second encoder, the step of linking each node extracted from the text modal knowledge graph to an external knowledge graph for alignment according to the first encoder and the second encoder includes: pre-encoding each node in the external knowledge graph using the first encoder to obtain external text representation information; encoding the nodes in the text modal knowledge graph using the second encoder to obtain internal text node representation information; performing a dot product calculation between each node representation in the internal text node representation information and the external text representation information to obtain a numerical result; comparing the numerical result with a target threshold, and determining a linking result from the numerical result based on the comparison result; and linking each node extracted from the text modal knowledge graph to the external knowledge graph for alignment according to the linking result.

[0101] In specific implementation, the external knowledge graph is denoted as external knowledge graph kg_foreign, which can be an external encyclopedia, popular science, or knowledge database. This embodiment does not limit this.

[0102] It should be noted that, through entity linking technology, encoder 1 is used to pre-encode all nodes in kg_foreign to obtain the text representation {emb_i} of each node in kg_foreign. Encoder 2 is used to encode the extracted entities to obtain the representation {emb_j} of each node in kg_text. The dot product of the representation of each node in kg_text and {emb_i} is then performed. The entity in kg_foreign with the largest numerical result that is greater than the given threshold thres is taken as the linking result of the current node.

[0103] In one feasible implementation, in order to process the knowledge graph of image and / or video modalities, the step of linking the nodes extracted from the image modal knowledge graph and / or video modal knowledge graph in the single-modal knowledge graph to the external knowledge graph for alignment using a preset image processing strategy includes: determining a face recognition strategy according to the preset image processing strategy; recognizing the image modal knowledge graph and / or video modal knowledge graph using the face recognition strategy to obtain a person recognition result; and linking the nodes extracted from the image modal knowledge graph and / or video modal knowledge graph to the external knowledge graph for alignment based on the person recognition result.

[0104] It should be understood that, through facial recognition technology, hot topics are identified in {Ent_i,img,i=0,1,2,…,I_img} and / or {Ent_i,video,i=0,1,2,…,I_video}, and linked to kg_foreign based on the person's name.

[0105] Step S30: Complete the information of the multimodal synthesis graph based on the external knowledge graph.

[0106] It should be understood that after obtaining the multimodal synthesis graph, an external knowledge graph is then introduced to supplement the relationships between the nodes in the multimodal synthesis graph.

[0107] In one feasible implementation, in order to complete the information, step S30 includes: obtaining the synthetic entity information of the multimodal synthetic graph; querying the external knowledge graph based on the synthetic entity information to obtain supplementary relationship information and supplementary attribute information; and updating the multimodal synthetic graph based on the supplementary relationship information and the supplementary attribute information.

[0108] In practice, after merging the graphs of each modality into a single graph KG based on the linking results, the KG is then augmented with relationships and information using the known knowledge of the external graph kg_foreign. For example, entities ent_1 and ent_2 both exist in the KG, but the relationship between them is not found in the extraction results of all modalities. However, the external graph kg_foreign records a relationship between ent_1 and ent_2, which can be used to augment the KG. Furthermore, if kg_foreign records additional attribute information, that information can also be augmented.

[0109] This embodiment provides a knowledge graph construction method. It constructs multiple unimodal knowledge graphs based on multimodal information to be processed; links nodes extracted from each unimodal knowledge graph to an external knowledge graph for alignment, resulting in a multimodal composite graph; and completes the multimodal composite graph based on the external knowledge graph. In this way, by introducing an external knowledge graph, nodes extracted from multiple modalities are linked to the same external knowledge graph, and the information existing in the external knowledge graph is used to expand and supplement the constructed graph. This effectively utilizes the complex and diverse data sources in the open-source intelligence field and expands the graph construction using external knowledge.

[0110] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3Step S10 includes steps S101 to S102:

[0111] Step S101: Obtain the multimodal information to be processed, and obtain multiple single-modal information based on the multimodal information;

[0112] It should be noted that the first step is to obtain the pre-prepared multimodal information to be processed, which is information that combines multiple modalities. The multimodal information to be processed can be a whole scientific research textbook, paper materials, etc., which may include information from different modalities.

[0113] Step S102: Construct multiple single-modal knowledge graphs based on the single-modal information.

[0114] It should be understood that after splitting multimodal information into different unimodal information, unimodal knowledge graphs are constructed for different individual modalities, thereby obtaining multiple unimodal knowledge graphs with different modalities.

[0115] In one feasible implementation, when the unimodal information is text modal information, the step of constructing multiple unimodal knowledge graphs based on the unimodal information includes: naming entities based on the text modal information to obtain an entity list; extracting relationships based on the entity list information to obtain an entity relationship list; extracting attributes based on the entity relationship list information to obtain an attribute list; and constructing a text modal knowledge graph based on the attribute list information.

[0116] In specific implementation, the entity list {Ent_i,text,i=0,1,2,…,I_text} is obtained through named entity recognition, the entity relation list {Rel_j,text,j=0,1,2,…,J_text} is obtained through relation extraction technology, the attribute list {Attri_k,text,k=0,1,2,…,K_text} is obtained through attribute extraction technology, and the attribute relation list {Rel_Atrri_l,text,l=0,1,2,…,L_text} is obtained through attribute relation extraction, thereby constructing the graph kg_text.

[0117] In a feasible implementation, in order to perform corresponding processing for image modalities, when the single-modal information is image modal information, the step of constructing multiple single-modal knowledge graphs based on the single-modal information includes: determining image recognition entities based on the image modal information; determining co-occurrence relationship information based on the image recognition entities; and constructing an image modal knowledge graph based on the image recognition entities and the co-occurrence relationship information.

[0118] It should be noted that image recognition technology is used to obtain all identifiable entities {Ent_i,img,i=0,1,2,…,I_img} in the image, and then co-occurrence relationships are established for all identified entities to construct the graph kg_img.

[0119] In one feasible implementation, in order to process video modalities, when the single-modal information is video modal information, the step of constructing multiple single-modal knowledge graphs based on the single-modal information includes: determining video recognition entities based on the video modal information; performing behavior detection and video event detection on the video recognition entities to obtain video entity relationship information; and constructing a video modal knowledge graph based on the video recognition entities and the video entity relationship information.

[0120] It should be understood that the graph kg_video is constructed by obtaining all identifiable entities {Ent_i,video,i=0,1,2,…,I_video} in the video through image recognition technology, and obtaining the relationships between entities {Rel_j,video,j=0,1,2,…,J_video} in the video through behavior detection and video event detection technology.

[0121] In a feasible implementation, in order to process the information of audio modalities to construct a corresponding knowledge graph, when the single modal information is audio modal information, the step of constructing multiple single modal knowledge graphs based on the single modal information includes: converting the audio modal information to obtain recognized text information; and constructing a text modal knowledge graph based on the recognized text information.

[0122] In practical implementation, audio is converted into text using speech-to-text recognition technology. Then, the text modality processing method is repeated to obtain {Ent_i,audio,i=0,1,2,…,I_audio}, {Rel_j,audio,j=0,1,2,…,J_audio}, {Attri_k,audio,k=0,1,2,…,K_audio}, and {Rel_Atrri_l,audio,l=0,1,2,…,L_audio}, thus constructing the atlas kg_audio.

[0123] This embodiment acquires multimodal information to be processed and obtains multiple unimodal information based on the multimodal information; multiple unimodal knowledge graphs are then constructed based on the unimodal information. In this way, different processing is applied to information of different modalities, thereby enabling the construction of more complete and accurate multiple unimodal knowledge graphs.

[0124] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the knowledge graph construction method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0125] This application also provides a knowledge graph construction apparatus; please refer to... Figure 4 The knowledge graph construction device includes:

[0126] The graph construction module 10 is used to construct multiple unimodal knowledge graphs based on the multimodal information to be organized.

[0127] The graph synthesis module 20 is used to link the nodes extracted from each single-modal knowledge graph to an external knowledge graph for alignment, thereby obtaining a multimodal synthesized graph.

[0128] The information completion module 30 is used to complete the information of the multimodal synthetic graph based on the external knowledge graph.

[0129] This embodiment constructs multiple unimodal knowledge graphs based on the multimodal information to be processed; links the nodes extracted from each unimodal knowledge graph to an external knowledge graph for alignment, resulting in a multimodal composite graph; and completes the multimodal composite graph based on the external knowledge graph. In this way, by introducing an external knowledge graph, nodes extracted from multiple modalities are linked to the same external knowledge graph, and the information existing in the external knowledge graph is used to expand and supplement the constructed graph. This effectively utilizes the complex and diverse data sources in the open-source intelligence field and expands the graph construction using external knowledge.

[0130] In one embodiment, the graph construction module 10 is further configured to acquire multimodal information to be organized, and obtain multiple unimodal information based on the multimodal information; and construct multiple unimodal knowledge graphs based on the unimodal information.

[0131] In one embodiment, the graph construction module 10 is further configured to: name entities based on the text modality information to obtain entity list information; extract relationships based on the entity list information to obtain entity relationship list information; extract attributes based on the entity relationship list information to obtain attribute list information; and construct a text modality knowledge graph based on the attribute list information.

[0132] In one embodiment, the graph construction module 10 is further configured to determine image recognition entities based on the image modality information; determine co-occurrence relationship information based on the image recognition entities; and construct an image modality knowledge graph based on the image recognition entities and the co-occurrence relationship information.

[0133] In one embodiment, the graph construction module 10 is further configured to determine video recognition entities based on the video modal information; perform behavior detection and video event detection on the video recognition entities to obtain video entity relationship information; and construct a video modal knowledge graph based on the video recognition entities and the video entity relationship information.

[0134] In one embodiment, the graph construction module 10 is further configured to convert the audio modal information into recognized text information; and construct a text modal knowledge graph based on the recognized text information.

[0135] In one embodiment, the graph synthesis module 20 is further configured to link each node extracted from the text modality knowledge graph and / or audio modality knowledge graph in the unimodal knowledge graph to an external knowledge graph for alignment using a preset text processing strategy, and / or link each node extracted from the image modality knowledge graph and / or video modality knowledge graph in the unimodal knowledge graph to the external knowledge graph for alignment using a preset image processing strategy, thereby obtaining a multimodal synthesized graph.

[0136] In one embodiment, the graph synthesis module 20 is further configured to: determine a first encoder and a second encoder according to a preset text processing strategy; link each node extracted from the text modality knowledge graph to an external knowledge graph for alignment according to the first encoder and the second encoder; and / or: determine a text conversion knowledge graph according to the audio modality knowledge graph; link each node extracted from the text conversion knowledge graph to an external knowledge graph for alignment according to the first encoder and the second encoder.

[0137] In one embodiment, the graph synthesis module 20 is further configured to: pre-encode each node in the external knowledge graph using the first encoder to obtain external text representation information; encode the nodes in the text modality knowledge graph using the second encoder to obtain internal text node representation information; perform dot product calculation on each node representation in the internal text node representation information and the external text representation information to obtain a numerical result; compare the numerical result with a target threshold, and determine a linking result from the numerical result based on the comparison result; and link each node extracted from the text modality knowledge graph to the external knowledge graph for alignment based on the linking result.

[0138] In one embodiment, the graph synthesis module 20 is further configured to determine a face recognition strategy based on a preset image processing strategy; identify the image modal knowledge graph and / or video modal knowledge graph using the face recognition strategy to obtain a person recognition result; and link each node extracted from the image modal knowledge graph and / or video modal knowledge graph to the external knowledge graph for alignment based on the person recognition result.

[0139] In one embodiment, the information completion module 30 is further configured to acquire synthetic entity information of the multimodal synthetic graph; query the external knowledge graph based on the synthetic entity information to obtain supplementary relationship information and supplementary attribute information; and update the multimodal synthetic graph based on the supplementary relationship information and the supplementary attribute information.

[0140] The knowledge graph construction apparatus provided in this application, employing the knowledge graph construction method described in the above embodiments, can solve the technical problems of limited graph information constructed through single-modality extraction techniques and insufficient utilization and verification of information from each modality. Compared with the prior art, the beneficial effects of the knowledge graph construction apparatus provided in this application are the same as those of the knowledge graph construction method provided in the above embodiments, and other technical features in the knowledge graph construction apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0141] This application provides a knowledge graph construction device, which includes: at least one processor; and a memory communicatively linked to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the knowledge graph construction method in Embodiment 1 above.

[0142] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the knowledge graph construction device of the embodiments of this application. The knowledge graph construction device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The knowledge graph construction device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0143] like Figure 5As shown, the knowledge graph construction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the knowledge graph construction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also linked to the bus. Typically, the following systems can be linked to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the knowledge graph building device to communicate wirelessly or wiredly with other devices to exchange data. Although a knowledge graph building device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0144] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0145] The knowledge graph construction device provided in this application, employing the knowledge graph construction method described in the above embodiments, can solve the technical problem that the graph information constructed through single-modality extraction techniques is limited, and the information of each modality is not fully utilized and verified. Compared with the prior art, the beneficial effects of the knowledge graph construction device provided in this application are the same as those of the knowledge graph construction method provided in the above embodiments, and other technical features in this knowledge graph construction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0146] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0147] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0148] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the knowledge graph construction method in the above embodiments.

[0149] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical links having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0150] The aforementioned computer-readable storage medium may be included in the knowledge graph construction device; or it may exist independently and not be assembled into the knowledge graph construction device.

[0151] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the knowledge graph construction device, the knowledge graph construction device: constructs multiple unimodal knowledge graphs based on the multimodal information to be organized; links each node extracted from each unimodal knowledge graph to an external knowledge graph for alignment, thereby obtaining a multimodal composite graph; and completes the information of the multimodal composite graph based on the external knowledge graph.

[0152] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be linked to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be linked to an external computer (e.g., via the Internet using an Internet service provider).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0155] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described knowledge graph construction method. This solves the technical problem that the graph information constructed using single-modality extraction techniques is limited, and the information from each modality is not fully utilized and verified. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the knowledge graph construction method provided in the above embodiments, and will not be repeated here.

[0156] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the knowledge graph construction method described above.

[0157] The computer program product provided in this application can solve the technical problem that the knowledge graph information constructed by single-modality extraction technology is limited, and the information of each modality is not fully utilized and verified. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the knowledge graph construction method provided in the above embodiments, and will not be repeated here.

[0158] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0159] This invention also discloses A1. A method for constructing a knowledge graph, the method comprising:

[0160] Multiple unimodal knowledge graphs were constructed based on the multimodal information to be processed;

[0161] Each node extracted from each unimodal knowledge graph is linked to an external knowledge graph for alignment, resulting in a multimodal composite graph.

[0162] Information is supplemented into the multimodal synthetic graph based on the external knowledge graph.

[0163] A2. As described in A1, the step of constructing multiple unimodal knowledge graphs based on the multimodal information to be processed includes:

[0164] Obtain the multimodal information to be processed, and obtain multiple single-modal information based on the multimodal information;

[0165] Multiple unimodal knowledge graphs are constructed based on the unimodal information.

[0166] A3. As described in A2, when the single-modal information is text modal information;

[0167] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0168] Name entities based on the text modality information to obtain an entity list.

[0169] Based on the entity list information, relationship extraction is performed to obtain the entity relationship list information;

[0170] Based on the entity relationship list information, attribute extraction is performed to obtain the attribute list information;

[0171] A text modal knowledge graph is constructed based on the attribute list information.

[0172] A4. As described in A2, when the single-modal information is image modal information;

[0173] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0174] The image entity is determined based on the image modal information;

[0175] Co-occurrence relationship information is determined based on the image recognition entities;

[0176] A knowledge graph of image modalities is constructed based on the image recognition entities and the co-occurrence relationship information.

[0177] A5. As described in A2, when the single-modal information is video modal information;

[0178] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0179] The video recognition entity is determined based on the video modal information;

[0180] Behavior detection and video event detection are performed on the video recognition entities to obtain video entity relationship information;

[0181] A video modal knowledge graph is constructed based on the video-identified entities and the relationship information between the video entities.

[0182] A6. As described in A2, when the single-modal information is audio modal information;

[0183] The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes:

[0184] The audio modal information is converted to obtain the recognized text information;

[0185] A text modality knowledge graph is constructed based on the identified text information.

[0186] A7. As described in A1, the step of linking each node extracted from each unimodal knowledge graph to an external knowledge graph for alignment to obtain a multimodal synthetic graph includes:

[0187] By using a preset text processing strategy, nodes extracted from the text modality knowledge graph and / or audio modality knowledge graph in the unimodal knowledge graph are linked to an external knowledge graph for alignment, and / or by using a preset image processing strategy, nodes extracted from the image modality knowledge graph and / or video modality knowledge graph in the unimodal knowledge graph are linked to the external knowledge graph for alignment, thus obtaining a multimodal composite graph.

[0188] A8. As described in A7, the step of linking the nodes extracted from the text modal knowledge graph and / or audio modal knowledge graph in the unimodal knowledge graph to the external knowledge graph for alignment using a preset text processing strategy includes:

[0189] The first encoder and the second encoder are determined according to a preset text processing strategy;

[0190] The nodes extracted from the text modality knowledge graph are linked to an external knowledge graph for alignment based on the first encoder and the second encoder.

[0191] and / or;

[0192] A text transformation knowledge graph is determined based on the audio modality knowledge graph;

[0193] The nodes extracted from the text-to-knowledge graph are linked to an external knowledge graph for alignment using the first encoder and the second encoder.

[0194] A9. As described in A8, the step of linking each node extracted from the text modality knowledge graph to an external knowledge graph for alignment based on the first encoder and the second encoder includes:

[0195] The first encoder pre-encodes each node in the external knowledge graph to obtain external text representation information;

[0196] The nodes in the text modality knowledge graph are encoded by the second encoder to obtain internal text node representation information;

[0197] The dot product of each node representation in the internal text node representation information and the external text representation information is calculated to obtain the numerical result.

[0198] The numerical result is compared with the target threshold, and the link result is determined from the numerical result based on the comparison result;

[0199] Based on the linking results, each node extracted from the text modality knowledge graph is linked to an external knowledge graph for alignment.

[0200] A10. As described in A7, the step of linking the nodes extracted from the image modal knowledge graph and / or video modal knowledge graph in the unimodal knowledge graph to the external knowledge graph for alignment using a preset image processing strategy includes:

[0201] The face recognition strategy is determined based on the preset image processing strategy;

[0202] The face recognition strategy is used to identify image modal knowledge graphs and / or video modal knowledge graphs to obtain person recognition results;

[0203] Based on the person recognition results, the nodes extracted from the image modal knowledge graph and / or video modal knowledge graph are linked to the external knowledge graph for alignment.

[0204] A11. The method described in any one of A1-A10, wherein the step of completing the information of the multimodal synthetic graph based on the external knowledge graph includes:

[0205] Obtain the synthetic entity information of the multimodal synthetic map;

[0206] Based on the synthesized entity information, the external knowledge graph is queried to obtain supplementary relationship information and supplementary attribute information;

[0207] The multimodal synthetic map is updated based on the supplementary relationship information and the supplementary attribute information.

[0208] The present invention also discloses B12. A knowledge graph construction apparatus, the apparatus comprising:

[0209] The graph construction module is used to construct multiple unimodal knowledge graphs based on the multimodal information to be organized;

[0210] The graph synthesis module is used to link the nodes extracted from each single-modal knowledge graph to an external knowledge graph for alignment, thereby obtaining a multimodal synthesized graph.

[0211] The information completion module is used to complete the information of the multimodal synthetic graph based on the external knowledge graph.

[0212] B13. The apparatus as described in B12, wherein the graph construction module is further configured to acquire multimodal information to be organized, and obtain multiple unimodal information based on the multimodal information; and construct multiple unimodal knowledge graphs based on the unimodal information.

[0213] B14. The apparatus as described in B12, wherein the graph construction module is further configured to: name entities based on the text modality information to obtain entity list information; extract relationships based on the entity list information to obtain entity relationship list information; extract attributes based on the entity relationship list information to obtain attribute list information; and construct a text modality knowledge graph based on the attribute list information.

[0214] B15. The apparatus as described in B12, wherein the graph construction module is further configured to: determine image recognition entities based on the image modality information; determine co-occurrence relationship information based on the image recognition entities; and construct an image modality knowledge graph based on the image recognition entities and the co-occurrence relationship information.

[0215] B16. The apparatus as described in B12, wherein the graph construction module is further configured to: determine video recognition entities based on the video modal information; perform behavior detection and video event detection on the video recognition entities to obtain video entity relationship information; and construct a video modal knowledge graph based on the video recognition entities and the video entity relationship information.

[0216] B17. The apparatus as described in B12, wherein the graph construction module is further configured to convert the audio modal information into recognized text information; and to construct a text modal knowledge graph based on the recognized text information.

[0217] The present invention also discloses C18. A knowledge graph construction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the knowledge graph construction method as described above.

[0218] The present invention also discloses D19. A storage medium, which is a computer-readable storage medium, wherein a computer program is stored on the storage medium, and the computer program, when executed by a processor, implements the steps of the knowledge graph construction method described above.

[0219] The present invention also discloses E20. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the knowledge graph construction method described above.

Claims

1. A method for constructing a knowledge graph, characterized in that, The method includes: Multiple unimodal knowledge graphs were constructed based on the multimodal information to be processed; Each node extracted from each unimodal knowledge graph is linked to an external knowledge graph for alignment, resulting in a multimodal composite graph. Information is supplemented into the multimodal synthetic graph based on the external knowledge graph.

2. The method as described in claim 1, characterized in that, The steps for constructing multiple unimodal knowledge graphs based on the multimodal information to be processed include: Obtain the multimodal information to be processed, and obtain multiple single-modal information based on the multimodal information; Multiple unimodal knowledge graphs are constructed based on the unimodal information.

3. The method as described in claim 2, characterized in that, When the single-modal information is text modal information; The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes: Name entities based on the text modality information to obtain an entity list. Based on the entity list information, relationship extraction is performed to obtain the entity relationship list information; Based on the entity relationship list information, attribute extraction is performed to obtain the attribute list information; A text modal knowledge graph is constructed based on the attribute list information.

4. The method as described in claim 2, characterized in that, When the single-modal information is image modal information; The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes: The image entity is determined based on the image modal information; Co-occurrence relationship information is determined based on the image recognition entities; A knowledge graph of image modalities is constructed based on the image recognition entities and the co-occurrence relationship information.

5. The method as described in claim 2, characterized in that, When the single-modal information is video modal information; The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes: The video recognition entity is determined based on the video modal information; Behavior detection and video event detection are performed on the video recognition entities to obtain video entity relationship information; A video modal knowledge graph is constructed based on the video-identified entities and the relationship information between the video entities.

6. The method as described in claim 2, characterized in that, When the single-modal information is audio modal information; The step of constructing multiple single-modal knowledge graphs based on the single-modal information includes: The audio modal information is converted to obtain the recognized text information; A text modality knowledge graph is constructed based on the identified text information.

7. A knowledge graph construction device, characterized in that, The device includes: The graph construction module is used to construct multiple unimodal knowledge graphs based on the multimodal information to be organized; The graph synthesis module is used to link the nodes extracted from each single-modal knowledge graph to an external knowledge graph for alignment, thereby obtaining a multimodal synthesized graph. The information completion module is used to complete the information of the multimodal synthetic graph based on the external knowledge graph.

8. A knowledge graph construction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the knowledge graph construction method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the knowledge graph construction method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the knowledge graph construction method as described in any one of claims 1 to 6.

Images