Knowledge graph construction method and device, equipment and storage medium

Abstract

Embodiments of the present application disclose a knowledge graph construction method and device, equipment and a storage medium, wherein the method comprises: performing event detection on text data to obtain trigger words corresponding to at least one first instance event of the text data; performing image classification processing on image data to obtain event types corresponding to at least one second instance event of the image data; determining a first concept event matched with each trigger word based on a multi-modal ontology; determining a second concept event matched with each event type based on the multi-modal ontology; and constructing a multi-modal event knowledge graph based on the first concept event matched with each trigger word and the second concept event matched with each event type.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Singapore Patent Application No. 10202205360Q, filed on May 20, 2022, entitled “Method, Apparatus, Device and Storage Medium for Constructing Knowledge Graphs”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to, but is not limited to, the field of natural language processing technology, and in particular to a method, apparatus, device, and storage medium for constructing a knowledge graph. Background Technology

[0004] An event is a basic building block describing the activities and states of the objective world. Knowledge graphs with events as nodes summarize the events inherent in various fields of real life and the interactions between these events, which can be of great importance in a series of downstream tasks such as common sense reasoning.

[0005] In recent years, a large number of works have emerged related to the construction of event knowledge graphs. For example, small-scale, high-quality event knowledge graphs have been constructed through manual annotation; while large-scale, relatively low-quality event knowledge graphs have been constructed through automated rules. However, constructing high-quality, large-scale event knowledge graphs without overly relying on manual annotation remains an unresolved issue in previous work. Furthermore, there is a lack of rigorous design for an ontology of event knowledge graphs and a failure to implement the complementarity and fusion of multimodal information (multiple modalities, such as images and text, each representing a modality). Summary of the Invention

[0006] In view of this, embodiments of this application provide at least one method, apparatus, device, and storage medium for constructing a knowledge graph.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a knowledge graph construction method, the method comprising: performing event detection on text data to obtain trigger words for at least one first instance event corresponding to the text data; performing image classification processing on image data to obtain event types for at least one second instance event corresponding to the image data; determining a first concept event matching each trigger word based on a multimodal ontology; determining a second concept event matching each event type based on the multimodal ontology; and constructing a multimodal event knowledge graph based on the first concept event matching each trigger word and the second concept event matching each event type.

[0009] Secondly, embodiments of this application provide a method for constructing a multimodal ontology, comprising: matching word meanings in a lexicon with each event in a first frame to establish a first mapping relationship between the lexicon and the first frame, wherein the first frame is used to determine conceptual events of text data; matching word meanings in the lexicon with each event in a second frame to establish a second mapping relationship between the lexicon and the second frame, wherein the second frame is used to determine conceptual events of image data; and fusing the first mapping relationship and the second mapping relationship with the lexicon, the first frame, and the second frame to obtain the multimodal ontology.

[0010] Thirdly, embodiments of this application provide a knowledge graph construction apparatus, the apparatus comprising: a detection module for performing event detection on text data to obtain trigger words for at least one first instance event corresponding to the text data; a classification module for performing image classification processing on image data to obtain event types for at least one second instance event corresponding to the image data; a first determination module for determining a first concept event matching each trigger word based on a multimodal ontology; a second determination module for determining a second concept event matching each event type based on the multimodal ontology; and a construction module for constructing a multimodal event knowledge graph based on the first concept event matching each trigger word and the second concept event matching each event type.

[0011] Fourthly, embodiments of this application provide a multimodal ontology construction apparatus, the apparatus comprising: a first establishment module, configured to match word meanings in a lexicon with each event in a first frame to establish a first mapping relationship between the lexicon and the first frame, wherein the first frame is used to determine conceptual events of text data; a second establishment module, configured to match word meanings in the lexicon with each event in a second frame to establish a second mapping relationship between the lexicon and the second frame, wherein the second frame is used to determine conceptual events of image data; and a fusion module, configured to fuse the lexicon, the first frame, and the second frame based on the first mapping relationship and the second mapping relationship to obtain the multimodal ontology.

[0012] Fifthly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0013] Sixthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0014] In a seventh aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps for implementing the above-described method.

[0015] Eighthly, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements some or all of the steps in the above method.

[0016] In this embodiment, a first concept event of text data and a second concept event of image data are determined through a multimodal ontology. Then, based on the first concept event and the second concept event, a multimodal event knowledge graph is constructed, thereby realizing the complementarity and fusion between multimodal information, providing more information for the construction of the knowledge graph, and providing more possibilities for knowledge mining.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 An example of text and image events and a schematic diagram illustrating the relationship between them are provided for embodiments of this application;

[0020] Figure 2 A schematic diagram of three different levels in MMEKG from left to right, provided for embodiments of this application;

[0021] Figure 3 A schematic diagram of the ontology designed in MMEKG for embodiments of this application;

[0022] Figure 4A A schematic diagram of a knowledge extraction system provided in an embodiment of this application;

[0023] Figure 4B A schematic diagram illustrating a pairing example for constructing conceptual event induction and relation induction as provided in an embodiment of this application;

[0024] Figure 5A A schematic diagram illustrating the implementation process of a knowledge graph construction method provided in this application embodiment;

[0025] Figure 5BA schematic diagram illustrating the implementation process of step S104 in a knowledge graph construction method provided in this application embodiment;

[0026] Figure 5C A schematic diagram illustrating the implementation process of step S105 in a knowledge graph construction method provided in an embodiment of this application;

[0027] Figure 5D This is a schematic diagram illustrating the implementation process of adding event relationships to a multimodal event knowledge graph, as provided in an embodiment of this application.

[0028] Figure 5E for Figure 5D A schematic diagram of the implementation process of step S107a;

[0029] Figure 5F A schematic diagram illustrating the implementation flow of a method for updating an ontology provided in an embodiment of this application;

[0030] Figure 5G A schematic diagram illustrating the implementation flow of a method for constructing a multimodal ontology provided in this application embodiment;

[0031] Figure 5H A schematic diagram illustrating the implementation flow of a multimodal ontology update method provided in this application embodiment;

[0032] Figure 6A A schematic diagram of the composition structure of a knowledge graph construction device provided in an embodiment of this application;

[0033] Figure 6B A schematic diagram of the composition structure of a multimodal ontology construction device provided in an embodiment of this application;

[0034] Figure 7 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0038] Furthermore, the term "A matching each B" as used below means that each of the at least one B matches a corresponding A among the at least one A. For example, "first concept event matching each trigger word" means that each of the at least one trigger word matches a corresponding first concept event among the at least one first concept event.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0040] This application provides a large-scale, multimodal event knowledge graph (MMEKG). The MMEKG unifies different knowledge patterns through events, complementing each other and eliminating ambiguity. Specifically, the MMEKG includes: (i) over 990,000 conceptual events and 644 relationship types (hierarchical and semantic relationships between conceptual events) to cover most types of events; and (ii) over 863 million instance events connected by 934 million relationships (relationships between instance events, which can be further subdivided into finer-grained subtypes such as temporal, causal, co-occurrence, and other semantic relationships), providing rich contextual information in text and / or images.

[0041] This application also provides an efficient text / visual knowledge extraction system pipeline for collecting billions of instance events and their relationships. This application further provides an inductive strategy for creating millions of conceptual events and ontology, where the ontology is used to organize all events and relationships between events in the MMEKG. This application also provides a pipeline capable of seamlessly parsing text / images into event graphs and retrieving multimodal knowledge at the concept and instance levels.

[0042] Recently, many knowledge graphs (KGs) have been organized and successfully applied to various scenarios, from information extraction to information retrieval. KGs typically store billions of world facts in a directed graph, where nodes represent entities and edges represent relationships between them. While simple and effective, this entity-centric approach to KGs has limited expressive power. How to represent more complex knowledge, such as events, situations, or different modalities, has become a key issue for the wider application of knowledge graphs.

[0043] Based on this, embodiments of this application propose a large-scale multimodal event knowledge graph, which can connect, supplement, and eliminate different forms of knowledge to better understand or reason. Similar to events in the real world, the MMEKG uses events as its basic building blocks, with each event defined by a concept, several arguments, and corresponding roles. Various types of relationships exist between events, such as causal relationships, temporal relationships, or sub-event relationships; entities are the arguments in the KG.

[0044] Figure 1 Examples of text and image events and their relationship are shown: a visual sleep event with arguments cat(sleeper) and sofa(place), and a textual dressing event with arguments cat(wearer) and scarf(clothing), where CO-REF stands for co-reference, and the argument roles are enclosed in parentheses. These two events not only connect text and image with complementary arguments but also provide an underlying common sense: covering oneself with a scarf while sleeping.

[0045] Compared to event-based knowledge graphs (KGs) in related technologies, MMEKG has advanced this field in three aspects: (1) A large ontology contains 990,000 concept events and 644 relation types, covering most types of events in the real world. (2) Natural fusion of multimodal knowledge. This is the first time that knowledge graphs have achieved the fusion of different modal data through fine-grained alignment of events and arguments. (3) Integration of concept events and instance events, which not only extends the ontology from instance events but also obtains concept-level common sense knowledge through comprehensive reasoning provided by context.

[0046] The construction of MMEKG mainly includes two steps: (1) In order to construct patterns and obtain conceptual events, firstly, FrameNet and WordNet are manually combined to initialize a high-quality event ontology; then, the ontology is automatically expanded through inductive reasoning of instance events. This application extends the Simple Event Model (SEM) to define the ontology provided by this application in the Resource Description Framework (RDF). (2) In order to extract instance events from text or images, this application develops a knowledge extraction system to support fast and large-scale extraction in real-world scenarios. The system includes event extraction and event relationship extraction in two modalities, as well as alignment between the two modalities. In addition, the system can parse the input text / image into an event graph and seamlessly retrieve multimodal knowledge from MMEKG.

[0047] The knowledge graph provided in the embodiments of this application is described in detail below.

[0048] Part 1: Introduction to MMEKG: as follows Figure 2 The MMEKG is shown in three different levels from left to right. The left side shows the extracted multimodal content, the middle shows the instance events integrated from the original content, and the right side shows the conceptual events for reasoning. The MMEKG proposed in this application differs from the traditional event-centric KG. The MMEKG includes four types of nodes and four types of relationships. Nodes include conceptual events, instance events, entity arguments, and non-entity arguments (e.g., text). The four types of relationships include: (1) Relationships between instance events. This type of relationship can be further divided into more granular subtypes, such as time, causality, co-occurrence, and other semantic relationships. (2) Relationships between conceptual events, called subclassOf, which represents hierarchical relationships. (3) Relationships between conceptual events and instance events, called instanceOf, which is used to integrate conceptual events and instance events. (4) Role relationships, which reflect the roles of associated event arguments (entity or non-entity). Different conceptual events have different roles.

[0049] Definition: MMEKG = {(h,r,t)h,t∈ε,r∈R}; ε = εcpt∪εins∪εent∪εnent, R = Rins-ins∪Rcpt-cpt∪Rcpt-ins∪Rrole}, where,

[0050] εcpt represents conceptual events, εins represents instance events, εent represents entities, and εnent represents non-entities. Rins-ins represents relationships between instance events, Rcpt-cpt represents relationships between conceptual events, Rcpt-ins represents relationships between conceptual events and instance events, and Rrole represents role relationships. w(h, r, t) represents the weight of the triple (h, r, t) relation, expressed as confidence, where h, r, and t are the head, relation, and tail of the triple, respectively.

[0051] Part 2: User Interface and System Architecture: Based on MMEKG and a text / visual knowledge extraction system, this application embodiment developed a prototype system that can parse text or images into an event graph, where nodes represent instance events and edges represent the relationships between them. Each instance event can refer to specific information: concept events, synonyms, arguments corresponding to roles, and connected neighbors in MMEKG.

[0052] For each instance event, this embodiment of the application identifies the trigger word and its synonyms, links them to the concept event in MMEKG, and then determines the corresponding role based on the concept event, extracting arguments from the text / image.

[0053] The text / visual knowledge extraction system comprises three main components: text knowledge extraction, visual knowledge extraction, and event relationship extraction. Event relationship extraction is used to extract various relationships between events, including the fusion of textual and visual events. It should be noted that conceptual events, synonyms, and relationship types are defined by the cross-modal event ontology provided in the embodiments of this application.

[0054] Part 3: Introduction to Multimodal Ontologies: Ontologies are crucial, not only limiting the types of knowledge that can be focused on, but also providing reasoning capabilities, i.e., the induction from instance events to conceptual events brings new knowledge. The establishment of an ontology includes: (1) modeling ontology data, (2) creating an initial ontology by combining external resources, and (3) continuously expanding the ontology for induction.

[0055] The three steps described above are explained below:

[0056] (1) Ontology modeling

[0057] The embodiments of this application inherit and extend the basic SEM as the basis for knowledge representation. Figure 3This is a schematic diagram of an ontology designed in MMEKG. Dashed boxes represent characters, solid boxes represent events, entities, and relationships, grid-filled boxes represent event-related relationships, gray-filled boxes represent relationship-related relationships, dot-filled boxes represent text / image information extracted from instance events, and unfilled circles represent virtual nodes connecting instance events to their source information.

[0058] 1. Extending single-event architecture from SEM and FrameNet, meaning the single-event RDF structure is extended from the definitions of SEM and FrameNet. For example... Figure 3 As shown, each role has an associated ekg:[role] linking instance events and arguments, rdf: refers to the class defined using the rdf ontology, sem: refers to the class defined using the SEM ontology, and ekg: refers to the new class defined in this application embodiment.

[0059] 2. Add virtual nodes, which are connected to instance events via `ekg:contextOf` to indicate the source of the event. Edges of the virtual node, such as `ekg:trigger`, `ekg:modity`, and `ekg:content`, represent the trigger word, modality, and sentence / image index of the instance event, respectively. Virtual nodes are designed for clear structural integration and are only queried when needed, reducing unnecessary queries.

[0060] 3. Establish relationships between events, mainly including:

[0061] The first type: rdf:instanceOf: the relationship between instance events and concept events;

[0062] The second type: rdf:subclassOf: Relationship between conceptual events, used to represent the hierarchical relationship between conceptual events;

[0063] The third type: various other relationships between instance events, such as temporal or causal relationships. Based on this relationship, link nodes are added between instance events to represent the relationships between them. For example... Figure 3As shown, the marker [EventRel_id] is the link node, used to represent the relationship between ins-Event_317 and other events (not shown in the diagram). Square brackets indicate that the content within the brackets is replaceable. Thus, when multiple event relationships exist between two instance events, different link nodes can represent different event relationships, enabling quick querying of different event relationships (only the link node corresponding to the required event relationship needs to be queried) and modification of relationships between instance events (no need to modify the description of the edges between instance events, only the information of the link nodes needs to be modified), giving the ontology good extensibility. For example, event relationships have many major categories, such as time relationships, which include subcategories such as before and after. One [Reltype_id] represents one relationship, and one subcategory corresponds to one [Reltype_id]. If it is necessary to query event information with a time relationship of "before," then the event node with a time relationship of "after" can be directly deleted using [Reltype_id], achieving a quick query.

[0064] 4. Integrate and supplement information statistics. For example, add frequency and significance scores to the reasoning used for uncertain events. For instance, if the events are "frying eggs in the morning" and "frying eggs with a spatula", where "frying eggs" has a higher frequency and a higher significance score, then the final inferred concept event is "frying eggs". See the ontology induction section below for details.

[0065] (2) Create the initial ontology by combining external resources

[0066] This application embodiment initializes the ontology by merging the WordNet, FrameNet, and imSitu ontology.

[0067] 1. Map the synonym set of each verb and adjective in WordNet to a frame in FrameNet (e.g., roast.v.01 → Apply_heat). A frame represents a high-level conceptual event; one frame represents one conceptual event. Because events in FrameNet are broad (e.g., a chemical substance changing into another called Cause_change is also called Cause_change), and WordNet has richer semantics, aligning the synonym sets allows for more granular conceptual events. Furthermore, WordNet brings hierarchical information to FrameNet. By aligning WordNet with FrameNet, knowing the meaning of the trigger word in a text or image allows us to identify the event being described. For example, "Fire" representing firing corresponds to an attack event in FrameNet, and "Fire" representing dismissal corresponds to a job loss event in FrameNet.

[0068] For mapping, this application embodiment first considers the results of structural mapping. That is, in previous work, the mapping relationship between some word meanings in WordNet and frames in FrameNet has been given, and this application embodiment directly adopts this mapping result. For word meanings in WordNet for which no mapping relationship is given, the mapping relationship is determined by calculating the cosine similarity score between the word meaning in WordNet and the event type of the frame, and the matching relationship with the highest cosine similarity is selected for mapping.

[0069] In this application embodiment, 100 pairs of synonyms (synset) and frames were randomly selected to check whether the definitions of the mapped synset and frame were aligned, and it was found that 89% were reasonable.

[0070] 2. During implementation, the word meanings in WordNet can be manually aligned with the events in Imitu. Since events in Imitu are already labeled with event types, WordNet and Imitu can be aligned based on the word meanings in WordNet and the event types in Imitu, thus achieving the association between WordNet and Imitu. Because Imitu is used to describe image events, and image events are triggered by verbs, and the verbs corresponding to the event types in Imitu do not have word meanings, they need to be aligned with the word meanings in WordNet. Furthermore, since WordNet and FrameNet are already aligned, the event types in FrameNet can be determined through WordNet, thereby obtaining the concept events corresponding to the images.

[0071] (3) Continuously expand ontology summarization

[0072] 1. Summarizing conceptual events:

[0073] Step 1: Assume an initial concept event o and a specific role r for concept event o. Select all arguments under that specific role r, which are connected to the instance event O through r. in Connection, where instance event O in It is classified as a conceptual event.

[0074] Step 2: Perform lemma reduction and heuristic clustering on the central words of all arguments. Lemma reduction involves removing affixes from words and extracting the core components, which are usually words from a dictionary. For example, if the initial concept event is "fry" and the specific role is "food," find all instances of "fry" and count the arguments for the "food" role: fish, a fish, the fish, a big fish, chicken, the chicken. Here, "fish," "a fish," "the fish," and "a big fish" all become "fish" after lemma reduction, and "chicken" and "the chicken" both become "chicken." Heuristic clustering yields two groups: "fish" and "chicken."

[0075] Step 3: Calculate the significance score for each type, considering the following two aspects: (1) Confidence score for each type (number of occurrences) (2) Importance of each type. For example, if the type is "me", its importance is not high, and the weight of this type can be reduced.

[0076] Step 4: Select the K types with the highest salience scores and create new conceptual events by combining the role r and these names with trigger words. For example, since "fish" appears frequently and has a high salience score, when K is 1, combining "fish" with the trigger word "fry" and the role "food" derives the new conceptual event "fry fish".

[0077] like Figure 2 As shown, embodiments of this application derive new conceptual events, such as "complete.v.01__Activity:tour" and "complete.v.01__Activity:Time:fall.". These fine-grained conceptual events are derived by summarizing instance events through the instanceOf relationship and the relationship between complete.v.01 and its subclasses.

[0078] 2. Summarizing the relationships between concepts and events:

[0079] The purpose of relation induction is to discover common-sense relationships between conceptual events based on the relationships between instance events. In practice, this is achieved by calculating the relationships between each pair of instance events (o) on r. h o t The significance score sr(o) h o tTo determine whether to classify a relationship between instance events, the following two aspects are considered in the calculation of the significance score: (1) the confidence score of the relationship r between instance events, and (2) the number of tail events (as few as possible). If the significance score exceeds the threshold of MMEKG, the instance events (o h o t The relationship is added to the relationship of the concept event. For example, in Figure 2 In this embodiment, since the significance scores of the triples (talk.v.01, cooccur, sit.v.01) exceed the threshold, this application extends this relationship from the instance level to the concept level. h with o t The subscripts represent the head event and tail event in the triplet corresponding to each pair of instance events, respectively.

[0080] In this embodiment of the application, a more hierarchical and finer-grained classification structure is achieved by summarizing conceptual events and the relationships between conceptual events.

[0081] Part Four: Knowledge Extraction System

[0082] The knowledge extraction system collects large-scale instance events and relationships for MMEKG. Embodiments of this application follow the overall framework of previous knowledge extraction systems but extend and optimize the event-related components, enabling efficient extraction of high-quality events at the billion-scale level. Through more advanced models, tuning strategies, and component architecture, the system achieves, if not better, at the same performance on each component using common benchmarks. In some embodiments, all cross-encoders in the system can also be replaced with dual encoders, and a joint model trained on multiple tasks can be executed during event-relation extraction to improve efficiency.

[0083] Figure 4A The diagram illustrates a knowledge extraction system provided in an embodiment of this application, such as... Figure 4A As shown, the knowledge extraction system mainly consists of four components: cross-modal event ontology 43, text knowledge extraction component 41, visual knowledge extraction component 42, and event relationship extraction component 44. Please refer to the above for an introduction to the cross-modal event ontology; the other three parts are described below.

[0084] 1. Text Knowledge Extraction Component

[0085] The text knowledge extraction component extracts nodes from event graphs in unstructured text through event detection and argument extraction, including the following steps:

[0086] Step 1: Preprocess the corpus. Next-Sentence Prediction (NSP) models based on pre-trained Language Representation from Transformers (BERT) and heuristic rules are used to identify document boundaries (5-10 sentences per document). Then, part-of-speech (POS) tags and dependency trees are obtained using entity naming tools (e.g., Stanza). The dependency trees are derived through dependency parsing to determine candidate trigger words for the text. Verbs and adjectives are considered as candidate trigger words.

[0087] The second step involves transforming event detection into an unsupervised WordSense Disambiguation (WSD) task based on the thesaurus in the ontology, thus avoiding expensive training data. A bilateral decoder model is applied to predict the most likely set of synonyms for candidate trigger words. Since each synset refers to a concept event, the text is then connected to the MMEKG.

[0088] Step 3: This application proposes an effective event argument extraction method, denoted as PAIE. Its basic idea is to extend a question-and-answer (QA) model to simultaneously predict all roles in a target event. This application proposes a template pre-trained language model (PLM) for the extraction task, designing a role interaction prompt template for each concept event. All role embedded text vectors serve as query vectors to determine the argument span as the answer, thus obtaining the arguments. The model is trained on annotations provided by FrameNet.

[0089] 2. Visual Knowledge Extraction Component

[0090] For the visual knowledge extraction component, a two-stage extraction network is employed. Both models are trained using the largest visual context recognition dataset.

[0091] (1) Use a pre-trained visual model (ViT, Vision Transformer) to perform action recognition to obtain image patch features and perform image classification. Based on the image classification results, determine the concept events.

[0092] (2) Use a pre-trained residual network such as ResNet 50 as the backbone network of Faster R-CNN (Faster Region-CNN) and use a Conditional Long Short-Term Memory (LSTM) decoder to aggregate role information and extract arguments from the image.

[0093] The ViT model is a modification of the traditional Transformer architecture. The traditional Transformer architecture includes an encoder-decoder framework, while ViT only uses the encoder. The standard Transformer takes one-dimensional sequential data as input, so the image needs to be converted into sequential data. ViT's approach is to divide an image into non-overlapping fixed-size patches, then stretch each patch to convert it into a one-dimensional vector, and finally use a linear transformation layer to convert the input patches into a fixed-length vector called patch_embedding.

[0094] 3. Event Relation Extraction Component

[0095] The event relationship extraction component aims to extract the temporal, causal, co-occurrence, and semantic relationships between instance events. Co-occurrence includes text / image alignment.

[0096] (1) Regarding temporal and causal relationships, this application proposes a novel method: constructing a document-level graph to globally infer relationships between events. The method of this application can perform cross-sentence reasoning even without explicit temporal / causal indicators and complex heuristics. This enables this application to simultaneously and effectively identify all temporal and causal relationships within a document. This application jointly predicts temporal and causal relationships as a multi-label, multi-task classification and trains a model based on a causal temporal database, including six relationship types: before, after, during, including, included, and causal.

[0097] (2) Text co-occurrence: If there is a conjunction relationship between triggering words, the text is considered to co-occur. The embodiments of this application identify the conjunction relationship through dependency analysis.

[0098] Cross-modal co-occurrence: Embodiments of this application extract events from pairs of image text and assume their co-occurrence, thereby enabling the mining of information between pairs of image text. Embodiments of this application also observe semantic transformation between different modalities. For example... Figure 1 As shown, the text dressing event may be a sub-event of the visual sleep event.

[0099] (3) Semantic relations:

[0100] In this embodiment, when the argument of event A is the gerund phrase B, B can also be considered a sub-event of A, where event A is triggered by the gerund as its semantic component. For example, two events are extracted from the sentence "eating too much fried chicken leads to being overweight": leading to being overweight (event A) and eating too much chicken (event B). Since A is also an argument of the role `influencing_entity` of B, the events "eating too much chicken" and "leading to being overweight" are connected through the relation `influencing_entity`. Based on this assumption, arguments from FrameNet are used to expand the relation types.

[0101] Event pairs that satisfy the following two conditions are defined as semantic relations: (1) In dependency analysis, trigger words are connected by csubj (subject clause), xcomp (open complement clause), ccomp (complement clause), or advcl (adverbial clause); (2) The trigger word of one event is an argument of another event.

[0102] Part Five: Evaluation

[0103] (1)MMEKG

[0104] Table 1 shows the statistics for MMEKG and other event KGs. This application embodiment constructs a full version (MMEKG full) and a core version (MMEKG core), as shown in Table 1. MMEKG core filters out infrequent events (<3 times) to obtain a denser and more accurate version. MMEKG not only involves a larger ontology but also more instance events.

[0105] Table 1

[0106]

[0107] Among them, ConceptNet, ATOMIC, and ASER are all event commonsense libraries in the knowledge graph field used for comparison, ASER full is the complete version of ASER, and ASER core is the core version of ASER.

[0108] (2) Performance of text / visual extraction system

[0109] Table 2 shows the results of the components trained on public datasets in the embodiments of this application, since there is no unified benchmark to evaluate the entire extraction process. It can be seen that, except for WSD, all knowledge extraction components in the embodiments of this application achieve better performance. The WSD model provided in the embodiments of this application is highly efficient in large-scale event detection.

[0110] Table 2

[0111]

[0112] (3) Instance-level evaluation

[0113] Considering the different data distributions between the training data and the extracted corpus, the instance-level quality of the MMEKG can be manually evaluated during implementation. 1000 instance events were randomly selected from the text, and 500 instance events were randomly selected from the images. In addition to the original context, multiple personnel were invited to label whether the extracted events represented the semantics of the original source. For example, instance event relationships, the embodiments considered: (1) causal / temporal relationships from the text, and (2) cross-modal co-occurrence from image caption pairs. 200 text relationships and 300 cross-modal relationships were sampled. These extracted relationships, in addition to the context, were also provided to the same multiple personnel, who were asked whether the extracted relationships matched the original resources. The results in Table 3 show that, considering the complexity of the entire pipeline, there was almost no performance degradation, and the quality of the MMEKG proposed in the embodiments of this application is acceptable.

[0114] Table 3 Instance-level Evaluation

[0115] Modality PrecisionPrecision Event Text Texual 84.0％ Event Visual 64.6％ Triple Text Texual 66.9％ Triple Cross-modal 63.8％

[0116] (4) Ontology-level assessment

[0117] Large-scale ontology is crucial for knowledge-based inductive reasoning. Embodiments of this application further evaluate the quality of conceptual event induction and relation induction. Unlike instance-level evaluation, no context is provided for reference in ontology evaluation. For ease of comparison, embodiments of this application construct pairings using a positive sample and a negative sample. Figure 4B Examples of pairings constructed for conceptual event induction (top) and relation induction (bottom) are shown, where each pair includes one positive sample and one negative sample. The positive sample is drawn from the inductive conceptual events or relations. Negative samples are generated by replacing arguments (conceptual event induction) or tail events (relation induction) in the positive samples. The same person is asked which sample is more common sense. The results, as shown in Tables 4 and 5, show that both negative samples have a negative impact of around 20%. In particular, for relation induction, it is difficult to distinguish which of some similar pairs is better. This embodiment of the application attributes this to low recall and random negative sampling, which can lead to false negatives, and also provides insights for future improvements.

[0118] Figure 4. Conceptual Event Summary

[0119] Sample Positive Samples Negative samples 1000 80.1％ 19.9％

[0120] Table 5. Relation Induction

[0121]

[0122] Existing event knowledge graphs often face a dilemma between quality and quantity. This application not only develops a larger KG through a larger corpus and an advanced extraction system, but also derives a complex ontology and integrates cross-modal information to control the quality of the KG.

[0123] Previous multimodal knowledge extraction systems jointly extracted information from relatively small resources within a narrow domain. The knowledge extraction system provided in this application inherits their overall framework but is designed to extract events on a scale of billions and globally. Therefore, this application specifically optimizes event-related components to improve efficiency and effectiveness.

[0124] Cross-media event parameter alignment: Some previous works have also connected text and images by performing fine-grained alignment of event parameters for various tasks, such as multimodal event extraction and fake news detection. In contrast, embodiments of this application integrate knowledge from different modalities to construct such a large-scale KG.

[0125] This application proposes the first multimodal event knowledge group (MMEKG) with a large-scale event ontology, which not only communicates and complements different knowledge patterns through more expressive events, but also facilitates comprehensive reasoning with rich cross-modal contexts. Furthermore, this application also provides a demonstration system that can seamlessly parse and link any text / image through a knowledge extraction system.

[0126] This application provides a knowledge graph construction method, which can be executed by a processor of a computer device. The computer device refers to a device with data processing capabilities, such as a server, laptop, tablet, desktop computer, smart TV, set-top box, or mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device). The method includes:

[0127] Figure 5A This is a schematic diagram illustrating the implementation process of a knowledge graph construction method provided in an embodiment of this application, such as... Figure 5A As shown, the method includes the following steps S101 to S105:

[0128] Step S101: Perform event detection on the text data to obtain the trigger word of at least one first instance event corresponding to the text data.

[0129] Here, textual data refers to data in text form; instance events refer to events that specifically describe actual examples. The components of an event can include trigger words, event type, arguments, and roles. Trigger words are the core words that indicate the occurrence of the event, often verbs and adjectives, such as attack or discussion. Event type refers to the category of the event, such as a sports event or an attack event. Arguments refer to the participants in the event, such as Zhang San (a specific person) or the school. Roles refer to the roles that arguments play in the event, such as attacker, location, or time.

[0130] The following example illustrates this: The private Dogan news agency reported that more than 30 of those wounded in A, primarily children, were transferred to a hospital in B.

[0131] It showed footage of ambulances arriving at C hospital and medicalpersonnel <t> unloading< / t> A girl on stretchers.

[0132] The report, reported by the private Doan News Agency, states that over 30 injured people, mostly children, were transferred from location A to a hospital in location B. The report includes footage of ambulances arriving at hospital C and medical staff unloading a girl from a stretcher.

[0133] The event type is a motion event triggered by the trigger word "unloading". The arguments are "A", "ambulances", "C hospital", "medical personnel", and "a girl". Each argument plays a specific role in this event, namely the origin (the location indicated by A), the vehicle (the vehicle indicated by ambulances), the destination (the location indicated by C hospital), the transporter (the transporter indicated by medical personnel), and the passenger (the passenger indicated by a girl).

[0134] In some embodiments, step S101 may involve performing part-of-speech tagging (POS tagging) and dependency parsing on the text data, and then obtaining the triggering words for at least one first instance event corresponding to the text data based on specific rules. In some embodiments, the specific rules refer to most verbs except stop words and adjectives that satisfy several dependency relationships.

[0135] In some embodiments, when the text data is a document, step S101 can be implemented by first preprocessing the text data, for example, using a BERT-based Next Sentence Prediction (NSP) model and heuristic rules to identify document (5 to 10 sentences per document) boundaries, and then performing part-of-speech tagging and dependency analysis on the preprocessed text data to obtain the trigger words of at least one first instance event corresponding to the text data.

[0136] The embodiments of this application do not limit the method for obtaining the trigger word of at least one first instance event corresponding to the text data.

[0137] Step S102: Perform image classification processing on the image data to obtain the event type of at least one second instance event corresponding to the image data.

[0138] Here, image data refers to data in the form of images. Step S102 can be implemented using a deep learning-based image classification network, such as ViT or ResNet, to perform image classification processing on the image data and obtain the event type of at least one second instance event corresponding to the image data. To enable the image classification network to identify the event type of the second instance event, the network can be pre-trained using a data sample set with labeled event types, thus giving it the ability to recognize event types. This application embodiment does not limit the type of image classification network.

[0139] Step S103: Determine the first concept event that matches each of the trigger words based on the multimodal ontology.

[0140] Here, multimodal refers to having at least two modalities, such as text and image modalities, or other modalities. This application does not limit the form of the modalities. A conceptual event refers to an abstract event generalized from instance events.

[0141] In some embodiments, a multimodal ontology may include a lexicon, a first frame, and a second frame, wherein the first frame is used to determine conceptual events of text data, and the second frame is used to determine conceptual events of image data.

[0142] Here, a lexicon refers to a database containing word meanings used to determine the meaning of words. In some embodiments, the lexicon can be WordNet. In WordNet, nouns, adjectives, and adverbs are stored in this database in the form of synonym sets (synsets). Each synset represents a set of synonyms, and the synsets are connected equilaterally through semantic and part-of-speech relationships. This application does not limit the type of lexicon. In some embodiments, the first frame can be FrameNet, and the second frame can be Imitu. This application does not limit the types of the first and second frames.

[0143] In some embodiments, a multimodal ontology is formed by establishing a mapping relationship between the lexicon and the first frame, and between the lexicon and the second frame, and by fusing the lexicon, the first frame, and the second frame.

[0144] In some embodiments, the construction of the multimodal ontology may further include, prior to the fusion lexicon, the first frame, and the second frame:

[0145] By extending the Simple Event Model as the basis for knowledge representation, ontology can be defined in the Resource Description Framework (RDF).

[0146] When the first framework is FrameNet, the single-event representation in the ontology can be extended from SEM and FrameNet; that is, the RDF structure of a single event is extended from the definitions of SEM and FrameNet. In some embodiments, the relationships between events include the following three types:

[0147] (1) The relationship between instance events and concept events;

[0148] (2) Relationships between conceptual events, used to represent hierarchical relationships between conceptual events;

[0149] (3) Other relationships between instance events, such as temporal or causal relationships. In some embodiments, the relationship between instance events can be represented by adding link nodes. In this way, when there are multiple event relationships that may exist between two instance events, the link nodes can be used to quickly query (only query the required event relationships) and modify (without modifying the description of the edge between subject and object, only the link node information needs to be modified), making the ontology have good scalability.

[0150] In some embodiments, the ontology may also include virtual nodes connected to instance events to represent the source of the instance event. The edges of the virtual nodes can be used to represent information such as trigger words, modalities, image / text indexes, etc., of the instance event. Virtual nodes are designed for clear structural integration and are only queried when needed, reducing unnecessary queries.

[0151] Based on the construction of the multimodal ontology, the implementation of step S103 may include the following steps S1031 to S1032: Step S1031: Determine the first meaning matching each trigger word in the lexicon of the multimodal ontology; here, the implementation of step S1031 may be based on the meaning of the trigger word to determine the first meaning matching each trigger word in the lexicon of the multimodal ontology. For example, if the trigger word is "buy", then "buy" in the lexicon of the multimodal ontology can be determined as the first meaning. Step S1032: Determine the first concept event matching each first meaning in the first frame of the multimodal ontology. Here, since a mapping relationship has been established between the first frame and the lexicon in the multimodal ontology, after determining the first meaning in the lexicon, the first concept event matching each first meaning can be determined in the first frame.

[0152] In this embodiment of the application, by determining the first word meaning matching each trigger word in the lexicon of the multimodal ontology, and then determining the first concept event matching each first word meaning in the first framework of the multimodal ontology, the concept event corresponding to at least one first instance event of the text data is determined, which facilitates the subsequent extraction of arguments from the text data.

[0153] Step S104: Determine a second conceptual event that matches each of the event types based on the multimodal ontology;

[0154] Here, based on the construction of a multimodal ontology, such as Figure 5B As shown, the implementation of step S104 may include the following steps S1041 to S1043:

[0155] Step S1041: Determine a first event matching each of the event types within the second framework of the multimodal ontology;

[0156] Here, since the second frame contains the event type for each event, the implementation of step S1041 can be based on the event type for each event contained in the second frame, and the first event in the second frame that matches each event type determined in step S102 can be determined.

[0157] Step S1042: Determine the second meaning that matches each of the first events in the lexicon of the multimodal ontology;

[0158] Here, since a mapping relationship has been established between the lexicon of the multimodal ontology and the second frame, the second meaning that matches each first event can be determined in the lexicon of the multimodal ontology.

[0159] Step S1043: Determine the second concept event that matches each of the second semantic meanings within the first framework of the multimodal ontology.

[0160] Here, since a mapping relationship has been established between the first frame and the lexicon in the multimodal ontology, after determining the second word meaning in the lexicon, the first concept event matching each second word meaning can be determined in the first frame.

[0161] In this embodiment, by determining a first event matching each event type in the second framework of the multimodal ontology, then determining a second meaning matching each first event in the lexicon of the multimodal ontology, and finally determining a second concept event matching each second meaning in the first framework of the multimodal ontology, the concept event corresponding to at least one second instance event of the image data is determined, which facilitates the subsequent extraction of arguments from the image data.

[0162] Step S105: Construct a multimodal event knowledge graph based on the first concept event matched by each trigger word and the second concept event matched by each event type.

[0163] Since the roles that match the conceptual events can be identified after obtaining the conceptual events, arguments can be extracted from text or images based on the roles, thereby constructing a multimodal event knowledge graph.

[0164] In some embodiments, such as Figure 5C As shown, the implementation of step S105 may include the following steps S1051 to S1053:

[0165] Step S1051: Based on the first concept event matched by each trigger word, extract events from the text data to obtain the arguments of each first instance event;

[0166] Here, the implementation of step S1051 may include: determining the role corresponding to the first concept event based on the first concept event; and then extracting events from the text data based on the determined role to obtain the arguments of each first instance event.

[0167] Step S1052: Based on the second concept event matched for each event type, perform event extraction on the image data to obtain the arguments of each second instance event;

[0168] Here, the implementation of step S1052 may include: determining the role corresponding to the second concept event based on the second concept event; and then extracting events from the image data based on the determined role to obtain the arguments of each second instance event.

[0169] Step S1053: Construct a multimodal event knowledge graph based on the arguments of each first instance event and each second instance event.

[0170] In some embodiments, a multimodal event knowledge graph may include arguments, and thus, a multimodal event knowledge graph can be constructed using the arguments of each first instance event and the arguments of each second instance event.

[0171] In this embodiment, a first concept event of text data and a second concept event of image data are determined through a multimodal ontology. Then, based on the first concept event and the second concept event, a multimodal event knowledge graph is constructed, thereby realizing the complementarity and fusion between multimodal information, providing more information for the construction of the knowledge graph, and providing more possibilities for knowledge mining.

[0172] In some embodiments, after step S105, as Figure 5D As shown, the method further includes steps S106a, S107a, and S108a, wherein:

[0173] Step S106a: Determine at least one first instance event and at least one second instance event as the first instance event set;

[0174] That is, the first instance event set includes the first instance event and the second instance event.

[0175] Step S107a: Determine the event relationship between two instance events within the first instance event set;

[0176] Here, the event relationship between two instance events can include temporal and causal relationships, semantic relationships, and co-occurrence relationships.

[0177] For temporal causal relationships, this application proposes a novel method: constructing a document-level graph to globally infer relationships between events. This method enables cross-sentence reasoning even without explicit temporal / causal metrics and complex heuristics. This allows the application to simultaneously and effectively identify all temporal and causal relationships within a document. The application jointly predicts temporal and causal relationships as a multi-label, multi-task classification and trains a model based on a causal temporal database, including six relationship types: before, after, during, including, included, and causal.

[0178] Co-occurrence relationships include textual co-occurrence and cross-modal co-occurrence. For textual co-occurrence, if there is a conjunction relationship between triggering words, it is considered textual co-occurrence. This application embodiment identifies conjunction relationships through dependency analysis. For cross-modal co-occurrence, this application embodiment extracts events from pairs of image text and assumes they co-occur, thereby achieving information mining between pairs of image text.

[0179] Regarding semantic relationships, this application proposes a method for determining semantic relationships. Correspondingly, the two instance events in step S107a include a third instance event and a fourth instance event, such as... Figure 5E As shown, the implementation of step S107a includes the following steps S1071a to S1073a:

[0180] Step S1071a: Determine the syntactic relationship between the third instance event trigger word and the fourth instance event trigger word;

[0181] Here, grammatical relations include csubj (subject clause), xcomp (open complement clause), ccomp (complement clause), or advcl (adverbial clause).

[0182] Step S1072a: Determine the dependency relationship between the third instance event trigger word and the fourth instance event argument;

[0183] In some embodiments, step S1072a may be implemented by determining whether the third instance event trigger word is an argument of the fourth instance event. If so, the dependency relationship between the third instance event trigger word and the fourth instance event argument may be determined.

[0184] Step S1073a: When the grammatical relation satisfies the preset conditions and the third instance event trigger word is the fourth instance event argument, the third instance event and the fourth instance event have a semantic relation.

[0185] Here, the default condition is that, in dependency analysis, trigger words are connected by csubj (subject clause), xcomp (open complement clause), ccomp (complement clause), or advcl (adverbial clause).

[0186] In other words, when the third and fourth instance event trigger words are connected by csubj (subject clause), xcomp (open complement clause), ccomp (complement clause), or advcl (adverbial clause), and the third instance event trigger word is an argument of the fourth instance event, the third instance event and the fourth instance event have a semantic relationship. For example, from the sentence "Eating too much fried chicken led to being overweight," two events are extracted: leading to being overweight (event A) and eating too much chicken (event B). Since A is also an argument of the role influencing_entity of B, and the trigger word for event A is "led to," while the trigger word for event B is "eat," the grammatical relationship between the trigger words of event A and event B is a subject clause. Therefore, event A and event B have a semantic relationship.

[0187] This application provides a new type of relationship (semantic relationship) between instance events and a method for determining semantic relationships, thereby enriching the composition of the knowledge graph, expanding its scale, and providing more information for mining.

[0188] Step S108a: Add the event relationship between the two instance events to the multimodal event knowledge graph.

[0189] In this embodiment of the application, by determining the event relationship between two instance events and adding it to the multimodal event knowledge graph, the composition of the knowledge graph is enriched and more information that can be mined is provided.

[0190] In some embodiments, after step S105, this application embodiment also provides a method for updating the ontology, such as... Figure 5F As shown, the method further includes steps S106b, S107b, and S108b, wherein:

[0191] Step S106b: Determine the first salience score of the event relationship between two instance events within the first instance event set;

[0192] Here, the event relationship between two instance events can include temporal and causal relationships, semantic relationships, and co-occurrence relationships. The calculation of the first salience score can consider the following two aspects: (1) the confidence score of the relationship r between instance events, and (2) the number of times the tail event occurs (as few as possible). In this way, more important and meaningful information can be extracted.

[0193] Step S107b: Based on the first saliency score, infer the relationship between the conceptual events corresponding to the instance events;

[0194] Here, the implementation of step S107b may include: if the first saliency score exceeds the threshold, reasoning that there is also a relationship between the conceptual event corresponding to the instance event and the instance event.

[0195] Step S108b: Based on the reasoning, obtain the relationship between the conceptual events corresponding to the instance events, and update the ontology.

[0196] Here, the implementation of step S108b includes: adding the relationship between conceptual events to the ontology and updating the ontology.

[0197] In this embodiment of the application, a first salience score is determined for the event relationship between two instance events within a first instance event set. Then, based on the first salience score, the relationship between the concept events corresponding to the instance events is inferred, thereby updating the ontology. This makes the hierarchical relationship between the ontology more granular and facilitates the acquisition of richer knowledge information.

[0198] This application provides a method for constructing a multimodal ontology, such as... Figure 5G As shown, the method includes the following steps 31 to 33:

[0199] Step 31: Match the word meanings in the lexicon with each event in the first frame to establish a first mapping relationship between the lexicon and the first frame;

[0200] The first frame is used to determine the conceptual events of the text data; the first frame can be FrameNet, and the vocabulary can be WordNet.

[0201] In some embodiments, the implementation of step 31 may include the following steps 311 to 314:

[0202] Step 311: Determine the event type of each event in the first frame;

[0203] Here, the event type for each event in the first frame can be a manually labeled event type. In the case of a FrameNet as the first frame, each event in the FrameNet has its own built-in event type.

[0204] Step 312: Determine the cosine similarity between each word meaning in the lexicon and the event type of each event in the first frame;

[0205] Here, step 312 can be implemented by converting each word meaning in the lexicon and the event type of each event in the first frame into vector form (e.g., by converting into vector form through Sentence-BERT), and then calculating the cosine similarity between each word meaning in the lexicon and the event type of each event in the first frame.

[0206] Step 313: Based on the cosine similarity, determine the events in the first frame that match the meaning of each word in the lexicon;

[0207] Here, step 313 can be implemented by selecting the event corresponding to the event type with the highest cosine similarity based on each word meaning in the lexicon, as the event in the first frame that matches the word meaning, and thus determining the event in the first frame that matches each word meaning in the lexicon.

[0208] Step 314: Based on the events in the first frame that match the meaning of each word in the lexicon, establish a first mapping relationship between the lexicon and the first frame.

[0209] Because the events in the first framework FrameNet are broad in scope—for example, a chemical substance transforming into another is called "Cause_change," and the name is also "Cause_change"—while the word library WordNet has a richer vocabulary, aligning WordNet with FrameNet can make the concept events corresponding to FrameNet more granular. In addition, WordNet also brings hierarchical information to FrameNet, thus making the constructed knowledge graph richer in information.

[0210] Step 32: Match the word meanings in the lexicon with each event in the second frame to establish a second mapping relationship between the lexicon and the second frame;

[0211] The second framework is used to determine conceptual events in the image data; the second framework can be Imitu, and the implementation of step 32 may include the following steps 321 to 323:

[0212] Step 321: Determine the event type for each event in the second frame;

[0213] Here, the event type for each event in the second frame can be a manually labeled event type. In the case of an Imitu second frame, each event in an Imitu comes with its own event type.

[0214] Step 322: Based on the event type of each event in the second frame, determine the word meaning in the lexicon that matches each event in the second frame;

[0215] Here, step 322 can be implemented by manually aligning each event in the second frame with the word meaning in the lexicon, based on the event type of each event in the second frame and the word meaning in the lexicon.

[0216] Step 323: Based on the word meanings in the lexicon that match each event in the second frame, establish a second mapping relationship between the lexicon and the second frame.

[0217] Since the second framework, Imsitu, is used to describe image events, and image events are triggered by verbs, the verbs corresponding to the event types in Imsitu do not have semantic meanings. Therefore, they need to be aligned with the semantic meanings in the WordNet lexicon. Since there is a mapping relationship between WordNet and the first framework, FrameNet, the event types in FrameNet can be associated with WordNet to obtain the concept events corresponding to the images. FrameNet is used to obtain the concept events corresponding to the text, thereby realizing the construction of a multimodal knowledge graph under the same ontology.

[0218] Step 33: Based on the first mapping relationship and the second mapping relationship, fuse the lexicon, the first frame and the second frame to obtain the multimodal ontology.

[0219] Because there is a second mapping relationship between the second frame and the lexicon, and a first mapping relationship between the first frame and the lexicon, the second frame and the first frame are linked through the lexicon, thus realizing the construction of knowledge graphs for both text and image modalities.

[0220] In this embodiment, firstly, since the second framework Imitu is used to describe image events, and image events are triggered by verbs, while the verbs corresponding to event types in Imitu do not have semantic meaning, they need to be aligned with the semantic meanings in the WordNet lexicon. Furthermore, because there is a mapping relationship between WordNet and the first framework FrameNet, event types in FrameNet can be associated with WordNet to obtain the concept events corresponding to images. FrameNet is used to obtain the concept events corresponding to text, thereby enabling the construction of a multimodal knowledge graph under the same ontology. Secondly, since the events in the first framework FrameNet have a broad coverage, while the semantic meanings in the WordNet lexicon are richer, aligning WordNet with FrameNet allows for more granular concept events corresponding to FrameNet. Moreover, WordNet brings more hierarchical information to FrameNet, thus enriching the information in the constructed knowledge graph.

[0221] In some embodiments, after obtaining the multimodal ontology in step 33, this application embodiment also provides a method for updating the multimodal ontology, such as... Figure 5H As shown, it also includes steps 34 to 38, wherein:

[0222] Step 34: Obtain the concept events in the multimodal ontology, the specific roles in the concept events, and the second set of instance events that match the concept events;

[0223] Here, the concept event and the role within the concept event can be any concept event and any role. For example, the concept event is "fry," and the specific role is "food." The second set of instance events is all instance events under the concept event "fry."

[0224] Step 35: Determine the set of arguments corresponding to the specific role in the second set of instance events;

[0225] Here, we continue to use the concept event "fry" and the specific role "food" as an example. The arguments of the role "food" in the second set of instance events include fish, a fish, the fish, a big fish, chicken, and the chicken, forming an argument set.

[0226] Step 36: Perform clustering on the set of arguments to obtain the second salience score for each class;

[0227] Here, the implementation of step 36 may include: performing lexical reduction and heuristic clustering on the center words of all arguments in the argument set, and then calculating the second salience score for each class.

[0228] Continuing with the above example, the words "fish," "a fish," "the fish," and "a big fish" all become "fish" after word form reduction, and "chicken" and "the chicken" all become "chicken" after word form reduction. After heuristic clustering, two categories are obtained: "fish" and "chicken." Calculate the significance score for each category, considering the following two aspects: (1) the confidence score (number of occurrences) for each category; (2) the importance of each category. For example, if the category is "I," its importance is not high, and the weight of this category can be reduced.

[0229] Step 37: Based on the second saliency score of each class and the trigger word of the conceptual event, infer new conceptual events;

[0230] Here, the implementation of step 37 may include: selecting the K types with the highest salience scores, and inferring new conceptual events by combining the role r and these names with trigger words. For example, if fish appears frequently and has a high salience score, when K is 1, combining fish with the trigger word fry and the role food will derive the new conceptual event "fry fish".

[0231] Step 38: Update the multimodal ontology based on the new conceptual events.

[0232] Here, the implementation of step 38 can add new conceptual events to the multimodal ontology, thereby updating the multimodal ontology.

[0233] In this embodiment, by determining the saliency score of arguments in an instance event, and then combining it with trigger word reasoning to obtain a new concept event, the new concept event obtained by reasoning is then added to the multimodal ontology to update the multimodal ontology, thereby making the relationships in the ontology richer and more granular, and thus realizing the construction of a large-scale event knowledge graph.

[0234] The event argument extraction method (PAIE, Prompting Argument Interaction for Event Argument Extraction) proposed in this application includes: an event extraction method, characterized by comprising: acquiring input text; wherein the input text includes trigger words and at least one event argument; extracting a first set of text feature vectors corresponding to the input text through an encoder and a decoder; acquiring a target extraction template corresponding to the trigger words; decoding the target extraction template and the input text through the decoder to obtain an argument feature vector for each event argument; and determining a value range corresponding to each event argument based on the first set of text feature vectors and the argument feature vector for each event argument.

[0235] In some embodiments, the method further includes: converting the input text and the target extraction template into a text word vector set and a template word vector set, respectively; extracting a first text feature vector set corresponding to the input text through an encoder and a decoder includes: encoding the text word vector set through the encoder to obtain a second text feature vector set; decoding the text word vector set and the second text feature vector set through the decoder to obtain the first text feature vector set corresponding to the input text; correspondingly, decoding the target extraction template and the input text through the decoder to obtain the argument feature vector of each event argument includes: decoding the template word vector set and the second text feature vector set through the decoder to obtain the argument feature vector of each event argument.

[0236] In some embodiments, the step of decoding the template word vector set and the second text feature vector set using the decoder to obtain the argument feature vector of each event argument includes: decoding the template word vector set and the second text feature vector set using the decoder to obtain a template feature vector set that fuses the input text information; and determining the argument feature vector of each event argument corresponding to each argument role from the template feature vector set based on the position information of each argument role in the template word vector set.

[0237] In some embodiments, determining the argument feature vector of each event argument corresponding to each argument role from the template feature vector set based on the position information of each argument role in the template word vector set includes: extracting the feature vector of each event argument corresponding to the argument role from the template feature vector set based on the position information of each argument role in the template word vector set; and performing average pooling on the feature vector of each event argument to obtain the argument feature vector of the event argument.

[0238] Events are the basic building blocks describing activities and states in the objective world. Knowledge graphs, with events as nodes, summarize the events inherent in various fields of real life and the interactions between them, and can be applied significantly in a range of downstream tasks such as common-sense reasoning. In recent years, a large number of works have emerged related to building event knowledge graphs: ATOMIC built a small-scale, high-quality event knowledge graph through manual annotation, while ASER built a large-scale, relatively low-quality event knowledge graph through automated rules. Building high-quality, large-scale event knowledge graphs without overly relying on manual annotation remains an unresolved problem in previous work.

[0239] In addition, previous work did not rigorously design an ontology for event knowledge graphs, nor did it consider the complementarity and fusion of multimodal information (such as text and visual images). Therefore, this application's embodiments will utilize a deep learning-based event knowledge extraction system to extract a greater amount of event knowledge at a lower cost (meaning without relying on extensive manual annotation), and will consider the fusion and complementarity of event knowledge between multimodal information (images and text), as well as define a high-quality, large-scale, and practically useful ontology for the event knowledge graph.

[0240] This application provides a multimodal event knowledge extraction system. This system utilizes cutting-edge deep learning technology as its core, supplemented by mature rules and templates, enabling it to quickly and efficiently extract large amounts of event knowledge from text and images. This extraction system achieves the best results among currently known systems of its type.

[0241] This application provides an ontology for a multimodal event knowledge graph, defining four different types of nodes and edges for the graph, and specifying the association constraints between different events and relationships. By aligning external resources (merging similar nodes from FrameNet and WordNet), an initial ontology is obtained, and based on this ontology, combined with the extracted event instances, a hierarchical ontology of events / event relationships is summarized.

[0242] The method provided in this application mainly includes the following two processes: the first process is an ontology construction process, and the second process is a knowledge extraction process for multimodal events, wherein:

[0243] First Process: Ontology Construction Process

[0244] A new ontology was built on top of the traditional Simple Event Model, as follows: Figure 3 As shown.

[0245] Figure 3 In the diagram, dashed boxes represent character attributes, and solid boxes represent entities / relationships / events, etc. Entities: Basic elements in a knowledge graph that can have relationships with other entities or character attributes. Non-entities (character attributes): Basic elements in a knowledge graph that can only exist as attributes of a certain entity. For example, the knowledge graph contains two triples: (Tom, born in, USA) and (USA, established in, 1776). (1) Tom, USA is an entity, (2) born in, established in is a relation, and (3) 1776 is a non-entity / attribute.

[0246] The upper left part (solid-lined boxes in the grid, such as cpt_Event_565, ins_Event_317) represents the event-related ontology, the right part (gray solid-lined boxes, such as [EventRel_id], [RelType_id]) represents the relationship-related ontology, and the lower part (the part connected by circles, i.e., virtual nodes, such as [span]) represents the raw event information obtained by the extraction system. ekg represents a new class defined in this application embodiment, rdf represents a triple, ins_Event represents an instance event, and cpt_Event represents a conceptual event.

[0247] Generally, the entire event ontology can be divided into two parts: the ontology level and the instance level. For example, `cpt_Event_565` and above belong to the ontology level, while `ins_Event_317` and below belong to the instance level. First, an initial ontology is obtained through alignment between FrameNet and WordNet. Based on this initial ontology, the instance events obtained by the event extraction system are statistically analyzed, and those with higher frequency occurrences are merged into the ontology level as new ontology events. Furthermore, a similar method is used to merge frequently occurring instance level relationships into the ontology level.

[0248] Second process: Knowledge extraction process for multimodal events

[0249] The process is divided into three parts: text knowledge extraction process, visual knowledge extraction process, and event relationship extraction process.

[0250] Text knowledge extraction: First, the processed text undergoes part-of-speech tagging (POS tagging) and dependency parsing. Trigger words for the events are then obtained based on specific rules. POS tagging can be machine-generated, and the specific rules may include: the vast majority of verbs (maintaining a stop word list) and adjectives satisfying several dependency relationships. Then, word sense disambiguation is performed on these words to obtain basic event types. Finally, given the event types, a prompt-based deep learning model is used for event argument extraction to obtain the complete instance event, i.e., the PAIE method provided in this application.

[0251] Among these, part-of-speech tagging determines the part of speech of a word (verb? noun? adjective?), word meaning disambiguation determines the meaning of polysemous words in the text (e.g., does the English word "fire" mean to dismiss or to fire a weapon?), and stop words are generally considered in dictionaries to be function words or non-search terms in the search. Stop words can also be understood as words or phrases that search engines automatically ignore when processing search requests to save storage space and improve search efficiency; these words or phrases are called stop words.

[0252] Visual knowledge extraction: Similar to text knowledge extraction, the event represented by the image is first determined by action recognition technology, and then the relevant arguments of the event are determined by situation recognition technology.

[0253] Event Relationship Extraction: Event relationships are categorized into different types and processed accordingly. For causal and temporal relationships, a Longform-based relational graph reasoning technique is used for event relationship extraction. For image-text co-occurrence relationships, the relationships in image-text pairs are aligned to obtain the corresponding relationships. Paired image-text data can be used to extract events from both images and text separately; two events extracted from a single image-text pair are considered aligned. For other relationships, extraction is based on defined rules. The classification of other event relationships is more complex, including text co-occurrence relationships and hundreds of semantic relationships.

[0254] From the above technical solutions, it can be seen that the embodiments of this application have the following characteristics: 1) The embodiments of this application use an automated knowledge extraction system based on deep learning to extract event knowledge to construct a knowledge graph, while related technologies use manual annotation or simple rule extraction to construct knowledge graphs. 2) The knowledge graph constructed in the embodiments of this application considers the interaction and fusion of multiple modal events, while related technologies only consider the single modality of text. 3) The embodiments of this application integrate external databases and extracted event instances, and the event knowledge graph constructs a large-scale, high-quality ontology, while related technologies have no ontology or only a simple ontology, and do not inductively derive new event ontology based on event instances. Therefore, the embodiments of this application have a faster, higher-quality event knowledge extraction speed, a more reasonable event knowledge organization and storage method, and finally can realize the interactive fusion of multimodal knowledge. The methods and systems provided in the embodiments of this application can be widely applied to structured information extraction service products, common sense reasoning-related service products, and other products.

[0255] Based on the foregoing embodiments, this application provides a knowledge graph construction device. The device includes various modules and units, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0256] Figure 6A This is a schematic diagram of the composition structure of a knowledge graph construction device provided in an embodiment of this application, such as... Figure 6A As shown, the knowledge graph construction device 610 includes:

[0257] Detection module 611 is used to perform event detection on text data to obtain the trigger word of at least one first instance event corresponding to the text data;

[0258] The classification module 612 is used to perform image classification processing on image data to obtain the event type of at least one second instance event corresponding to the image data;

[0259] The first determining module 613 is used to determine a first concept event that matches each of the trigger words based on the multimodal ontology;

[0260] The second determining module 614 is used to determine a second conceptual event that matches each of the event types based on the multimodal ontology;

[0261] The construction module 615 is used to construct a multimodal event knowledge graph based on the first concept event matched by each of the trigger words and the second concept event matched by each of the event types.

[0262] In some embodiments, the multimodal ontology includes a lexicon, a first frame, and a second frame, wherein the first frame is used to determine conceptual events of text data, and the second frame is used to determine conceptual events of image data; the first determining module includes: a first determining unit, used to determine a first meaning matching each trigger word in the lexicon of the multimodal ontology; and a second determining unit, used to determine a first conceptual event matching each first meaning in the first frame of the multimodal ontology.

[0263] In some embodiments, the second determining module includes: a third determining unit, configured to determine a first event matching each event type in the second framework of the multimodal ontology; a fourth determining unit, configured to determine a second word meaning matching each first event in the lexicon of the multimodal ontology; and a fifth determining unit, configured to determine a second concept event matching each second word meaning in the first framework of the multimodal ontology.

[0264] In some embodiments, the construction module includes: a first extraction unit, configured to extract events from the text data based on a first concept event matched by each trigger word, to obtain arguments for each first instance event; a second extraction unit, configured to extract events from the image data based on a second concept event matched by each event type, to obtain arguments for each second instance event; and a construction unit, configured to construct a multimodal event knowledge graph based on the arguments for each first instance event and the arguments for each second instance event.

[0265] In some embodiments, the apparatus further includes: a third determining module, configured to determine at least one first instance event and at least one second instance event as a first instance event set; a fourth determining module, configured to determine the event relationship between two instance events within the first instance event set; and an adding module, configured to add the event relationship between the two instance events to the multimodal event knowledge graph.

[0266] In some embodiments, the two instance events include a third instance event and a fourth instance event. The fourth determining module is used to determine the syntactic relationship between the third instance event trigger word and the fourth instance event trigger word; determine the subordinate relationship between the third instance event trigger word and the fourth instance event argument; and, if the syntactic relationship satisfies a preset condition and the third instance event trigger word is the fourth instance event argument, the third instance event and the fourth instance event have a semantic relationship.

[0267] In some embodiments, the apparatus further includes: a fifth determining module, configured to determine a first saliency score of the event relationship between two instance events within the first instance event set; a first reasoning module, configured to reason about the relationship between conceptual events corresponding to the instance events based on the first saliency score; and an updating module, configured to update the ontology based on the reasoned relationship between the conceptual events corresponding to the instance events.

[0268] Based on the foregoing embodiments, this application provides a multimodal ontology construction apparatus. The apparatus includes the modules and units included in each module, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0269] Figure 6B This is a schematic diagram of the composition structure of a multimodal ontology construction device provided in an embodiment of this application, as shown below. Figure 6B As shown, the construction apparatus 620 for the multimodal ontology includes:

[0270] The first establishment module 621 is used to match the word meanings in the lexicon with each event in the first frame to establish a first mapping relationship between the lexicon and the first frame, wherein the first frame is used to determine the conceptual events of the text data.

[0271] The second establishment module 622 is used to match the word meanings in the lexicon with each event in the second frame, and establish a second mapping relationship between the lexicon and the second frame, wherein the second frame is used to determine the conceptual events of the image data.

[0272] The fusion module 623 is used to fuse the lexicon, the first frame, and the second frame based on the first mapping relationship and the second mapping relationship to obtain the multimodal ontology.

[0273] In some embodiments, the first establishing module includes: a sixth determining unit, configured to determine the event type of each event in the first frame; a seventh determining unit, configured to determine the cosine similarity between each word meaning in the lexicon and the event type of each event in the first frame; an eighth unit, configured to determine, based on the cosine similarity, the events in the first frame that match each word meaning in the lexicon; and a first establishing unit, configured to establish a first mapping relationship between the lexicon and the first frame based on the events in the first frame that match each word meaning in the lexicon.

[0274] In some embodiments, the second establishing module includes: a ninth determining unit, configured to determine the event type of each event in the second frame; a tenth determining unit, configured to determine the word meaning in the lexicon that matches each event in the second frame based on the event type of each event in the second frame; and a second establishing unit, configured to establish a second mapping relationship between the lexicon and the second frame based on the word meaning in the lexicon that matches each event in the second frame.

[0275] In some embodiments, the apparatus further includes: an acquisition module, configured to acquire concept events in the multimodal ontology, specific roles in the concept events, and a second set of instance events matching the concept events; a sixth determination module, configured to determine the set of arguments corresponding to the specific roles in the second set of instance events; a clustering module, configured to perform clustering processing on the set of arguments to obtain a second saliency score for each class; a second inference module, configured to infer new concept events based on the second saliency score of each class and the trigger words of the concept events; and an update module, configured to update the multimodal ontology based on the new concept events.

[0276] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0277] It should be noted that, in the embodiments of this application, if the above-mentioned knowledge graph construction method or multimodal ontology construction method is implemented in the form of software functional modules and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0278] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0279] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0280] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0281] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0282] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0283] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0284] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the computer device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein:

[0285] Processor 701 typically controls the overall operation of computer device 700.

[0286] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0287] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0288] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0289] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0290] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0291] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0292] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0293] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0294] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0295] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for constructing a knowledge graph, characterized in that, include: Event detection is performed on the text data to obtain the trigger word corresponding to at least one first instance event; The image data is subjected to image classification processing to obtain the event type corresponding to at least one second instance event of the image data; The first conceptual event matching each of the aforementioned trigger words is determined based on the multimodal ontology; Based on the multimodal ontology, a second conceptual event matching each of the event types is determined; A multimodal event knowledge graph is constructed based on the first concept event matched by each trigger word and the second concept event matched by each event type. The multimodal ontology includes a lexicon, a first frame, and a second frame, wherein the first frame is used to determine conceptual events of text data, and the second frame is used to determine conceptual events of image data. The determination of the first concept event matching each trigger word based on the multimodal ontology includes: In the lexicon of the multimodal ontology, determine the first meaning that matches each of the trigger words; Within the first framework of the multimodal ontology, a first conceptual event matching each of the first word meanings is determined; The step of determining the second conceptual event matching each event type based on the multimodal ontology includes: In the second framework of the multimodal ontology, a first event matching each of the event types is determined; Determine a second meaning that matches each of the first events in the lexicon of the multimodal ontology; In the first framework of the multimodal ontology, a second conceptual event matching each of the second semantic meanings is determined.

2. The method according to claim 1, characterized in that, The construction of a multimodal event knowledge graph based on the first concept event matched by each trigger word and the second concept event matched by each event type includes: Based on the first concept event matched by each of the trigger words, the text data is extracted to obtain the arguments of each first instance event; Based on the second concept event that matches each event type, the image data is subjected to event extraction to obtain the arguments of each second instance event; A multimodal event knowledge graph is constructed based on the arguments of each first instance event and each second instance event.

3. The method according to claim 1, characterized in that, The method further includes: The at least one first instance event and the at least one second instance event are determined as the first instance event set; Determine the event relationship between two instance events within the first instance event set; The event relationship between the two instance events is added to the multimodal event knowledge graph.

4. The method according to claim 3, characterized in that, The two instance events include a third instance event and a fourth instance event. Determining the event relationship between the two instance events within the first instance event set includes: Determine the syntactic relationship between the third instance event trigger word and the fourth instance event trigger word; Determine the dependency relationship between the third instance event trigger word and the fourth instance event argument; When the grammatical relationship satisfies the preset conditions and the third instance event trigger word is the fourth instance event argument, the third instance event and the fourth instance event have a semantic relationship.

5. The method according to claim 3 or 4, characterized in that, The method further includes: Determine the first salience score of the event relationship between two instance events within the first instance event set; Based on the first salience score, the relationship between the conceptual events corresponding to the two instance events is inferred; Based on the reasoning, the relationship between the conceptual events corresponding to the two instance events is obtained, and the ontology is updated.

6. A method for constructing a multimodal ontology, characterized in that, include: The word meanings in the lexicon are matched with each event in the first frame to establish a first mapping relationship between the lexicon and the first frame, whereby the first frame is used to determine the conceptual events of the text data. The word meanings in the lexicon are matched with each event in the second frame to establish a second mapping relationship between the lexicon and the second frame, whereby the second frame is used to determine the conceptual events of the image data. Based on the first mapping relationship and the second mapping relationship, the lexicon, the first frame and the second frame are fused to obtain the multimodal ontology; The step of matching word meanings in the lexicon with each event in the second frame to establish a second mapping relationship between the lexicon and the second frame includes: Determine the event type for each event in the second frame; Based on the event type of each event in the second framework, determine the word meaning in the lexicon that matches each event in the second framework; Based on the word meanings in the lexicon that match each event in the second frame, a second mapping relationship is established between the lexicon and the second frame. The second frame and the first frame are associated through the lexicon.

7. The method according to claim 6, characterized in that, The step of matching word meanings in the lexicon with each event in the first frame to establish a first mapping relationship between the lexicon and the first frame includes: Determine the event type for each event in the first frame; Determine the cosine similarity between each word meaning in the lexicon and the event type of each event in the first frame; Based on the cosine similarity, determine the events in the first frame that match each word meaning in the lexicon; A first mapping relationship is established between the lexicon and the first frame based on the events in the first frame that match the meaning of each word in the lexicon.

8. The method according to claim 6 or 7, characterized in that, After obtaining the multimodal ontology, the following is also included: Obtain the conceptual events in the multimodal ontology, the specific roles in the conceptual events, and the second set of instance events that match the conceptual events; Determine the set of arguments corresponding to the specific role in the second set of instance events; Cluster the argument set to obtain the second saliency score for each class; Based on the second saliency score of each class and the trigger word of the conceptual event, new conceptual events are inferred; The multimodal ontology is updated based on the new conceptual events.

9. A knowledge graph construction device, characterized in that, include: The detection module is used to perform event detection on text data to obtain the trigger word of at least one first instance event corresponding to the text data; The classification module is used to perform image classification processing on image data to obtain the event type corresponding to at least one second instance event of the image data; The first determining module is used to determine the first conceptual event that matches each of the trigger words based on the multimodal ontology; The second determining module is used to determine a second conceptual event that matches each of the event types based on the multimodal ontology; The construction module is used to construct a multimodal event knowledge graph based on the first concept event matched by each of the trigger words and the second concept event matched by each of the event types. The multimodal ontology includes a lexicon, a first frame, and a second frame, wherein the first frame is used to determine conceptual events of text data, and the second frame is used to determine conceptual events of image data. The first determining module includes: The first determining unit is used to determine a first word meaning that matches each of the trigger words in the lexicon of the multimodal ontology; The second determining unit is used to determine a first concept event that matches each of the first word meanings within the first framework of the multimodal ontology; The second determining module includes: The third determining unit is used to determine a first event that matches each of the event types within the second framework of the multimodal ontology; The fourth determining unit is used to determine a second word meaning that matches each of the first events in the lexicon of the multimodal ontology; The fifth determining unit is used to determine a second conceptual event that matches each of the second semantic meanings within the first framework of the multimodal ontology.

10. A device for constructing a multimodal ontology, characterized in that, The device includes: The first module is used to match the word meanings in the lexicon with each event in the first frame, and to establish a first mapping relationship between the lexicon and the first frame, wherein the first frame is used to determine the conceptual events of the text data. The second establishment module is used to match the word meanings in the lexicon with each event in the second frame, and establish a second mapping relationship between the lexicon and the second frame, wherein the second frame is used to determine the conceptual events of the image data. A fusion module is used to fuse the lexicon, the first frame, and the second frame based on the first mapping relationship and the second mapping relationship to obtain the multimodal ontology; The second establishment module includes: The ninth determining unit is used to determine the event type of each event in the second frame; The tenth determining unit is used to determine the word meaning in the lexicon that matches each event in the second frame based on the event type of each event in the second frame; The second establishing unit is used to establish a second mapping relationship between the lexicon and the second frame based on the word meanings in the lexicon that match each event in the second frame. The second frame and the first frame are associated through the lexicon.

11. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images