A multimodal event ontology semi-automatic construction method

Abstract

This invention relates to the technical field, specifically to a semi-automatic method for constructing a multimodal event ontology, comprising the following steps: S1: Based on existing event ontology construction processes and automatic text event information extraction technology, a method for semi-automatic construction of a multimodal event ontology is proposed; S2: Image scene classification and target detection oriented towards visual modal fusion of the event ontology; S3: Image element correlation evaluation and representative selection; S4: Supplementation and summarization of multimodal information of event classes. In this invention, existing event ontology construction methods are improved by combining event information extraction technology to form a semi-automatic event ontology construction technology. This further integrates visual descriptive information, enhances the multimodal expressive power of the ontology, and solves the problem of insufficient expressive power of single-modal event ontology. It also improves the similarity between image and event elements by proposing image scene element matching degree and image entity element matching degree.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of event ontology construction, in particular to a multi-modal event ontology semi-automatic construction method. BACKGROUND

[0002] An event ontology is a knowledge representation system taking events as knowledge representation units, which models knowledge of a certain field through event classes and event relationships, and the event ontology can be widely applied in fields such as event knowledge acquisition, information retrieval, event reasoning, service recommendation and emergency decision-making of emergencies as a knowledge base.

[0003] Thanks to the rapid development of computers and networks, the means of information expression gradually diversifies, and a single text mode is insufficient to express the whole picture of information, how to fully utilize various information modes to realize more intelligent application is a problem to be solved in the current field of artificial intelligence, in the field of traditional event ontology construction, the existing event ontology construction technology is mainly aimed at event ontology construction under the single text mode, and there is certain limitation, and the construction method is mostly manual construction, which usually needs knowledge engineers to carry out construction work under the guidance of field experts, and the low construction efficiency affects the application of the event ontology. SUMMARY

[0004] The application aims to solve the problems in the prior art and provides a multi-modal event ontology semi-automatic construction method.

[0005] In order to achieve the above-mentioned purpose, the application adopts the following technical scheme: a multi-modal event ontology semi-automatic construction method, comprising the following steps:

[0006] S1: based on the existing event ontology construction process and the automatic extraction technology of text event information, a method for semi-automatic construction of multi-modal event ontology is proposed;

[0007] S2: image scene classification and target detection for event ontology visual modal fusion;

[0008] S3: image element correlation evaluation and representative selection;

[0009] S4: event class multi-modal information supplement and induction.

[0010] Further, the method for semi-automatic construction of multi-modal event ontology based on the existing event ontology construction process and the automatic extraction technology of text event information specifically comprises:

[0011] S101: determining the field and category of the event ontology, and collecting field text data according to the field characteristics;

[0012] S102: Reuse the existing event ontology pattern to quickly generate event classes and event relationships within the domain, which can be used as candidate event classes and event relationships for the construction of the domain event ontology. If there is no reusable ontology pattern, proceed to the third step.

[0013] S103: After segmenting the text, the event type is extracted using the existing event trigger word extraction model based on attention mechanism and Bi-GRU. Then, the event element is extracted using the existing event element extraction model based on BERT and attention mechanism. Finally, Bi-LSTM is used to encode and classify the information of the two events to achieve event relationship extraction, and finally, candidate event classes and event relationships are obtained.

[0014] S104: Domain experts manually screen and supplement the events and event relationships obtained in steps S102 and S103 to obtain the core event classes and event relationships in the domain.

[0015] S105: Based on the event six-tuple model and the event elements obtained in S102, define the main elements of the core event class, such as the event class's action, object, time, location, preceding state, and following state.

[0016] S106: By defining the hierarchical relationship between candidate event classes, a domain event class hierarchy is formed. Domain experts expand the event class hierarchy based on domain common sense, and finally form a relatively complete domain event class hierarchy model.

[0017] S107: Based on the domain event classes and event relationships obtained in step S104, construct event scenario models for the core event classes of the domain based on domain common sense. The event scenario model consists of event classes and event logical relationships, such as composition, causality, following, concurrency, etc.

[0018] S108: Collect images based on the source of the event text data, and formulate a multi-keyword search for images based on the event and its elements. Keywords consist of event name, object, location, and action, separated by spaces. Finally, the image set l = {p1, p2, ..., p...} is created. n This information is added to the multimodal information of the event and awaits further fusion.

[0019] Furthermore, the method of using Bi-LSTM to encode and classify the information of the two events to extract the event relationship is specifically represented as follows:

[0020] The event information is input into a Bi-LSTM for encoding, specifically represented as follows:

[0021] Bi-LSTM(X) = x c ={x0,x1,…x n}

[0022] Where X represents the input word vector, which can be vectorized using a Word2Vec pre-trained model. c This represents an intermediate vector with contextual information output by the model.

[0023] Finally, after passing through a fully connected layer with an output dimension of 4 and an output layer using Softmax as the activation function, the classification result is obtained, specifically represented as follows:

[0024]

[0025] Where W is the weight matrix to be learned, and b is the bias term. The probabilities output by softmax represent the classification probabilities of composition, causality, following, and concurrency relationships, respectively.

[0026] Furthermore, the image scene classification and target detection based on event ontology visual modality fusion specifically include:

[0027] S201: Using the existing VGG16-Places365 model pre-trained on the Places365 dataset, scene recognition and annotation are performed on each image of the event, and the top N scene labels P are selected. sce ={s1,s2,…,s N}, to be added to the multimodal information in the event;

[0028] S202: Using the existing YOLOv5 model, perform object detection and annotation on each image of the event to obtain N entity labels P. obj ={o1,o2,…,o N This information is added to the multimodal information of the event.

[0029] Furthermore, the image element correlation assessment and representativeness assessment specifically include:

[0030] S301: For all events in each event class, use a word vector pre-trained model to calculate the vector representations of N scene labels and the location elements of the event class for each image in the event, and calculate the cosine similarity between the N scene labels and the event location elements. The average similarity score is taken as the image scene element matching score Score1 for this image and added to the multimodal information of the event.

[0031] S302: For all events in each event class, use a word vector pre-training model to calculate the N image entity labels and the vector representation of the object elements of the event class for each image in the event, and calculate the cosine similarity between each object in the event class object elements and the N image entity labels. The average similarity is taken as the matching degree of the object. Finally, the average matching degree of all objects in the event object elements is taken as the image object element matching degree Score2 of the image, which is added to the multimodal information of the event.

[0032] S303: Combining the image scene element matching score Score1 and the image object element matching score Score2, calculate the comprehensive element matching score Score of the image. From all events in each event class, select the top M images with the highest scores as the candidate image set L = {p1, p2, ..., p...} M The comprehensive feature matching degree of the images and the candidate image set are added to the multimodal information of the event.

[0033] S304: Use representative image selection techniques to select representative images from the candidate image set.

[0034] Furthermore, the calculation of the cosine similarity between N scene labels and event location elements, and the taking of the average similarity as the image scene element matching degree of the image, specifically satisfies the following formula:

[0035]

[0036]

[0037] Where, p i Let represent the embedding representation of the i-th scene label, and 'a' represent the embedding representation of the event location element. Score1 represents the similarity between the i-th scene label and the event location element, N represents the number of scene labels, and Score1 represents the matching degree of the image scene elements.

[0038] Furthermore, the cosine similarity between each object in the event object elements and N image entity labels is calculated, and the average similarity is taken as the matching degree of the object. Finally, the average matching degree of all objects in the event object elements is taken as the image object element matching degree of the image, specifically satisfying the following formula:

[0039]

[0040]

[0041] Among them, o i b represents the embedding representation of the i-th image entity label. j This represents the embedding representation of the j-th object in the event object elements. Score2 represents the similarity between the i-th image entity label and the j-th object in the event object elements, M represents the number of objects in the event object elements, N represents the number of image entity labels, and Score2 represents the matching degree of image entity elements.

[0042] Furthermore, by combining the image scene element matching degree and the image object element matching degree, the comprehensive element matching degree of the image is calculated, and the top M images with the highest scores are selected as the candidate image set, specifically satisfying the following formula:

[0043] Score=α×Score1+β×Score2 (α+β=1)

[0044] Where α represents the weight of the matching degree of image scene elements, and β represents the weight of the matching degree of image entity elements.

[0045] Furthermore, the process of using representative image selection technology to filter representative images in the candidate image set specifically includes:

[0046] For each image in the candidate image set of the event class, extract the SIFT feature T of the image. i The similarity between images is calculated using cosine similarity, resulting in the image similarity feature matrix H. n×n Image clustering is performed using the AP algorithm based on the feature matrix; the visual feature overlap rate V of each cluster is calculated. j =match_avg(P j ), where match_avg(·) represents the average number of SIFT feature matching points between images within a cluster, P j Let P represent the set of image features for the j-th cluster; finally, the cluster center of the first-ranked cluster is used as the representative image P. rep .

[0047] Furthermore, the supplementation and summarization of the event-type multimodal information specifically includes:

[0048] The candidate image set, representative images, image entity set, image scene tags, image scene matching degree, image entity matching degree, and overall image matching degree are supplemented into the visual description of the multimodal information elements of the event category, i.e., M visual ={L,P rep ,P obj ,P sce ,Score1,Score2,Score}.

[0049] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0050] This invention improves upon existing event ontology construction methods by combining event information extraction techniques to form a semi-automatic event ontology construction technique. It further integrates visual description information to enhance the multimodal expressive power of the ontology and solve the problem of insufficient expressive power of single-modal event ontology. It improves the similarity between images and event elements by proposing image scene element matching degree and image entity element matching degree, and uses representative image selection techniques to summarize event-type multimodal information elements from image sets. Attached Figure Description

[0051] Figure 1 The present invention provides a semi-automatic method for constructing a multimodal event ontology, and presents an architecture diagram of the multimodal event ontology.

[0052] Figure 2 The present invention proposes a semi-automatic method for constructing a multimodal event ontology, and provides a flowchart of the semi-automatic construction process for this multimodal event ontology. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0055] Example 1

[0056] Please see Figure 1 This invention provides a technical solution: a semi-automatic method for constructing a multimodal event ontology, comprising the following steps:

[0057] S1: Based on the existing event ontology construction process and automatic text event information extraction technology, a semi-automatic method for constructing multimodal event ontology is proposed;

[0058] S2: Image scene classification and target detection based on visual modal fusion of event ontology;

[0059] S3: Image element correlation assessment and representative selection;

[0060] S4: Supplementing and summarizing multimodal information related to events.

[0061] An event class (EC) refers to a collection of events that share common characteristics.

[0062] EC = (E, O, A, P, T, S, M)

[0063] Here, E is a set of events, and O, A, P, T, S, and M are called event class elements. They are sets of elements in E that share common characteristics with each event. O (Object): Common characteristics of the objects (including participants, entity events, etc.) involved in this type of event; A (Action): The change process and characteristics of this type of event; P (Location): Common characteristics of the location where all events in the event class occur; T (Time): Common characteristics of the time when all events in the event class occur; S (State): Common pre-state and post-state of all events in the event class; M (Multimodal Information): Common descriptive features of multimodal information of all events in the event class, including text description and visual description. The text description includes a set of trigger word collocations, a set of core words, and the names of each element; the visual description includes representative images, a set of image entities, image scene tags, image entity matching degree, image scene matching degree, and image comprehensive matching degree, etc.

[0064] An event is an event that occurs at a specific time and place, involves several characters, and involves several actions. Formally, an event can be represented as 'e', ​​and defined as a six-tuple:

[0065] e = (o, a, p, t, s, m)

[0066] In this event six-tuple, the elements are called event elements, representing object, action, time, location, state, and multimodal information, respectively. o (object): the object involved in the event (including participants, entities, etc.); a (action): the action that occurs in the event; p (location): the place where the event occurs; t (time): the time (segment) when the event occurs; s (state): the pre- and post-event states of the event; m (multimodal information): a structured set of event modal information, including text description and visual description. The text description includes trigger word collocations, core words, and names of each element; the visual description includes a set of images, a set of image entities, and image scene tags, etc.

[0067] Please see Figure 2In S1, the domain and scope of the event ontology are first determined. Domain-specific text data, sourced from news website reports, is collected based on domain characteristics. The text data is extracted automatically while preserving its source. Secondly, if a reusable existing event ontology pattern exists, it can be used to quickly generate event classes and relationships within the domain, serving as candidate event classes and relationships for the domain event ontology construction. If no reusable ontology pattern exists, the text is segmented, and existing event trigger word extraction models based on attention mechanisms and Bi-GRU are used to extract event types. Then, existing event element extraction models based on BERT and attention mechanisms are used to extract event elements. Finally, Bi-LSTM is used to encode and classify the information of the two events, achieving event relationship extraction and ultimately obtaining candidate event classes and relationships. Finally, domain experts manually screen the obtained events and relationships. The process involves selecting and supplementing data to obtain core event classes and event relationships within the domain. Then, domain experts, following the event six-tuple model and combining event elements, define the main elements of the core event classes, such as the event class's action, object, time, location, preceding state, and following state. Next, by defining the hierarchical relationships between candidate event classes, a domain event class hierarchy is formed. Domain experts then expand the event class hierarchy vertically based on domain common sense, ultimately forming a relatively complete domain event class hierarchy model. Following this, event scenario models are constructed for the core event classes based on domain event classes, event relationships, and domain common sense. These event scenario models consist of event classes and logical relationships (such as sequence, causality, concurrency, etc.). Finally, images are collected based on the source of the event text data, and images are searched using keywords based on the events and their elements. Keywords consist of the event name, object, location, and action, separated by spaces. The final image set is then compiled into a set l = {p1, p2, ..., p...}. n This information is added to the multimodal information of the event and awaits further fusion.

[0068] In S1, Bi-LSTM is used to encode and classify the information of the two events to extract the event relationship, as shown below:

[0069] The event information is input into a Bi-LSTM for encoding, specifically represented as follows:

[0070] Bi-LSTM(X) = x c ={x0,x1,…x n}

[0071] Where X represents the input word vector, which can be vectorized using a Word2Vec pre-trained model. c This represents an intermediate vector with contextual information output by the model.

[0072] Finally, after passing through a fully connected layer with an output dimension of 4 and an output layer using Softmax as the activation function, the classification result is obtained, specifically represented as follows:

[0073]

[0074] Where W is the weight matrix to be learned, and b is the bias term. The probabilities output by softmax represent the classification probabilities of composition, causality, following, and concurrency relationships, respectively.

[0075] In S2, the existing VGG16-Places365 model, pre-trained on the Places365 dataset, is first used to perform scene recognition and annotation on each image of the event, and the top N scene labels P are selected. sce ={s1,s2,…,s N The following steps are taken: First, the existing YOLOv5 model is used to perform object detection and annotation on each image of the event, resulting in N entity labels P. obj ={o1,o2,…,o N This information is added to the multimodal information of the event.

[0076] In S3, firstly, a BERT word vector pre-trained model is used to calculate the vector representation of N scene labels and the location elements of the event class for each image in the event. The cosine similarity between the N scene labels and the event location elements is calculated, and the average similarity is taken as the image scene element matching score Score1 for that image. Secondly, a BERT word vector pre-trained model is used to calculate the vector representation of N image entity labels and text event object elements for each image. The cosine similarity between each object in the event object elements and the N image entity labels is calculated, and the average similarity is taken as the matching score of that object. Finally, the average matching score of all objects in the event object elements is taken as the image object element matching score Score2 for that image. Then, the image scene element matching score and the image object element matching score are combined to calculate the comprehensive element matching score Score of the image. The top M images with the highest scores from all events in each event class are selected as the candidate image set L = {p1, p2, ..., p...} for the event class. M The comprehensive feature matching degree of the images and the candidate image set are added to the multimodal information of the event class; finally, representative image selection technology is used to filter representative images in the candidate image set.

[0077] In S3, the cosine similarity between N scene labels and event location elements is calculated, and the average similarity is taken as the image scene element matching degree of the image, specifically satisfying the following formula:

[0078]

[0079]

[0080] Where, p i Let represent the embedding representation of the i-th scene label, and 'a' represent the embedding representation of the event location element. Score1 represents the similarity between the i-th scene label and the event location element, N represents the number of scene labels, and Score1 represents the matching degree of the image scene elements.

[0081] In S3, the cosine similarity between each object in the event object elements and N image entity labels is calculated. The average similarity is taken as the matching degree of the object. Finally, the average matching degree of all objects in the event object elements is taken as the image object element matching degree of the image, which satisfies the following formula:

[0082]

[0083]

[0084] Among them, o i b represents the embedding representation of the i-th image entity label. j This represents the embedding representation of the j-th object in the event object elements. Score2 represents the similarity between the i-th image entity label and the j-th object in the event object elements, M represents the number of objects in the event object elements, N represents the number of image entity labels, and Score2 represents the matching degree of image entity elements.

[0085] In S3, the overall feature matching degree of the image is calculated by combining the image scene feature matching degree and the image object feature matching degree. The top M images with the highest scores are selected as the candidate image set, specifically satisfying the following formula:

[0086] Score=α×Score1+β×Score2 (α+β=1)

[0087] Where α represents the weight of the matching degree of image scene elements, and β represents the weight of the matching degree of image entity elements.

[0088] S3 employs a representative image selection technique to filter representative images from the candidate image set, as detailed below:

[0089] Extracting SIFT features T from the image i The similarity between images is calculated using cosine similarity, resulting in the image similarity feature matrix H. n×n Image clustering is performed using the AP algorithm based on the feature matrix; the visual feature overlap rate V of each cluster is calculated. j =match_avg(P j), where match_avg(·) represents the average number of SIFT feature matching points between images within a cluster, P j Let P represent the set of image features for the j-th cluster; finally, the cluster center of the first-ranked cluster is used as the representative image P. rep .

[0090] In S4, the candidate image set, representative images, image entity set, image scene label, image scene matching degree, image entity matching degree, and image comprehensive matching degree are supplemented into the visual description of the multimodal information elements of the event class, i.e., M. visual ={L,P rep ,P obj ,P sce ,Score1,Score2,Score}.

[0091] Working principle: First, based on the existing event ontology construction process and automatic text event information extraction technology, a semi-automatic method for constructing multimodal event ontology is proposed. Then, image scene classification and target detection are performed for visual modal fusion of event ontology. Next, image element correlation evaluation and representative selection are carried out. Finally, event-type multimodal information is supplemented and summarized.

[0092] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A semi-automatic method for constructing a multimodal event ontology, characterized in that, Includes the following steps: S1: Based on the existing event ontology construction process and automatic text event information extraction technology, a semi-automatic method for constructing multimodal event ontology is proposed; S2: Image scene classification and target detection based on visual modal fusion of event ontology; S3: Image element correlation assessment and representative selection; S4: Supplementing and summarizing multimodal information related to events; The proposed method for semi-automatically constructing a multimodal event ontology, based on existing event ontology construction processes and automatic text event information extraction techniques, specifically includes: S101: Determine the domain and scope of the event ontology, and collect domain text data based on the characteristics of the domain; S102: Reuse the existing event ontology pattern to quickly generate event classes and event relationships within the domain, which can be used as candidate event classes and event relationships for the construction of the domain event ontology. If there is no reusable ontology pattern, proceed to S103. S103: After segmenting the text, the event type is extracted using the existing event trigger word extraction model based on attention mechanism and Bi-GRU. Then, the event element is extracted using the existing event element extraction model based on BERT and attention mechanism. Finally, Bi-LSTM is used to encode and classify the information of the two events to achieve event relationship extraction, and finally, candidate event classes and event relationships are obtained. S104: Domain experts manually screen and supplement the events and event relationships obtained in steps S102 and S103 to obtain the core event classes and event relationships in the domain. S105: Based on the event six-tuple model and combined with the event elements obtained in S103, define the main elements of the core event class, including the event class's action, object, time, location, preceding state, and following state; S106: By defining the hierarchical relationship between candidate event classes, a domain event class hierarchy is formed. Domain experts expand the event class hierarchy based on domain common sense, and finally form a relatively complete domain event class hierarchy model. S107: Based on the domain event classes and event relationships obtained in step S104, construct event scenario models for the core event classes of the domain based on domain common sense. The event scenario model consists of event classes and event logical relationships, including composition, causality, following, and concurrency. S108: Collect images based on the source of the event text data, and formulate a multi-keyword search for images based on the event and its elements. Keywords consist of event name, object, location, and action, separated by spaces. Finally, the image set is... It is added to the multimodal information of the event, awaiting subsequent fusion.

2. The semi-automatic construction method for multimodal event ontology according to claim 1, characterized in that, The method of using Bi-LSTM to encode and classify the information of two events to extract event relationships is specifically represented as follows: The event information is input into a Bi-LSTM for encoding, specifically represented as follows: Where X represents the input character vector, which is vectorized using the Word2Vec pre-trained model. This represents an intermediate vector with contextual information output by the model. Finally, after passing through a fully connected layer with an output dimension of 4 and an output layer using Softmax as the activation function, the classification result is obtained, specifically represented as follows: Where W is the weight matrix to be learned, and b is the bias term. The probabilities output by softmax represent the classification probabilities of composition, causality, following, and concurrency relationships, respectively.

3. The semi-automatic construction method for multimodal event ontology according to claim 2, characterized in that, The image scene classification and target detection based on event ontology visual modal fusion specifically include: S201: Using the existing VGG16-Places365 model pre-trained on the Places365 dataset, perform scene recognition annotation on each image of the event, and take the top N scene labels with the highest scores. This information is added to the multimodal information of the event. S202: Using the existing YOLOv5 model, perform object detection and annotation on each image of the event to obtain N entity labels. This information is added to the multimodal information of the event.

4. The semi-automatic construction method for multimodal event ontology according to claim 3, characterized in that, The image element correlation assessment and representative selection specifically include: S301: For all events in each event class, use a word vector pre-trained model to calculate the vector representations of N scene labels and the location elements of the event class for each image in the event, and calculate the cosine similarity between the N scene labels and the event location elements. The average similarity is taken as the image scene element matching degree of the image. This information is added to the multimodal information of the event. S302: For all events in each event class, use a word vector pre-training model to calculate the N image entity labels and the vector representation of the object elements of the event class for each image in the event, and calculate the cosine similarity between each object in the event class object elements and the N image entity labels. The average similarity score is taken as the matching degree of the object. Finally, the average matching degree score of all objects in the event object element is taken as the image object element matching degree of the image. This information is added to the multimodal information of the event. S303: Matching degree of image scene elements Matching degree with image object features Calculate the overall feature matching degree of the image. For each event class, select the top M images with the highest scores from all events as the candidate image set for that event class. The comprehensive feature matching degree of the images and the candidate image set are added to the multimodal information of the event; S304: Use representative image selection techniques to select representative images from the candidate image set.

5. The semi-automatic construction method for multimodal event ontology according to claim 4, characterized in that, The calculation of the cosine similarity between N scene labels and event location elements, and the taking of the average similarity as the image scene element matching degree of the image, specifically satisfies the following formula: in, Let i represent the embedding representation of the i-th scene label. Embedded representation of event location elements Let N represent the similarity between the i-th scene label and the event location element, and let N represent the number of scene labels. This indicates the matching degree of image scene elements.

6. The semi-automatic construction method for multimodal event ontology according to claim 5, characterized in that, The calculation of the cosine similarity between each object in the event object element and N image entity labels, taking the average similarity as the matching degree of the object, and finally taking the average matching degree of all objects in the event object element as the image object element matching degree of the image, specifically satisfying the following formula: in, This represents the embedding representation of the i-th image entity label. This represents the embedding representation of the j-th object in the event object elements. Let M represent the similarity between the i-th image entity label and the j-th object in the event object elements, where M represents the number of objects in the event object elements, and N represents the number of image entity labels. This indicates the matching degree of image entity elements.

7. The semi-automatic construction method for multimodal event ontology according to claim 6, characterized in that, The comprehensive element matching degree of the image is calculated by combining the image scene element matching degree and the image object element matching degree, and the top M images with the highest scores are selected as the candidate image set, specifically satisfying the following formula: in, The weights representing the matching degree of image scene elements. The weight representing the matching degree of entity elements in the image.

8. The semi-automatic construction method for multimodal event ontology according to claim 7, characterized in that, The process of using representative image selection techniques to filter representative images from the candidate image set specifically includes: For each image in the candidate image set of the event class, extract the SIFT features of the image. Cosine similarity is used to calculate the similarity between images, resulting in an image similarity feature matrix. Image clustering is performed using the AP algorithm based on the feature matrix; the visual feature overlap rate of each cluster is calculated. ,in This represents the average number of SIFT feature matching points between images within a cluster. Let j represent the set of image features for the j-th cluster; finally, the cluster center of the first-ranked cluster is used as the representative image. .

9. The semi-automatic construction method for multimodal event ontology according to claim 8, characterized in that, The supplementation and summarization of the event-type multimodal information specifically includes: The candidate image set, representative images, image entity set, image scene tags, image scene matching degree, image entity matching degree, and overall image matching degree are supplemented into the visual description of the multimodal information elements of the event category. .

Images