Multi-modal event extraction method and device, equipment, storage medium and program product

Abstract

The invention discloses a multi-modal event extraction method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the feature extraction of text data in multi-modal data, generating a text label sequence based on a text feature extraction result, carrying out the target detection of image data in the multi-modal data, and obtaining a target detection result; obtaining image-text fine-grained information of the multi-modal data based on the text label sequence and the target detection result, generating a text entity set and a visual entity set based on the image-text fine-grained information, performing entity matching on the text entity set and the visual entity set, performing fine-grained image-text alignment on the multi-modal data based on an entity matching result, and obtaining the multi-modal data. According to the method, the multi-modal data are subjected to fine-grained image-text alignment, and the multi-modal data subjected to fine-grained image-text alignment are input into the pre-constructed event extraction model for event extraction, so that effective analysis of global and local semantic information is realized, events can be accurately extracted from the multi-modal data with complex semantic scenes, and missing of potential event values is effectively avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal event extraction method, device, equipment, storage medium and program product. Background Art

[0002] In the era of big data, massive multimodal data (including text, images, audio, video and other modalities) are widely available. In particular, the rise of graphic and text social applications has increased the proportion of multimodal data, providing multimodal information extraction technology with massive data and rich application scenarios. How to efficiently obtain valuable information from multimodal data has attracted widespread attention from researchers and has become a research hotspot of great significance. Events come from cognitive science and are widely used in philosophy, linguistics, computer science and other fields. The Automatic Content Extraction Evaluation Conference defines events as: things or changes in state that occur at a specific time point or time period, within a specific geographical area, and consist of one or more actions participated by one or more roles. The task of event extraction automatically extracts event information of interest to users from unstructured or semi-structured data and expresses it in a structured form, which has far-reaching significance for people's cognition of the world.

[0003] Traditional event extraction research mainly focuses on text data, while multimodal event extraction aims to extract richer event information from multimodal data (such as graphic data) by introducing other modalities such as images as a supplement to text modality. The core of the multimodal event extraction task lies in how to overcome the differences between data of different modalities and establish cross-modal associations. Since data of different modalities usually have relatively independent semantic representations, it is difficult to establish direct connections. Although it is currently possible to extract global semantic information from text and images, the understanding of local semantic information is relatively weak. Since event information is usually more complex and requires comprehensive global and local semantics to understand, it is currently impossible to effectively and accurately extract event information from multimodal data of complex semantic scenes, and therefore it is impossible to deeply explore potential event value information. Summary of the invention

[0004] The main purpose of the present invention is to provide a multimodal event extraction method, device, equipment, storage medium and program product, aiming to solve the problem that the existing technology cannot effectively and accurately extract event information technology from multimodal data in complex semantic scenes.

[0005] To achieve the above object, the present invention provides a multimodal event extraction method, which comprises the following steps: Performing feature extraction on text data in the multimodal data, and generating a text label sequence based on the text feature extraction result, wherein the text label sequence includes a plurality of text labels, and the text labels include text event types of text events in the text data and text entities corresponding to each text event; Performing target detection on the image data in the multimodal data to obtain target detection results, wherein the target detection results include target bounding boxes in the image data and a determination result of whether there are visual entities in each target bounding box; Acquire fine-grained text-image information of the multimodal data based on the text label sequence and the target detection result, and perform entity extraction on the multimodal data based on the fine-grained text-image information to generate a text entity set and a visual entity set; Perform entity matching on the text entity set and the visual entity set to obtain entity matching results, wherein the entity matching results include matching results between text entities and visual entities belonging to the same argument role; Based on the entity matching result, fine-grained image-text alignment is performed on the multimodal data, and the multimodal data after the fine-grained image-text alignment is input into a pre-built event extraction model for event extraction.

[0006] Optionally, the text feature extraction result includes text sequence feature information; the feature extraction of the text data in the multimodal data and generating a text label sequence based on the text feature extraction result includes: Input the text data in the multimodal data into the pre-trained language model for feature extraction to obtain the initial sequence feature information of the text data: in, is the initial sequence feature information, is the length of the text sequence, is the dimension of word vector features, It is the text data in multimodal data; Input the initial sequence feature information into the pre-trained sequence feature extraction model to obtain text sequence feature information: ， in, is the text sequence feature information, is the length of the text sequence, is the size of the hidden layer dimension in the pre-trained sequence feature extraction model; A text label sequence is generated based on the text sequence feature information.

[0007] Optionally, generating a text label sequence based on the text sequence feature information includes: Perform labeling based on the text sequence feature information to obtain multiple text labels; Analyze the dependency relationship between the text labels, and generate a predicted label sequence based on the dependency relationship; Calculate the sequence score of each predicted label sequence by using a score function, wherein the sequence score includes an emission score and a transfer score, wherein the emission score represents the label score of each element in the predicted label sequence, and the transfer score represents the score of transferring from one label to another label; The predicted label sequence is normalized based on the sequence score to obtain sequence probability information: in, represents the predicted label sequence, Indicates the length of the text sequence, represents the set of all possible label sequences, represents a possible tag sequence, represents the score function, is the text sequence feature information, is the text data in multimodal data, represents sequence probability information, which includes the predicted label sequence The credible probability of A text label sequence is determined from the predicted label sequence according to the sequence probability information.

[0008] Optionally, performing target detection on the image data in the multimodal data to obtain a target detection result includes: Performing target positioning on the image data in the multimodal data to obtain entity position information in the image data; annotating a target bounding box in the image data based on the entity location information; The labeled image data is input into the pre-trained target detection model for target detection to obtain the target detection result. The training loss function of the pre-trained target detection model includes: in, represents the training loss function of the pre-trained object detection model, Represents the bounding box loss function, which is used to measure the difference between the predicted bounding box and the true bounding box. Represents the confidence loss function, which is used to measure the confidence that the object is contained in the target bounding box. is a binary classification loss function, which is used to determine whether there is a visual entity in the target bounding box.

[0009] Optionally, performing entity matching on the text entity set and the visual entity set to obtain an entity matching result includes: The text entity set and the visual entity set are respectively input into a pre-trained feature encoding model for feature encoding, to obtain text feature representation information corresponding to the text entity set and visual feature representation information corresponding to the visual entity set: in, The text feature representation information of the text entity set, The visual features of the visual entity set represent information, For the text encoder in the pre-trained feature encoding model, For the visual encoder in the pre-trained feature encoding model, is a set of text entities, is a set of visual entities, The size of the hidden layer dimension in the pre-trained feature encoding model; Calculate the cosine similarity between the text entity set and each entity in the visual entity set based on the text feature representation information and the visual feature representation information: in, Represents information for text features, To represent information for visual features, For text entities With visual entities The cosine similarity between represents the dot product of eigenvectors, represents the magnitude of the eigenvector; Entity matching is performed according to the cosine similarity to obtain an entity matching result.

[0010] Optionally, after performing fine-grained image-text alignment on the multimodal data based on the entity matching result and inputting the fine-grained image-text aligned multimodal data into a pre-built event extraction model for event extraction, the method further includes: Acquire an event extraction result output by the event extraction model, wherein the event extraction result includes a predicted event type and predicted argument information of the multimodal data, wherein the predicted argument information includes an argument role, a text entity, and a visual entity; The performance of the event extraction model is evaluated based on the predicted event type and the prediction argument information to obtain a performance evaluation result, which includes: Among them, TP represents the number of correctly predicted entities, FP represents the number of incorrectly predicted entities, and FN represents the number of actual entities that the event extraction model failed to recognize. Represents the model accuracy, represents the model recall rate, Represents the harmonic mean of model precision and model recall; The event extraction model is optimized according to the performance evaluation result.

[0011] In addition, to achieve the above-mentioned purpose, the present invention also proposes a multimodal event extraction device, the multimodal event extraction device comprising: A text event analysis module, used for performing feature extraction on text data in multimodal data, and generating a text label sequence based on the text feature extraction result, wherein the text label sequence comprises a plurality of text labels, and the text labels comprise text event types of text events in the text data and text entities corresponding to each text event; An image event analysis module, configured to perform target detection on the image data in the multimodal data to obtain a target detection result, wherein the target detection result includes a target bounding box in the image data and a determination result of whether a visual entity exists in each target bounding box; A fine-grained analysis module, used to obtain fine-grained text-graphic information of the multimodal data based on the text label sequence and the target detection result, and to perform entity extraction on the multimodal data based on the fine-grained text-graphic information to generate a text entity set and a visual entity set; An entity matching module, used for performing entity matching between the text entity set and the visual entity set to obtain entity matching results, wherein the entity matching results include matching results between text entities and visual entities belonging to the same argument role; The event extraction module is used to perform fine-grained image-text alignment on the multimodal data based on the entity matching result, and input the multimodal data after the fine-grained image-text alignment into a pre-built event extraction model for event extraction.

[0012] In addition, to achieve the above-mentioned objectives, the present application also proposes a multimodal event extraction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multimodal event extraction method described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal event extraction method described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multimodal event extraction method as described above.

[0015] The present invention performs feature extraction on text data in multimodal data, and generates a text label sequence based on the text feature extraction result, wherein the text label sequence includes multiple text labels, and the text labels include text event types of text events in the text data and text entities corresponding to each text event; performs target detection on image data in the multimodal data to obtain target detection results, wherein the target detection results include target bounding boxes in the image data and judgment results of whether there are visual entities in each target bounding box; obtains fine-grained image and text information of the multimodal data based on the text label sequence and the target detection results; performs entity extraction on the multimodal data based on the fine-grained image and text information to generate a text entity set and a visual entity set; and compares the text entity set with the visual entity set. Entity matching, obtaining entity matching results, the entity matching results including matching results between text entities and visual entities belonging to the same argument role, performing fine-grained image-text alignment on the multimodal data based on the entity matching results, and inputting the multimodal data after fine-grained image-text alignment into a pre-built event extraction model for event extraction; because the present invention obtains fine-grained image-text information of multimodal data, extracts text entity sets and visual entity sets respectively based on the fine-grained image-text information, and implements fine-grained image-text alignment based on entity matching, realizes effective analysis of global and local semantic information, and effectively improves the efficiency of multimodal event extraction, so that events can be accurately extracted from multimodal data with complex semantic scenes, effectively avoiding missing potential event values, and accurately extracting event information from multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 It is a structural diagram of a multimodal event extraction device of a hardware operating environment involved in an embodiment of the present invention; Figure 2 It is a flowchart of the first embodiment of the multimodal event extraction method of the present invention; Figure 3 A schematic diagram of a process for extracting text sequence feature information in an embodiment of a multimodal event extraction method of the present invention; Figure 4 A schematic diagram of target detection in an embodiment of a multimodal event extraction method of the present invention; Figure 5 It is a flow chart of entity matching in an embodiment of a multimodal event extraction method of the present invention; Figure 6 It is a schematic diagram of the process of model optimization in an embodiment of the multimodal event extraction method of the present invention; Figure 7 It is a structural block diagram of the first embodiment of the multimodal event extraction device of the present invention.

[0018] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0020] Reference Figure 1 , Figure 1 It is a schematic diagram of the structure of a multimodal event extraction device in a hardware operating environment involved in an embodiment of the present invention.

[0021] like Figure 1 As shown, the multimodal event extraction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (RandomAccess Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0022] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the multimodal event extraction device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0023] like Figure 1 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a multimodal event extraction program.

[0024] exist Figure 1 In the multimodal event extraction device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the multimodal event extraction device of the present invention can be set in the multimodal event extraction device, and the multimodal event extraction device calls the multimodal event extraction program stored in the memory 1005 through the processor 1001, and executes the multimodal event extraction method provided by the embodiment of the present invention.

[0025] The embodiment of the present invention provides a multimodal event extraction method, referring to Figure 2 , Figure 2 It is a flowchart of the first embodiment of the multimodal event extraction method of the present invention.

[0026] In this embodiment, the multimodal event extraction method includes the following steps: Step S10: extracting features from text data in the multimodal data, and generating a text label sequence based on the text feature extraction results.

[0027] It should be noted that the core of the multimodal event extraction task lies in how to overcome the differences between data of different modalities and establish cross-modal associations. Since data of different modalities usually have relatively independent semantic representations, it is difficult to establish connections directly. With the advancement of multimodal pre-training technology, multimodal pre-training models represented by the CLIP (Contrastive Language-Image Pre-training) model have demonstrated powerful image-text alignment capabilities, and can model text and image data in a unified semantic space to achieve cross-modal information processing. However, the CLIP model still has shortcomings in understanding fine-grained concepts in complex scenes. Specifically, the CLIP model is good at extracting global semantic information of text and images, but its understanding of local semantic information is relatively weak. Since event information is usually more complex and requires comprehensive global and local semantics to understand, it makes it difficult to directly apply the CLIP model to complex semantic understanding scenarios such as multimodal event extraction.

[0028] To this end, this embodiment extracts multimodal events based on fine-grained image-text alignment, including two stages: unimodal information extraction and multimodal information fusion. Unimodal information extraction first uses text event extraction and visual entity extraction to extract fine-grained event information from images and texts respectively, and then performs fine-grained image-text alignment based on the CLIP model to achieve multimodal information fusion, and finally obtains multimodal event information.

[0029] It should be understood that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a computer, etc., or a terminal electronic device capable of realizing the above functions, etc. The following takes a multimodal event extraction device (referred to as the extraction device) as an example to illustrate this embodiment and the following embodiments.

[0030] It should be noted that the text tag sequence includes a plurality of text tags, and the text tags include text event types of text events in the text data and text entities corresponding to each text event.

[0031] It should be noted that the multimodal event extraction task aims to extract structured multimodal event information from unstructured text and image data. Specifically, for a given text T and its corresponding image I, the task needs to identify all events e and their corresponding arguments a contained therein. Each event e consists of an event type And multiple arguments a. Event type It is used to indicate the type of event, such as configuration event, detection event, etc. It is usually predefined in the data set. A text may contain multiple events at the same time. Argument a is used to indicate the entity involved in the event. It is usually composed of argument roles. , text entities t and visual entities If there is no corresponding visual entity in the image, the argument consists of only two parts: the argument role and the text entity.

[0032] In some embodiments, the extraction device can improve the BIO labeling strategy to expand the original unigram tags into binary tags so that each tag can simultaneously represent the event type and argument role, thereby simultaneously extracting events and arguments in the text in the sequence labeling task.

[0033] Further, refer to Figure 3 , Figure 3 This is a flow chart of extracting text sequence feature information in an embodiment of the present invention. In order to accurately extract information from text data, the text feature extraction result includes text sequence feature information. The above step S10 may include: Step S101: inputting text data in the multimodal data into a pre-trained language model for feature extraction to obtain initial sequence feature information of the text data; Step S102: inputting the initial sequence feature information into a pre-trained sequence feature extraction model to obtain text sequence feature information; Step S103: Generate a text label sequence based on the text sequence feature information.

[0034] In some embodiments, the extraction device may use a BERT-LSTM-CRF model for sequence tagging. , first use the pre-trained language model BERT to extract the initial sequence feature information of the text data: in, is the initial sequence feature information, is the length of the text sequence, is the dimension of BERT word vector features, It is the text data in multimodal data.

[0035] In some embodiments, the pre-trained sequence feature extraction model may be an LSTM model. The extraction device may input the initial sequence feature information extracted by the BERT model into the LSTM model, and utilize the sequence modeling capability of the LSTM to further enhance the contextual association information in the text sequence features, thereby improving the performance of the model for complex sequence labeling tasks and obtaining text sequence feature information: ， in, is the text sequence feature information, is the length of the text sequence, is the size of the hidden layer dimension in the pre-trained sequence feature extraction model.

[0036] Furthermore, for the accuracy of text sequence annotation, the above step S103 may include: Step S1031: labeling based on the text sequence feature information to obtain multiple text labels; Step S1032: analyzing the dependency relationship between the text labels, and generating a predicted label sequence based on the dependency relationship; Step S1033: Calculate the sequence score of each predicted label sequence using a scoring function; Step S1034: normalizing the predicted label sequence based on the sequence score to obtain sequence probability information; Step S1035: Determine a text label sequence from the predicted label sequence according to the sequence probability information.

[0037] It should be noted that the sequence score includes an emission score and a transfer score, wherein the emission score represents the label score of each element in the predicted label sequence, and the transfer score represents the score of transferring from one label to another label.

[0038] In some embodiments, the extraction device inputs the text sequence features into the CRF layer to obtain the BIO tag sequence, and decodes it to extract the events in the text and their corresponding arguments: in, represents the predicted label sequence, Indicates the length of the text sequence, represents the set of all possible label sequences, represents a possible tag sequence, represents the score function, is the text sequence feature information, is the text data in multimodal data, represents sequence probability information, which includes the predicted label sequence credible probability.

[0039] It should be noted that the above score function is used to calculate the score of the predicted label sequence, which consists of two parts: the emission score and the transfer score. Among them, the emission score refers to the label score of each element in the sequence, which is calculated by the probability distribution of the output label. The transfer score refers to the score transferred from one label to another. The CRF layer can effectively learn the constraint relationship between labels and improve the accuracy of sequence labeling tasks.

[0040] In some embodiments, the loss function in the training process of the BERT-LSTM-CRF model is a negative log-likelihood function, referring to the following formula: in, is the actual label sequence.

[0041] Step S20: performing target detection on the image data in the multimodal data to obtain a target detection result.

[0042] It should be noted that in the multimodal event extraction task, due to the wide variety of argument entities involved in the event, the entity types in the image are often difficult to fully define in advance. Traditional target detection methods usually require pre-definition of the target entity category and build a data set based on the category to train the model. This method limits the entity categories detected and is difficult to use for multimodal event extraction tasks. To this end, this embodiment can improve the traditional target detection model from the perspective of training strategy to make it a target detection model that does not rely on predefined categories. The improved target detection model can be trained with only bounding box annotations, and since there is no restriction on classification tasks, it also has the ability to detect new targets outside the training set.

[0043] It should be noted that the above target detection results include target bounding boxes in the image data and the determination result of whether there is a visual entity in each target bounding box.

[0044] In some embodiments, the extraction device may perform target detection on the image data through a target detection model (such as a YOLO model). Target detection may include target positioning and target classification. Target positioning is used to mark target bounding boxes for entities in the image data. Target classification is used to determine whether there are visual entities in each target bounding box to obtain target detection results. This eliminates the need for specific target classification, effectively improving target detection efficiency and ensuring detection accuracy.

[0045] Further, refer to Figure 4 , Figure 4 This is a schematic diagram of target detection in an embodiment of the present invention. In order to accurately detect visual entities in image data, the above step S20 may include: Step S201: performing target positioning on the image data in the multimodal data to obtain entity position information in the image data; Step S202: marking a target bounding box in the image data based on the entity position information; Step S203: input the labeled image data into the pre-trained target detection model to perform target detection and obtain target detection results.

[0046] In some embodiments, the extraction device can improve the YOLOv5 model from the perspective of training strategy, making it an object detection model that does not rely on predefined categories. The model can be trained with only bounding box annotations, and because it is not restricted by classification tasks, it also has the ability to detect new objects outside the training set.

[0047] It should be noted that the training of the pre-trained target detection model includes two main tasks: the localization task is responsible for obtaining the location of the target entity in the image, and the classification task is responsible for identifying the category of the target entity. This paper mainly improves the classification task, uniformly marking all target entity categories as one category, thereby transforming the original multi-category classification problem into a binary classification problem, that is, judging whether there is a target entity in the bounding box. The training loss of the model consists of three parts. The training loss function of the pre-trained target detection model includes: in, represents the training loss function of the pre-trained object detection model, Represents the bounding box loss function, which is used to measure the difference between the predicted bounding box and the true bounding box. Represents the confidence loss function, which is used to measure the confidence that the object is contained in the target bounding box. is a binary classification loss function, which is used to determine whether there is a visual entity in the target bounding box.

[0048] Step S30: obtaining fine-grained text-graphic information of the multimodal data based on the text label sequence and the target detection result, and performing entity extraction on the multimodal data based on the fine-grained text-graphic information to generate a text entity set and a visual entity set.

[0049] It should be noted that this embodiment can overcome the shortcomings of the CLIP model in fine-grained concept understanding based on text event extraction and visual entity extraction, and achieve more accurate multimodal information fusion. and image data First, a series of text entities and visual entities are obtained through text event extraction and visual entity extraction, thereby obtaining a text entity set and visual entity collection .

[0050] In some embodiments, for the text event extraction model, the extraction device may use the RoBERTa model as the text encoder. The optimizer uses AdamW, and the learning rate is set to 2e-5. To enhance the learning ability of the CRF layer, a larger differential learning rate of 2e-3 is used. To avoid overfitting, an early stopping strategy is used to control the training of the model. When the performance of the validation set does not improve within 10 rounds, the training will stop and the model with the best performance on the validation set will be selected as the final model. The upper limit of the number of training rounds is 100 rounds, the batch size is 32, and the Dropout layer is 0.1. To ensure that all texts are covered, the maximum length of the input text is set to 256.

[0051] In some embodiments, for the visual entity extraction model, this paper uses YOLOv5m as the target detector, the size of the input image is 640, the optimizer uses SGD, the learning rate is set to 1e-2, the batch size is 32, and the model with the best performance on the validation set is selected as the final model.

[0052] Step S40: performing entity matching on the text entity set and the visual entity set to obtain an entity matching result.

[0053] It can be understood that this embodiment can extract features from each text entity in the text entity set and each visual entity in the visual entity set, obtain the text feature vector of each text entity and the visual feature vector of each visual entity, perform entity matching based on the text feature vector and the visual feature vector, and obtain entity matching results.

[0054] Further, refer to Figure 5 , Figure 5 This is a schematic diagram of an entity matching process in an embodiment of the present invention. In order to accurately match the text entity with the visual entity, the above step S40 may include: Step S401: inputting the text entity set and the visual entity set into a pre-trained feature coding model for feature coding, respectively, to obtain text feature representation information corresponding to the text entity set and visual feature representation information corresponding to the visual entity set; Step S402: Calculating the cosine similarity between the text entity set and each entity in the visual entity set based on the text feature representation information and the visual feature representation information; Step S403: performing entity matching according to the cosine similarity to obtain an entity matching result.

[0055] In some embodiments, the pre-trained feature encoding model may be a CLIP model, and the extraction device inputs the extracted text entities and visual entities into the text encoder and visual encoder of CLIP respectively to obtain corresponding feature representations: in, The text feature representation information of the text entity set, The visual features of the visual entity set represent information, For the text encoder in the pre-trained feature encoding model, For the visual encoder in the pre-trained feature encoding model, is a set of text entities, is a set of visual entities, is the size of the hidden layer dimension in the pre-trained feature encoding model, represents the magnitude of the eigenvector; represents the eigenvector dot product.

[0056] In some embodiments, the extraction device may match text entities and visual entities pointing to the same argument role by calculating the cosine similarity between the text entity and the visual entity features. The cosine similarity is calculated according to the following formula: in, Represents information for text features, To represent information for visual features, For text entities With visual entities The cosine similarity between represents the dot product of eigenvectors, Represents the magnitude of the eigenvector.

[0057] At the same time, in order to maximize the image-text alignment capability of the CLIP model, the extraction device in some embodiments may also use a hint integration technology to enhance the robustness of the reasoning result. Specifically, for each text entity, a variety of hint templates are used for matching to obtain multiple visual entity matching results. Finally, the best matching result is determined through a voting mechanism.

[0058] Step S50: performing fine-grained image-text alignment on the multimodal data based on the entity matching result, and inputting the fine-grained image-text aligned multimodal data into a pre-built event extraction model for event extraction.

[0059] It should be noted that the event extraction model can be a CLIP model, which is good at extracting global semantic information of text and images, but has a weak understanding of local semantic information. Since event information is usually complex and requires comprehensive global and local semantics for understanding, it is difficult to directly apply the CLIP model to complex semantic understanding scenarios such as multimodal event extraction. Therefore, this embodiment extracts text data and image data from multimodal data in a single mode, then performs multimodal fusion, performs fine-grained text-image alignment on the multimodal data, and inputs the multimodal data after fine-grained text-image alignment into the event extraction model for event extraction, thereby effectively solving the problem that the CLIP model cannot understand the semantics of complex events, improving the accuracy of event extraction, and effectively mining potential value events.

[0060] It can be understood that this implementation extracts unimodal information from text data and image data in multimodal data respectively, thereby obtaining text entities in text data and visual entities in image data, and then aligns the image and text based on fine-grained information, and performs multimodal fusion of text entities and visual entities, thereby accurately extracting event information from multimodal data.

[0061] In some embodiments, the extraction device can jointly extract event information from the text data and image data of the multimodal data by analyzing multimodal data, model the event information in the text and image using abstract semantic representation and visual scene graph respectively, and then use the text and image to perform weakly supervised alignment of the data to achieve joint extraction of text and image event information, thereby extracting richer event information and deeply exploring the potential event value in the text and image data.

[0062] In some embodiments, the extraction device may use the Taiyi model as a multimodal pre-training model for multimodal information fusion. The model is based on the CLIP structure and is trained on a Chinese data set to adapt to Chinese characteristics.

[0063] Further, refer to Figure 6 , Figure 6 This is a flow chart of model optimization in one embodiment of the present invention. In order to effectively improve the performance of the event extraction model, after the above step S50, the following may be included: Step S501: obtaining the event extraction result output by the event extraction model; Step S502: performing a performance evaluation on the event extraction model based on the predicted event type and the prediction argument information to obtain a performance evaluation; Step S503: Optimizing the event extraction model according to the performance evaluation result.

[0064] It should be noted that the event extraction result includes the predicted event type and predicted argument information of the multimodal data, and the predicted argument information includes argument roles, text entities and visual entities.

[0065] It should be noted that this paper uses precision (P), recall (R) and F1 value (F1) as the main indicators to evaluate model performance. When the predicted event type, argument role, text entity and visual entity are consistent with the annotation, they are considered correct. In some embodiments, the visual argument consistency needs to meet the intersection over union (IoU) which can be set to be greater than 0.8. The performance evaluation results include: Among them, TP represents the number of correctly predicted entities, FP represents the number of incorrectly predicted entities, and FN represents the number of actual entities that the event extraction model failed to recognize. Represents the model accuracy, represents the model recall rate, Represents the harmonic mean of model precision and model recall.

[0066] In some embodiments, the extraction device may use an event element extraction dataset of open source multimodal data to evaluate and optimize the event extraction model. For example, the dataset covers a variety of different event types: detection, maintenance, protection, configuration, and mobile events, as well as a variety of argument roles: initiator, recipient, configuration equipment, time, and location. Each piece of data consists of a piece of news text in the field to which the data to be analyzed belongs and its corresponding picture. The dataset includes a training set, a validation set, and a test set. For example, the training set contains 1,136 pieces of data, the validation set contains 264 pieces of data, and the test set contains 200 pieces of data.

[0067] This embodiment performs feature extraction on text data in multimodal data, and generates a text label sequence based on the text feature extraction result, wherein the text label sequence includes multiple text labels, and the text labels include the text event type of the text event in the text data and the text entity corresponding to each text event. Target detection is performed on image data in the multimodal data to obtain a target detection result, wherein the target detection result includes a target bounding box in the image data and a judgment result of whether there is a visual entity in each target bounding box. Fine-grained image and text information of the multimodal data is obtained based on the text label sequence and the target detection result, and entity extraction is performed on the multimodal data based on the fine-grained image and text information to generate a text entity set and a visual entity set, and the text entity set and the visual entity set are compared. Entity matching, obtaining entity matching results, the entity matching results including matching results between text entities and visual entities belonging to the same argument role, performing fine-grained image-text alignment on the multimodal data based on the entity matching results, and inputting the multimodal data after fine-grained image-text alignment into a pre-built event extraction model for event extraction; since this embodiment obtains fine-grained image-text information of multimodal data, extracts text entity sets and visual entity sets respectively based on the fine-grained image-text information, and implements fine-grained image-text alignment based on entity matching, it realizes effective analysis of global and local semantic information, effectively improves the efficiency of multimodal event extraction, and can accurately extract events from multimodal data with complex semantic scenes, effectively avoid missing potential event values, and accurately extract event information from multimodal data.

[0068] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a multimodal event extraction program is stored. When the multimodal event extraction program is executed by a processor, the steps of the multimodal event extraction method described above are implemented.

[0069] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0070] The computer-readable storage medium mentioned above may be included in the multimodal event extraction device; or may exist independently without being assembled into the multimodal event extraction device.

[0071] In addition, an embodiment of the present invention further provides a computer program product, including a multimodal event extraction program, which implements the steps of the multimodal event extraction method as described above when executed by a processor.

[0072] The specific implementation of the computer program product of the present invention is basically the same as the various embodiments of the above-mentioned multimodal event extraction method, and will not be repeated here.

[0073] Reference Figure 7 , Figure 7 It is a structural block diagram of the first embodiment of the multimodal event extraction device of the present invention.

[0074] like Figure 7 As shown, the multimodal event extraction device proposed in the embodiment of the present invention includes: A text event analysis module 10, for performing feature extraction on text data in multimodal data, and generating a text label sequence based on the text feature extraction result, wherein the text label sequence comprises a plurality of text labels, and the text labels comprise text event types of text events in the text data and text entities corresponding to each text event; An image event analysis module 20, configured to perform target detection on the image data in the multimodal data to obtain a target detection result, wherein the target detection result includes a target bounding box in the image data and a determination result of whether there is a visual entity in each target bounding box; A fine-grained analysis module 30, configured to obtain fine-grained text-graphic information of the multimodal data based on the text label sequence and the target detection result, and to perform entity extraction on the multimodal data based on the fine-grained text-graphic information to generate a text entity set and a visual entity set; An entity matching module 40 is used to perform entity matching on the text entity set and the visual entity set to obtain entity matching results, wherein the entity matching results include matching results between text entities and visual entities belonging to the same argument role; The event extraction module 50 is used to perform fine-grained image-text alignment on the multimodal data based on the entity matching result, and input the fine-grained image-text aligned multimodal data into a pre-built event extraction model for event extraction.

[0075] Furthermore, the text feature extraction result includes text sequence feature information; the text event analysis module 10 is further used to input the text data in the multimodal data into the pre-trained language model for feature extraction, and obtain the initial sequence feature information of the text data: in, is the initial sequence feature information, is the length of the text sequence, is the dimension of word vector features, It is the text data in multimodal data; Input the initial sequence feature information into the pre-trained sequence feature extraction model to obtain text sequence feature information: ， in, is the text sequence feature information, is the length of the text sequence, is the size of the hidden layer dimension in the pre-trained sequence feature extraction model; A text label sequence is generated based on the text sequence feature information.

[0076] Furthermore, the text event analysis module 10 is also used to perform labeling based on the text sequence feature information to obtain multiple text labels; Analyze the dependency relationship between the text labels, and generate a predicted label sequence based on the dependency relationship; Calculate the sequence score of each predicted label sequence by using a score function, wherein the sequence score includes an emission score and a transfer score, wherein the emission score represents the label score of each element in the predicted label sequence, and the transfer score represents the score of transferring from one label to another label; The predicted label sequence is normalized based on the sequence score to obtain sequence probability information: in, represents the predicted label sequence, Indicates the length of the text sequence, represents the set of all possible label sequences, represents a possible tag sequence, represents the score function, is the text sequence feature information, is the text data in multimodal data, represents sequence probability information, which includes the predicted label sequence The credible probability of A text label sequence is determined from the predicted label sequence according to the sequence probability information.

[0077] Furthermore, the image event analysis module 20 is also used to perform target positioning on the image data in the multimodal data to obtain entity position information in the image data; annotating a target bounding box in the image data based on the entity location information; The labeled image data is input into the pre-trained target detection model for target detection to obtain the target detection result. The training loss function of the pre-trained target detection model includes: in, represents the training loss function of the pre-trained object detection model, Represents the bounding box loss function, which is used to measure the difference between the predicted bounding box and the true bounding box. Represents the confidence loss function, which is used to measure the confidence that the object is contained in the target bounding box. is a binary classification loss function, which is used to determine whether there is a visual entity in the target bounding box.

[0078] Furthermore, the entity matching module 40 is further used to input the text entity set and the visual entity set into a pre-trained feature encoding model for feature encoding, respectively, to obtain text feature representation information corresponding to the text entity set and visual feature representation information corresponding to the visual entity set: in, The text feature representation information of the text entity set, The visual features of the visual entity set represent information, The size of the hidden layer dimension in the pre-trained feature encoding model; Calculate the cosine similarity between the text entity set and each entity in the visual entity set based on the text feature representation information and the visual feature representation information: in, Represents information for text features, To represent information for visual features, For text entities With visual entities The cosine similarity between represents the dot product of eigenvectors, represents the magnitude of the eigenvector; Entity matching is performed according to the cosine similarity to obtain an entity matching result.

[0079] Furthermore, the event extraction module 50 is further used to obtain the event extraction result output by the event extraction model, wherein the event extraction result includes the predicted event type and predicted argument information of the multimodal data, and the predicted argument information includes argument roles, text entities, and visual entities; The performance of the event extraction model is evaluated based on the predicted event type and the prediction argument information to obtain a performance evaluation result, which includes: Among them, TP represents the number of correctly predicted entities, FP represents the number of incorrectly predicted entities, and FN represents the number of actual entities that the event extraction model failed to recognize. Represents the model accuracy, represents the model recall rate, Represents the harmonic mean of model precision and model recall; The event extraction model is optimized according to the performance evaluation result.

[0080] This embodiment performs feature extraction on text data in multimodal data, and generates a text label sequence based on the text feature extraction result, wherein the text label sequence includes multiple text labels, and the text labels include the text event type of the text event in the text data and the text entity corresponding to each text event. Target detection is performed on image data in the multimodal data to obtain a target detection result, wherein the target detection result includes a target bounding box in the image data and a judgment result of whether there is a visual entity in each target bounding box. Fine-grained image and text information of the multimodal data is obtained based on the text label sequence and the target detection result, and entity extraction is performed on the multimodal data based on the fine-grained image and text information to generate a text entity set and a visual entity set, and the text entity set and the visual entity set are compared. Entity matching, obtaining entity matching results, the entity matching results including matching results between text entities and visual entities belonging to the same argument role, performing fine-grained image-text alignment on the multimodal data based on the entity matching results, and inputting the multimodal data after fine-grained image-text alignment into a pre-built event extraction model for event extraction; since this embodiment obtains fine-grained image-text information of multimodal data, extracts text entity sets and visual entity sets respectively based on the fine-grained image-text information, and implements fine-grained image-text alignment based on entity matching, it realizes effective analysis of global and local semantic information, effectively improves the efficiency of multimodal event extraction, and can accurately extract events from multimodal data with complex semantic scenes, effectively avoid missing potential event values, and accurately extract event information from multimodal data.

[0081] The multimodal event extraction device provided by the present application adopts the multimodal event extraction method in the above-mentioned embodiment, which can solve the technical problems of multimodal event extraction. Compared with the prior art, the beneficial effects of the multimodal event extraction device provided by the present application are the same as the beneficial effects of the multimodal event extraction method provided by the above-mentioned embodiment, and the other technical features in the multimodal event extraction device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0082] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0083] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0084] In addition, for technical details not fully described in this embodiment, reference may be made to the multimodal event extraction method provided in any embodiment of the present invention, and will not be repeated here.

[0085] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0086] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0088] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal event extraction method, characterized in that: The multimodal event extraction method comprises: Performing feature extraction on text data in the multimodal data, and generating a text label sequence based on the text feature extraction result, wherein the text label sequence includes a plurality of text labels, and the text labels include text event types of text events in the text data and text entities corresponding to each text event; Performing target detection on the image data in the multimodal data to obtain a target detection result, wherein the target detection result includes a target bounding box in the image data and a determination result of whether a visual entity exists in each target bounding box; Acquire fine-grained text-image information of the multimodal data based on the text label sequence and the target detection result, and perform entity extraction on the multimodal data based on the fine-grained text-image information to generate a text entity set and a visual entity set; Perform entity matching on the text entity set and the visual entity set to obtain entity matching results, wherein the entity matching results include matching results between text entities and visual entities belonging to the same argument role; Based on the entity matching result, fine-grained image-text alignment is performed on the multimodal data, and the multimodal data after the fine-grained image-text alignment is input into a pre-built event extraction model for event extraction.

2. The multimodal event extraction method according to claim 1, characterized in that: The text feature extraction result includes text sequence feature information; the feature extraction of text data in the multimodal data and generating a text label sequence based on the text feature extraction result includes: Input the text data in the multimodal data into the pre-trained language model for feature extraction to obtain the initial sequence feature information of the text data: in, is the initial sequence feature information, is the length of the text sequence, is the dimension of the word vector feature, It is the text data in multimodal data; Input the initial sequence feature information into the pre-trained sequence feature extraction model to obtain text sequence feature information: ， in, is the text sequence feature information, is the length of the text sequence, is the size of the hidden layer dimension in the pre-trained sequence feature extraction model; A text label sequence is generated based on the text sequence feature information.

3. The multimodal event extraction method according to claim 2, characterized in that: The generating of a text label sequence based on the text sequence feature information includes: Perform labeling based on the text sequence feature information to obtain multiple text labels; Analyze the dependency relationship between the text labels, and generate a predicted label sequence based on the dependency relationship; Calculate the sequence score of each predicted label sequence by using a score function, wherein the sequence score includes an emission score and a transfer score, wherein the emission score represents the label score of each element in the predicted label sequence, and the transfer score represents the score of transferring from one label to another label; The predicted label sequence is normalized based on the sequence score to obtain sequence probability information: in, represents the predicted label sequence, Indicates the length of the text sequence, represents the set of all possible label sequences, represents a possible tag sequence, represents the score function, is the text sequence feature information, is the text data in multimodal data, represents sequence probability information, which includes the predicted label sequence The credible probability of A text label sequence is determined from the predicted label sequence according to the sequence probability information.

4. The multimodal event extraction method according to claim 1, characterized in that: The performing target detection on the image data in the multimodal data to obtain the target detection result includes: Performing target positioning on the image data in the multimodal data to obtain entity position information in the image data; annotating a target bounding box in the image data based on the entity location information; The labeled image data is input into the pre-trained target detection model for target detection to obtain the target detection result. The training loss function of the pre-trained target detection model includes: in, represents the training loss function of the pre-trained object detection model, Represents the bounding box loss function, which is used to measure the difference between the predicted bounding box and the true bounding box. Represents the confidence loss function, which is used to measure the confidence that the object is contained in the target bounding box. is a binary classification loss function, which is used to determine whether there is a visual entity in the target bounding box.

5. The multimodal event extraction method according to claim 1, characterized in that: The performing entity matching on the text entity set and the visual entity set to obtain an entity matching result includes: The text entity set and the visual entity set are respectively input into a pre-trained feature encoding model for feature encoding, to obtain text feature representation information corresponding to the text entity set and visual feature representation information corresponding to the visual entity set: in, The text feature representation information of the text entity set, The visual features of the visual entity set represent information, For the text encoder in the pre-trained feature encoding model, For the visual encoder in the pre-trained feature encoding model, is a set of text entities, is a set of visual entities, The size of the hidden layer dimension in the pre-trained feature encoding model; Calculate the cosine similarity between the text entity set and each entity in the visual entity set based on the text feature representation information and the visual feature representation information: in, Represents information for text features, To represent information for visual features, For text entities With visual entities The cosine similarity between represents the dot product of eigenvectors, represents the magnitude of the eigenvector; Entity matching is performed according to the cosine similarity to obtain an entity matching result.

6. The multimodal event extraction method according to any one of claims 1 to 5, characterized in that: After fine-grained image-text alignment is performed on the multimodal data based on the entity matching result, and the multimodal data after fine-grained image-text alignment is input into a pre-built event extraction model for event extraction, the method further includes: Acquire an event extraction result output by the event extraction model, wherein the event extraction result includes a predicted event type and predicted argument information of the multimodal data, wherein the predicted argument information includes an argument role, a text entity, and a visual entity; The performance of the event extraction model is evaluated based on the predicted event type and the prediction argument information to obtain a performance evaluation result, which includes: Among them, TP represents the number of correctly predicted entities, FP represents the number of incorrectly predicted entities, and FN represents the number of actual entities that the event extraction model failed to recognize. Represents the model accuracy, represents the model recall rate, Represents the harmonic mean of model precision and model recall; The event extraction model is optimized according to the performance evaluation result.

7. A multimodal event extraction device, characterized in that: The multimodal event extraction device comprises: A text event analysis module, used for performing feature extraction on text data in multimodal data, and generating a text label sequence based on the text feature extraction result, wherein the text label sequence comprises a plurality of text labels, and the text labels comprise text event types of text events in the text data and text entities corresponding to each text event; An image event analysis module, configured to perform target detection on the image data in the multimodal data to obtain a target detection result, wherein the target detection result includes a target bounding box in the image data and a determination result of whether a visual entity exists in each target bounding box; A fine-grained analysis module, used to obtain fine-grained text-graphic information of the multimodal data based on the text label sequence and the target detection result, and to perform entity extraction on the multimodal data based on the fine-grained text-graphic information to generate a text entity set and a visual entity set; An entity matching module, used for performing entity matching between the text entity set and the visual entity set to obtain entity matching results, wherein the entity matching results include matching results between text entities and visual entities belonging to the same argument role; The event extraction module is used to perform fine-grained image-text alignment on the multimodal data based on the entity matching result, and input the multimodal data after the fine-grained image-text alignment into a pre-built event extraction model for event extraction.

8. A multimodal event extraction device, characterized in that: The multimodal event extraction device includes: a memory, a processor, and a multimodal event extraction program stored in the memory and executable on the processor, wherein the multimodal event extraction program is configured to implement the multimodal event extraction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal event extraction program, and when the multimodal event extraction program is executed by a processor, the multimodal event extraction method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a multimodal event extraction program, and when the multimodal event extraction program is executed by a processor, the steps of the multimodal event extraction method as claimed in any one of claims 1 to 6 are implemented.

Images