A method and system for joint extraction of cross-modal events based on link matrix

Abstract

This invention discloses a multimodal event joint extraction method based on a link matrix. It determines the event structure system of the application domain and constructs a cross-modal event extraction dataset. A link matrix between image patches and text sequences is constructed. A multimodal event extraction model is built based on ViLT, where the multimodal Transformer encoder takes image patch pixel features, word instance features, and image patch-text word instance pair features as input to obtain a multimodal feature input decoder, acquiring image-text event type consistency features, event type features, and image-text event element features. Using maximum likelihood estimation, the image-text event type matching result with the highest probability, the event type, and the image-text event element are taken as an index, and the text event information is obtained by index mapping. This invention solves the problems of low recognition accuracy in weakly supervised and unsupervised methods and inaccurate matching of pipeline-style multimodal event element information, providing a new event extraction strategy for the field of multimodal information extraction.

Description

Technical Field

[0001] This invention relates to computer multimodal processing, and more specifically to a method and system for joint extraction of cross-modal events based on a link matrix. Background Technology

[0002] Event extraction tasks aim to discover event types and event roles in raw data. Based on the data-oriented modality, they can be divided into text event extraction and visual event extraction.

[0003] Text event extraction is an important research direction in the field of information extraction in Natural Language Processing (NLP), and event extraction algorithms for specific and open domains are becoming increasingly mature. Current methods employ basic networks such as RNNs (Recurrent Neural Networks), CNNs (Convolutional Neural Networks), GNNs (Graph Neural Networks), joint extraction, and transformers (a type of neural network model) for sentence-level and document-level text event extraction. Cue learning and other methods are also used to improve event extraction performance. However, when these methods are applied to news events, while the influence of textual contextual semantics on event structure extraction is considered, the information latent in visual semantic features is often overlooked.

[0004] Visual event recognition, defined in the field of computer vision (CV) as a visual scene recognition task (SR), involves predicting the type and roles of events in an image. However, unlike text-based modalities, most research has not focused on locating event elements within the input image. While some work has begun to address the problem of locating event elements in images, it has primarily focused on two roles: event subjects and participants. [6] However, it ignores the characteristic structure of other elements in visual events.

[0005] Multimodal event extraction is a completely new research field in both academia and industry. Current multimodal event extraction techniques employ methods such as weakly supervised learning, self-supervised learning, and large-scale pre-trained language models. These methods build models with few or no samples. However, the lack of supervision signals leads to insufficient recognition accuracy. At the same time, the pipeline extraction of event types and event elements indirectly leads to error accumulation during the model iteration and optimization process. Summary of the Invention

[0006] The purpose of this invention is to propose a cross-modal event joint extraction method and system based on a link matrix, in order to solve the problems of insufficient accuracy in weakly supervised and unsupervised multimodal event extraction, error accumulation caused by pipeline multimodal event extraction, and information loss in single-modal event extraction.

[0007] The technical solution to achieve the purpose of this invention is: a multimodal event joint extraction method based on a link matrix, comprising the following steps:

[0008] Step 1: Determine the event structure system of the application domain, including the fine-grained types of events and event elements, obtain text event annotation information, image event annotation information, cross-modal event annotation information, as well as original image and text information, and construct a cross-modal event extraction dataset;

[0009] Step 2: Split the image and construct a link matrix between image blocks and text sequences;

[0010] Step 3: Construct a multimodal event extraction model based on ViLT, including a multimodal Transformer encoder and decoders for different tasks. The multimodal Transformer encoder takes image patch pixel features, word instance features and image patch-text word instance pair features as input to obtain multimodal feature input decoder, and obtains image-text event type consistency features, event type features and image-text event element features.

[0011] Step 4: Based on the consistency features of text and image event types, event type features, and text and image event element features, use maximum likelihood estimation to take the text and image event type matching result with the highest probability, event type, and text and image event element as the index. Use the index mapping to obtain text event information, including event trigger words, event type, and event elements, as well as image event information, including image scene, event type, target location, and event elements.

[0012] Step 1: Determine the event structure of the application domain, obtain text event annotation information, image event annotation information, cross-modal event annotation information, and original image and text information, and construct a cross-modal event extraction dataset. The format of the cross-modal event extraction dataset is as follows:

[0013] 3) Each data point is an image-text pair. Positive samples are data where the image event and text event are consistent, and negative samples are data where the image and text description events are inconsistent.

[0014] 4) The format of text event annotation information is the original text, event trigger word, entity, and the relationship between the trigger word and the entity; the format of image event annotation information is the original image, target category, target bounding box coordinates, event type, and the relationship between the target and the event; the format of cross-modal event annotation information is the relationship between the image target and the text entity.

[0015] Step 2: Split the image and construct a link matrix between image blocks and text sequences. The specific method is as follows:

[0016] Step 2.1: Perform image preprocessing and text preprocessing. Image preprocessing includes denoising and zero-padding, while text preprocessing includes denoising and stop word extraction.

[0017] Step 2.2: Slice the image into several square segments and unfold them from left to right and top to bottom;

[0018] Step 2.3: Using the image patch as the x-axis and the text sequence as the y-axis, construct a link matrix between the image patch and the text sequence. The link matrix is ​​a binary matrix of 0 and 1. When there is a connection between the image patch and the text sequence, the matrix value is 1, otherwise it is 0.

[0019] Image m i Decompose into size p ×size p N image patches of size m i ={p1,p2,…,p N}, where p i Represents the pixel feature vector of each image patch; represents the text sentence as s j ={w1,w2,…,w j ,…,w L}, where w j Let represent the word feature vector obtained from word embedding. Then, the image block-text word pair is represented as (p i ,w j The feature vectors of image block-text word pairs are extracted using a convolutional neural network, and are represented as follows:

[0020] h i,j =CNN(p i ,w j )

[0021] Based on the feature vector h of image block-text word pairs i,j The pairing result predicted by the softmax function is:

[0022]

[0023] Among them, y h P(y) represents the pairing result of image patch-text word pairs. h |h) represents the pairing result y h The probability in the eigenvector h' of the true value of the link matrix.

[0024] Step 3: Construct a multimodal event extraction model based on ViLT, including a multimodal Transformer encoder and decoders for different tasks, wherein the multimodal Transformer encoder is shown in the following equation:

[0025]

[0026]

[0027] Among them, w class p class h class These are text word examples, image patches, and link matrix type masks, respectively, {w1,…,w L}、{p1,…,p N}、{h1,…,h R} represent the word feature vector, the pixel feature vector of the image patch, and the feature vector of the image patch-text word pair, respectively. W, P, and H are the parameter weights of the multimodal Transformer encoder, respectively. pos P pos H pos The location feature embeddings for text, image, and link matrix are respectively, w type p type h type Modal feature embeddings for text, image, and link matrix respectively, z d The tensor z is formed by concatenating the original data features, location features, and modal features. 0 Multimodal feature encoding after LN, MSA and MLP.

[0028] Step 3: Construct a multimodal event extraction model based on ViLT, including a multimodal Transformer encoder and decoders for different tasks. The decoder extracts consistent features of image and text event types through fully connected layers and pooling layers, extracts event type features through MLP, and extracts image and text event element features through entity detection and feature pyramids. The specific methods are as follows:

[0029] 1) Image and text event matching

[0030] A binary linear mapping p for image-text event matching is constructed using pooling and fully connected layers, as shown in the following equation.

[0031]

[0032] in, Represents a multimodal feature set, including a subset of text. and image subset W pool This indicates pooling and fully connected processing;

[0033] The matching result for the image and text event is:

[0034]

[0035] Among them, y p P(y) represents the consistency matching result of image and text event types. p |p) represents the probability of a text-image event matching in p, and p' represents the set of all text-image matching feature vectors.

[0036] 2) Event type detection

[0037] The event type feature representation is extracted using MLP, as shown in the following formula.

[0038]

[0039] Event type is

[0040]

[0041] Among them, y e P(y) represents the prediction result of the event type. e |z) represents the probability of predicting the outcome of this type of event, where z represents all event types. A set;

[0042] 3) Event Element Detection

[0043] Entity detection is performed on a subset of text features using MLP, as shown in the following equation.

[0044]

[0045] Text event element recognition is performed by combining event type features and a fully connected layer, as shown in the following formula.

[0046]

[0047] Among them, y a Represents the prediction of event element types, P(y) a |t) represents entity feature t predicted as event element type y. a The probability, W a and h a This represents the weights and biases of the fully connected layer for event feature detection. W represents the predicted event type vector. a' and b a' This represents the weights and parameters of the fully connected layer during entity a' prediction;

[0048] The target object features are extracted from a subset of image features using FPN, as shown in the following formula.

[0049]

[0050] Image event element recognition is performed by combining event type features and a fully connected layer, as shown in the following equation.

[0051]

[0052] Wherein, P(y a |o) indicates that the target object feature o is predicted as the event element type y.a The probability, W a and b a This represents the weights and biases of the fully connected layer for event feature detection. A vector representing the predicted event types;

[0053] 4) Model loss function

[0054] Constructing the model loss function A multimodal event extraction model is trained, where the model loss function includes event element linking loss, image-text event matching loss, event type detection loss, and image-text event element detection loss, as shown in the following formula.

[0055]

[0056] in These represent the predicted image block and text word matching results, image-text event type consistency results, event type results, and event element results, respectively.

[0057] A multimodal event joint extraction system based on a link matrix, which is based on the aforementioned multimodal event joint extraction method based on a link matrix.

[0058] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs multimodal event joint extraction based on the link matrix-based method.

[0059] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs multimodal event joint extraction based on the link matrix-based multimodal event joint extraction method.

[0060] Compared with existing technologies, the significant advantages of this invention are: it adopts a joint extraction method between different modal data, event types and event elements, and learns the link relationships between event elements in different modalities, thus solving the problems of low recognition accuracy of weakly supervised and unsupervised methods and inaccurate matching of pipeline multimodal event element information, providing a new event extraction strategy for the field of multimodal information extraction. Attached Figure Description

[0061] Figure 1 This is a flowchart of a cross-modal event joint extraction method based on a link matrix.

[0062] Figure 2 This is an example diagram of link matrix generation.

[0063] Figure 3 This is a diagram of the multimodal Transformer architecture. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] This invention discloses a multimodal event joint extraction method based on a link matrix, the specific implementation of which is as follows:

[0066] Step 1. Based on the application direction, domain experts determine the event structure system and obtain a cross-modal event extraction dataset by manual annotation, which serves as the training dataset for the multimodal event extraction model. At the same time, the multimodal information extraction problem is modeled.

[0067] The event structure system of the application domain includes event types and event elements. Let the image set be... The text sentence set is Where m i Represents the image, s i The representative sentences, each consisting of several word examples s = (w1, w2, ...), are as follows:

[0068] Event type defined

[0069] e = (y e ,{w,m})

[0070] Where 'e' represents the event designation, and 'y' represents the event designation. e The event type represents the trigger word w and the image m.

[0071] Event elements are defined as follows

[0072] a=(y a ,{t,o})

[0073] Where 'a' represents event 'e' of type 'y'. a The event elements are: t represents the text entity, and o represents the image target object.

[0074] Taking the extraction of international news events as an example, the following is a partial example of the event structure system:

[0075] 1) The event type is conflict-attack, and the event element types are attacker, tool, location, target, and time of occurrence;

[0076] 2) The event type is conflict-protest, and the event element types are entity, location, and time of occurrence;

[0077] 3) The event type is contact-interview, and the event element types are entity, location, and time of occurrence;

[0078] 4) The event type is life-death, and the event element types are subject, tool, location, victim, and time of occurrence;

[0079] 5) The event type is movement-transportation, and the event element types are subject, destination, departure point, location, means of transport, and time of occurrence;

[0080] 6) Other event types and their event elements.

[0081] The cross-modal event extraction dataset includes text event annotation information, image event annotation information, and cross-modal event annotation information. The format of the cross-modal event extraction dataset is described below:

[0082] 1) Each data point is an image-text pair. Positive samples are data where the image event and text event are consistent, while negative samples are data where the image and text description events are inconsistent.

[0083] 2) The format of text event annotation information is the original text, event trigger word, entity, and the relationship between the trigger word and the entity; the format of image event annotation information is the original image, target category, target bounding box coordinates, event type, and the relationship between the target and the event; the format of cross-modal event annotation information is the relationship between the image target and the text entity.

[0084] Step 2. Employ the image patch-text pairing method to provide feature link information for cross-modal event extraction, generating a link matrix between image patches and text sequences. This matrix is ​​a binary matrix of 0s and 1s. If there is a connection between the image patch and the text sequence in the cross-modal event annotation information, the matrix value is 1; otherwise, it is 0. The process is as follows:

[0085] 1) Perform image preprocessing and text preprocessing. Image preprocessing includes noise reduction and zero-padding; text preprocessing includes noise reduction and stop word removal.

[0086] 2) Slice the image into several square segments, and unfold them from left to right and top to bottom to form a link matrix with the horizontal axis representing the text sequence and the vertical axis representing the link matrix.

[0087] 3) Based on the cross-modal event annotation data, the link matrix value pointed to by the image patch and the associated text sequence is 1, otherwise it is 0. Construct the true value feature vector of the link matrix.

[0088] Image m i Decompose into size p ×size p N image patches of size m can be represented as m i ={p1,p2,…,pN}, where p i Represents the pixel feature vector of each image patch; represents the text sentence as s j ={w1,w2,…,w j ,…,w L}, where w j Let represent the word feature vector obtained from word embedding. Then, the image block-text word pair can be represented as (p i ,w j It employs a convolutional neural network to extract feature vectors from image block-text word pairs, i.e., the predicted link matrix feature values, expressed as:

[0089] h i,j =CNN(p i ,w j )

[0090] Based on the feature vector h of image block-text word pairs i,j The pairing result predicted by the softmax function is:

[0091]

[0092] Among them, y h P(y) represents the pairing result of image patch-text word pairs. h |h) represents the pairing result y h The probability in the eigenvector h' of the true value of the link matrix.

[0093] Step 3. Construct a multimodal event extraction model based on ViLT (Multimodal Language Model), including a multimodal Transformer encoder and decoders for different tasks. Input the image block pixel features, word instance features, and image block-text word instance feature vector obtained in Step 2 into the multimodal Transformer encoder, output multimodal features, and decode to obtain image-text event type consistency features, event type features, and image-text event element features.

[0094] The multimodal features obtained by the multimodal Transformer encoder are shown in the following equation.

[0095]

[0096]

[0097]

[0098] Among them, w class p class h class These are text word, image patch, and link matrix type masks, respectively, {w1,…,wL}、{p1,…,p N}、{h1,…,h R} represent the input text word instance vector, image patch feature vector, and link matrix feature vector, respectively. W, P, and H are the parameter weights in the multimodal Transformer encoder, respectively. pos P pos H pos The location feature embeddings for text, image, and link matrix are respectively, w type p type h type Modal feature embeddings for text, images, and link matrices respectively. The tensor z is formed by concatenating the original data features, location features, and modal features. 0 The intermediate feature tensor z obtained after LN (layer normalization) and MSA (multi-head self-attention) is d It is by Multimodal feature encoding obtained by MLP (Multilayer Perceptron) processing.

[0099] During the decoding process, consistent features of image and text event types are extracted through fully connected layers and pooling layers; multimodal features are processed using an MLP (Multilayer Perceptron) to extract event type features; and image and text event element features are extracted by combining entity detection and feature pyramids with event type features. The specific steps are as follows:

[0100] 1) Image and text event matching

[0101] To more robustly identify input data with consistent image and text events, random negative sample data is mixed in. A binary linear mapping p for image and text event matching is constructed through pooling and fully connected layers, as shown in the following equation.

[0102]

[0103] in, Represents a multimodal feature set, including (text subset) and (Image subset), W pool This indicates pooling and fully connected processing.

[0104] The matching result is:

[0105]

[0106] Among them, y p P(y) represents the consistency matching result of image and text event types. p |p) represents the probability of a text-image event matching in p, and p' represents the set of all text-image matching feature vectors.

[0107] 2) Event type detection

[0108] Event type features are extracted using MLP, therefore the event type features are:

[0109]

[0110] Event type is

[0111]

[0112] Among them, y e P(y) represents the prediction result of the event type. e |z) represents the probability of predicting the outcome of this type of event, where z represents all event types. A set of.

[0113] 3) Event Element Detection

[0114] Since event elements have different characteristics depending on the modality, the multimodal features output by the multimodal Transformer, including text modality and image modality features, are used for feature extraction. The text event element features are first converted into entity features by entity detection of a subset of text features through MLP, as shown in the following formula.

[0115]

[0116] Text event element identification is performed by combining event type features and a fully connected layer.

[0117]

[0118] Among them, y a Represents the prediction of event element types, P(y) a |t) represents entity feature t predicted as event element type y. a The probability, W a and b a This represents the weights and biases of the fully connected layer for event feature detection. Let W represent the predicted event type vector, Σa' represent the set of all entities, and W... a' ,b a' This represents the weights and parameters of the fully connected layer during entity a' prediction.

[0119] Image feature extraction includes bounding box extraction and image feature classification. First, target object features are extracted from a subset of image features using a Feature Pyramid Network (FPN).

[0120]

[0121] Image event element recognition is performed by combining event type features and fully connected layers.

[0122]

[0123] Wherein, P(y a |o) indicates that the target object feature o is predicted as the event element type y. a The probability, W a and b a This represents the weights and biases of the fully connected layer for event feature detection. This represents a vector of predicted event types.

[0124] 4) Model loss function

[0125] Model loss function This includes event element linking loss, image-text event matching loss, event type detection loss, and image-text event element detection loss, used for iterative updates of model optimization parameters during the training process of the cross-modal event joint extraction algorithm, as shown in the following formula.

[0126]

[0127] in These represent the predicted image block and text word matching results, image-text event type consistency results, event type results, and event element results, respectively.

[0128] Step 4. Based on the predicted feature vectors of different subtasks obtained from encoding and decoding in Step 3, including the consistency feature of image and text event types, event type features, and image and text event element features, use maximum likelihood estimation to select the image and text event type matching, event type, and image and text event element index with the highest probability. Use index mapping to obtain text event information, including event trigger words, event type, event elements, etc., as well as image event information, including image scene, event type, target location, and event elements, etc.

[0129] In summary, this invention constructs a sufficient supervised dataset to provide event element signals that connect images and issues across modalities in the form of a link matrix, achieving information complementarity for event recognition between different modalities. During the decoding process, a cascaded approach is used to achieve joint extraction of event image-text matching, event types, and elements, solving problems such as insufficient accuracy and missing information when applied to news event extraction scenarios.

[0130] Example

[0131] To verify the effectiveness of the present invention, the following experiment was conducted.

[0132] Step 1. As Figure 1As shown, the image annotation information is {"Event Type": "Movement / Transportation", "Event Element": [{"Name": "Person", "Area": ​​[45.56, 69.80, 169.80, 159.20]}, {"Name": "Tank", "Area": ​​[39.5, 156.92, 199.35, 251.62]}, {"Name": "Person", "Area": ​​[178.14, 103.89, 255.]}. 41,157.68]},{“Name”:“Tank”,“Area”:[173.59,128.14,273.59,230.41]},{“Name”:“Tank”,“Area”:[268.29,147.08,333.44,206.92],{“Name”:“Flag”,“Area”:[254.65,67.53,300.11,114.5],{“Name”:“Flag” The text annotation information is {"Event Type":{"Name":"Movement / Transportation","Location":[7,7],"Event Elements":[{"Name":"Subject","Location":[5,6]},{"Name":"Region","Location":[13,13]},{"Name":"Region","Location":[15,15]},"Region":[15,15]},"Region":[324.35,94.80,347.83,128.89]}]},"Region":[324.35,94.80,347.83,128.89]}]},"Event Type":{"Name":"Region","Location":[15,15]},"Region":[324.35,94.80,347.83,128.89]}]},"Event Name ... {“Name”:“Time”,“Location”:[16,18]}]}, cross-modal annotation information is [{“Event Type”:“Mobility·Transportation”,“Image”:[[39.5,156.92,199.35,251.62],[173.59,128.14,273.59,230.41],[268.29,147.08,333.44,206.92]],“Text”:[5,6]}].

[0133] Step 2. Preprocess the image to a size of 256×256, then divide it into 16×16 squares, for a total of 16×16 squares, as shown below. Figure 2 As shown, based on the labeled data, the event element link matrix is ​​obtained as [[…,0,0,0,1,1,1,0,1,1,1,1,1,0,…],[…,0,0,0,1,1,1,0,1,1,1,1,1,1,0,…],[…,…,…,…,…,…,…,…,…,…,…,…,…,…,…,…,…,…],[…,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,…],…] 16×16The feature vector of the image patch after linear mapping is [0.5884,0.3597,0.5729,0.6642,0.8851,0.1731,0.5704,0.5510,0.2372,0.1030,0.0856,0.4589,0.9705,0.1491,0.1614,0.2810,0.2215,0.7763,…,0.6559,0.1194,0.4313,0.4881,0.6154,0.4194,0.8049,0.3665,0.3139,0.8492,0.4920,0.2819,0.9918]. 256×1 The word feature vectors represented by the text word embeddings are [0.1351, 0.4667, 0.9711, 0.0447, 0.2952, 0.3228, 0.2894, 0.7722, 0.6241, 0.8972, 0.5196, 0.4169, 0.3260, 0.3443, ..., 0.3627, 0.2435, 0.4504, 0.6320, 0.4290, 0.3688, 0.7571, 0.5852, 0.7321, 0.6783, 0.9038]. 128×1 .

[0134] Step 3. Input the link matrix, image representation, and word embedding representation into the multimodal Transformer encoder, such as... Figure 3 As shown, the decoding results of image-text event matching, event type, and multimodal event elements are obtained through a cascaded decoding framework. First, the image-text event matching tensor is obtained through pooling and fully connected layers [1], that is, the image event and the text event are consistent; second, the event type feature vector is obtained by decoding with a multilayer perceptron as [0.1690,0.2959,0.9282,0.9715,0.7027,0.8655,0.4094,0.1685,0.6156,0.0892,0.5876,0.1131,0.3816,0.4763,0.7701,0.5999,0.7681,0.6936,0.9438,0.9851,0.6234,0.2804,0.4179,0.5181]. 24×1Finally, using entity detection and event type feature decoding, the text event element type feature vector is obtained as [[0.7779,0.9249,0.0602,...,0.1628,0.8574,0.1320],[0.8533,0.0586,0.7071,...,0.4662,0.1056,0.3668],[0.0301,0.7973,0.2917,...,0.1066, 0.7565,0.8729],...,[0.2720,0.9982,0.7596,...,0.0505,0.7387,0.6596],[0.3111,0.5744,0.3634,...,0.3186,0.0956,0.4179],[0.8932,0.7529,0.4045,...,0.6947,0.1882,0.4877]] 128×134 The feature vector of image event element type is [[0.8232,0.1327,0.1650,...,0.3047,0.8104,0.0725],[0.0093,0.6090,0.0901,...,0.4998,0.2472,0.6409],[0.4709,0.1826,0.5406,...,0.3972,0.7272,0. 9551],...,[0.3609,0.3492,0.0507,...,0.7944,0.0103,0.0430],[0.0604,0.9053,0.4503,...,0.3503,0.8380,0.4149],[0.9991,0.0395,0.5051,...,0.9934,0.2100,0.0957]] 256×134 .

[0135] Step 4. Based on the results in Step 3, the event type is Move.Transport, the trigger word is moves, and the event elements include the main elements that exist in both modalities. In the text, these are displayed as ××× (×××), and in the image, they are displayed as the target area of ​​the tank crew. The event also includes information specific to each modality, such as (location passed through, ×××) and (occurrence time, July 23, 2022) in the text, and (tool, tank 1) and (tool, tank 2) in the image.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for joint extraction of multimodal events based on a link matrix, characterized in that, Includes the following steps: Step 1: Determine the event structure system of the application domain, including the fine-grained types of events and event elements, obtain text event annotation information, image event annotation information, cross-modal event annotation information, as well as original image and text information, and construct a cross-modal event extraction dataset; Step 2: Split the image and construct a link matrix between image blocks and text sequences; Step 3: Construct a multimodal event extraction model based on ViLT, including a multimodal Transformer encoder and decoders for different tasks. The multimodal Transformer encoder takes image patch pixel features, word instance features and image patch-text word instance pair features as input to obtain multimodal feature input decoder, and obtains image-text event type consistency features, event type features and image-text event element features. Step 4: Based on the consistency features of text and image event types, event type features, and text and image event element features, use maximum likelihood estimation to take the text and image event type matching result, event type, and text and image event element with the highest probability as an index. Use the index mapping to obtain text event information, including event trigger words, event type, and event elements, as well as image event information, including image scene, event type, target location, and event elements. Step 2: Split the image and construct a link matrix between image blocks and text sequences. The specific method is as follows: Step 2.1: Perform image preprocessing and text preprocessing. Image preprocessing includes denoising and zero-padding, while text preprocessing includes denoising and stop word extraction. Step 2.2: Slice the image into several square segments and unfold them from left to right and top to bottom; Step 2.3: Using the image patch as the x-axis and the text sequence as the y-axis, construct a link matrix between the image patch and the text sequence. The link matrix is ​​a binary matrix of 0 and 1. When there is a connection between the image patch and the text sequence, the matrix value is 1, otherwise it is 0. Image m i Decompose into size p ×size p N image patches of size m i ={p1,p2,…,p N }, where p i Represents the pixel feature vector of each image patch; represents the text sentence as s j ={w1,w2,…,w j ,…,w L }, where w j Let represent the word feature vector obtained from word embedding. Then, the image block-text word pair is represented as (p i ,w j The feature vectors of image block-text word pairs are extracted using a convolutional neural network, and are represented as follows: h i,j =CNN(p i ,w j ) Based on the feature vector h of image block-text word pairs i,j The pairing result predicted by the softmax function is: Among them, y h P(y) represents the pairing result of image patch-text word pairs. h |h) represents the pairing result y h The probability in the eigenvector h' of the true value of the link matrix.

2. The multimodal event joint extraction method based on the link matrix according to claim 1, characterized in that, Step 1: Determine the event structure of the application domain, obtain text event annotation information, image event annotation information, cross-modal event annotation information, and original image and text information, and construct a cross-modal event extraction dataset. The format of the cross-modal event extraction dataset is as follows: 1) Each data point is an image-text pair. Positive samples are data where the image event and text event are consistent, and negative samples are data where the image and text description events are inconsistent. 2) The format for text event annotation information is: original text, event trigger word, entity, and the relationship between the trigger word and the entity; the format for image event annotation information is: original image, target category, target bounding box coordinates, event type, and the relationship between the target and the event. Cross-modal event annotation information is formatted as the relationship between image targets and text entities.

3. The multimodal event joint extraction method based on the link matrix according to claim 1, characterized in that, Step 3: Construct a multimodal event extraction model based on ViLT, including a multimodal Transformer encoder and decoders for different tasks, wherein the multimodal Transformer encoder is shown in the following equation: Among them, w class p class h class These are text word examples, image patches, and link matrix type masks, respectively, {w1,…,w L }、{p1,…,p N }、{h1,…,h R } represent the word feature vector, the pixel feature vector of the image patch, and the feature vector of the image patch-text word pair, respectively. W, P, and H are the parameter weights of the multimodal Transformer encoder, respectively. pos P pos H pos The location feature embeddings for text, image, and link matrix are respectively, w type p type h type Modal feature embeddings for text, images, and link matrices respectively. A tensor formed by concatenating the original data features, location features, and modal features. Multimodal feature encoding after LN, MSA and MLP.

4. The multimodal event joint extraction method based on the link matrix according to claim 3, characterized in that, Step 3: Construct a multimodal event extraction model based on ViLT, including a multimodal Transformer encoder and decoders for different tasks. The decoder extracts consistent features of image and text event types through fully connected layers and pooling layers, extracts event type features through MLP, and extracts image and text event element features through entity detection and feature pyramids. The specific methods are as follows: 1) Image and text event matching A binary linear mapping p for image-text event matching is constructed using pooling and fully connected layers, as shown in the following equation. in, Represents a multimodal feature set, including a subset of text. and image subset W pool This indicates pooling and fully connected processing; The matching result for the image and text event is: Among them, y p P(y) represents the consistency matching result of image and text event types. p |p) represents the probability of a text-image event matching in p, and p' represents the set of all text-image matching feature vectors; 2) Event type detection The event type feature representation is extracted using MLP, as shown in the following formula. Event type is Among them, y e The result represents the prediction of the event type. This indicates the probability of predicting the outcome of this type of event. Represents all event types A set; 3) Event Element Detection Entity detection is performed on a subset of text features using MLP, as shown in the following equation. Text event element recognition is performed by combining event type features and a fully connected layer, as shown in the following formula. Among them, y a Represents the prediction of event element types, P(y) a |t) represents entity feature t predicted as event element type y. a The probability, W a and b a This represents the weights and biases of the fully connected layer for event feature detection. W represents the predicted event type vector. a' and b a' This represents the weights and parameters of the fully connected layer during entity a' prediction; The target object features are extracted from a subset of image features using FPN, as shown in the following formula. Image event element recognition is performed by combining event type features and a fully connected layer, as shown in the following equation. Wherein, P(y a |o) indicates that the target object feature o is predicted as the event element type y. a The probability, W a and b a This represents the weights and biases of the fully connected layer for event feature detection. A vector representing the predicted event type; 4) Model loss function Constructing the model loss function A multimodal event extraction model is trained, wherein the model loss function includes event element linking loss, image-text event matching loss, event type detection loss, and image-text event element detection loss, as shown in the following formula. in These represent the predicted image block and text word matching results, image-text event type consistency results, event type results, and event element results, respectively.

5. A multimodal event joint extraction system based on a link matrix, characterized in that, The method for joint extraction of multimodal events based on a link matrix as described in any one of claims 1-4 is used for joint extraction of multimodal events based on a link matrix.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it performs multimodal event joint extraction based on a link matrix according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it performs multimodal event joint extraction based on a link matrix according to any one of claims 1-4.

Images