Text-video cross-modal event element extraction method

Abstract

The invention discloses a text-video cross-modal event element extraction method, which comprises the following steps of: collecting video data and video brief introduction text data thereof, and respectively labeling event types of the text and the video data and corresponding event argument roles, event types pre-marked by the text data and the video data are consistent with event argument roles corresponding to the event types; performing multi-modal event anaphora resolution to realize co-reference event pairing between any text-video data, namely matching a text and a video which refer to the same event to form a group of text-video co-reference event pairs; the matched text-video data are converted into corresponding feature vectors, text marking and text embedding are carried out on the text data, and the text data are converted into a word vector form; directly using a ResNet algorithm for video data to obtain global level event element features, and constructing a video global feature vector; identifying a local object of the video data through Fast-R-CNN, obtaining a local level time element by using a ResNet algorithm, and constructing a video local feature vector; unifying the vector dimensions of the text word vector, the video global feature vector and the video local feature vector through a full connection layer, and constructing a text-video shared vector space; and inputting a text word vector, a video global feature vector and a video local feature vector into a Transformer encoder, carrying out event element information extraction of a text mode by adopting an ONEIE algorithm, and carrying out event element information extraction of a video mode by adopting a T5-base algorithm. According to the method, the relevance among the multimodal interiors can be more accurately captured, and the extraction precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and in particular to a text-video cross-modal event element extraction method. Background Art

[0002] The purpose of event extraction is to extract event-related element data from unstructured raw data and represent it in a structured manner. Event extraction mainly covers event type detection, event argument extraction, and event argument role extraction. Events are usually composed of event trigger words and elements that describe the event structure, where event trigger words refer to the occurrence of events, while event arguments and argument roles are components of events. Usually, the difference between arguments and argument roles is that arguments refer to the entity type of the event element, while argument roles refer to the role played by the event element in the event.

[0003] At present, common event extraction technology can only extract events from data in text format. However, for information carriers such as images and videos that contain richer semantic information, the event information they contain is richer than that of text format data, and can express event elements that cannot be intuitively expressed by text data. Event extraction technology for this type of modality is less developed and can only be used for a single mode. However, information publishing media in the real world usually publish content in multiple modes, such as a news report composed of text and pictures, or a brief introduction to the content of the video when publishing a video. The data of these modes together constitute the description of the event. Therefore, event extraction of single-modal data may lead to incomplete extraction of event elements due to the limitations of its data format itself. Summary of the invention

[0004] The purpose of the present invention is to propose a text-video cross-modal event element extraction method to enrich the event element extraction results of text-video multimodal event data and improve the integrity of event extraction.

[0005] The technical solution to achieve the purpose of the present invention is: a text-video cross-modal event element extraction method, comprising the following steps:

[0006] Step 1: Collect video data and its video introduction text data, and annotate the event types and corresponding event argument roles of the text and video data respectively. The event argument roles represent entities that play different roles in the event. The event types and event argument roles corresponding to the event types in the text and video data are consistent;

[0007] Step 2: Perform multimodal event coreference resolution to achieve co-referential event pairing between any "text-video" data, that is, match the text and video referring to the same event to form a set of text-video co-referential event pairs;

[0008] Step 3: Convert the matched "text-video" data into corresponding feature vectors, where text data is tokenized and embedded into word vectors; the ResNet algorithm is used directly on the video data to obtain global event element features and construct the video global feature vector; the Fast-R-CNN is used to identify local objects in the video data, and the ResNet algorithm is used to obtain local time elements and construct the video local feature vector;

[0009] Step 4: Unify the vector dimensions of the text word vector, video global feature vector, and local feature vector through a fully connected layer to construct a text-video shared vector space;

[0010] Step 5: Input the text word vector and the video global feature vector and local feature vector into the Transformer encoder, and then use the ONEIE algorithm to extract event element information in the text mode, and use the T5-base algorithm to extract event element information in the video mode;

[0011] Step 6: Based on the training process of steps 3-5 above, a text-video cross-modal event extraction model is obtained.

[0012] Furthermore, in step 1, long videos with a duration of more than 15 minutes are filtered out, and the filtered videos and their introduction texts are checked to ensure that the video content contains at least one visually observable event, the text content contains at least one event, and contains at least one text-video event matching pair.

[0013] Furthermore, in step 1, when a video contains multiple event types, the video data needs to be labeled with all the event types it contains and the start and end time of each event in the video. For each event segment in the video, at least three frames need to be selected, and two to three entities are labeled in each frame as the argument roles of the event.

[0014] Furthermore, in step 2, multimodal event reference resolution is performed to achieve co-referential event pairing between any "text-video" data, that is, text and video referring to the same event are matched to form a set of text-video co-referential event pairs. The specific method is:

[0015] Construct the standard noise contrast loss NCE:

[0016]

[0017] Among them, represents the ability to be combined with sentence x iA video clip to be matched; f and g are fully connected layer networks that project sentences and video clips into a common vector space to realize the conversion of sentences and video clips into feature vectors with consistent latitude; n represents the number of all text-video pairs used for judgment; Represents the negative sample set of "text-video" mismatch; e is the natural base; (x', y') represents the negative sample set Unmatched text-video pairs;

[0018] Constructing object-based multi-instance learning losses

[0019]

[0020] Among them, x represents a sentence; z represents an object that appears in a certain frame of the video; f and h are fully connected layer networks that project sentences and objects into a common vector space to realize the conversion of sentences and local objects into feature vectors with consistent latitude; Represents a set of positive “text-object” samples that can be matched with the video clip;

[0021] Construct the final multimodal event reference resolution loss function mmcoref:

[0022]

[0023] Based on the constructed multimodal event reference resolution loss function mmcoref, the fully connected layer networks f, g, and h in the training loss function are trained. By providing arbitrary text and video data, it is possible to determine whether the two belong to the same event.

[0024] Further, step 3: convert the matched "text-video" data into corresponding feature vectors, where:

[0025] For text data, perform text tokenization and text embedding to convert all words in the input sentence into word vectors. If a word is divided into multiple word fragments, use the average of all fragment vectors as the vector representation of the word.

[0026] (1) Global event context: extracted from the entire video clip using the ResNet algorithm;

[0027] (2) Frame-level object label: generated by the Fast-R-CNN object detector, indicating the argument role to which the object belongs;

[0028] (3) Frame-level region features: Using the ResNet algorithm, the image is extracted from the bounding box detected by the Fast-R-CNN object detector;

[0029] (4) Frame-level object coordinates: The Fast-R-CNN object detector provides bounding box coordinates for parameter positioning;

[0030] The entire video clip is placed in the ResNet architecture to obtain the global feature vector of the entire video. To obtain the local feature vector, t frames are extracted from the video. For each frame, the Fast-R-CNN target detector is used to identify and obtain k objects with the highest confidence scores for sampling. For each object, the object image is passed through the ResNet architecture to obtain the local target image features; the object label and coordinates are converted into word vector form by text tokenization and text embedding, and the image feature vector, label word vector, and coordinate word vector corresponding to each object after processing are collectively used as the local feature vector of the object.

[0031] Further, step 4: unify the vector dimensions of the text word vector and the video global feature vector and local feature vector through a fully connected layer to construct a text-video shared vector space, where:

[0032] The dimensions of text embedding vectors are unified as (m, 768), the dimensions of video global feature vectors are unified as (1, 768), and the dimensions of video local feature vectors are unified as (n, 768). Based on this, a 768-D space is obtained as the text-video shared vector space.

[0033] Further, step 5: using the ONEIE algorithm to extract event element information in the text mode, and using the T5-base algorithm to extract event element information in the video mode, where:

[0034] The ONEIE algorithm is used to extract event element information in text mode, specifically:

[0035] Identify entity mentions and event trigger words in sentences as nodes in the information network, that is, first use the feedforward network FFN to generate the intermediate vector w for each text output by the Transformer encoder. i Calculate the corresponding score vector Score vector Each item in represents a word w i For each label, that is, the score of the entity position and entity type to which the word belongs; then use the conditional random field layer to restrict the dependency between the predicted labels and obtain the optimal label for each word; finally, classify the word or phrase as an entity mention or event trigger word as a node in the information network;

[0036] For the identified nodes that are single words or phrases consisting of multiple words, for the case of multiple words, the lexical representations of all words are averaged to obtain the lexical representation of the multi-word node, and then the feedforward network is used to calculate the v of each node i The score vector for a specific task t

[0037] Put the edge ij Defined as the relationship between the i-th node and the j-th node, the lexical representations of the two nodes are concatenated to obtain the vector representation of this edge, and the score vector of this edge for a specific task t is calculated for the vector representation of this edge

[0038] Use the cross entropy loss function to shrink the score vectors of nodes and edges calculated for a specific task t and the target score vector Based on the gap between them, the classification of nodes and edges for tasks is completed, that is, to identify which event the node / edge belongs to;

[0039] Use beam search to combine each inferred node / edge to complete the event extraction of text data;

[0040] The T5-base algorithm is used to extract event element information of the video modality, specifically:

[0041] The video intermediate vector output by the Transformer encoder is directly put into the decoder of the T5-base model as input to complete the event element information extraction of the video modality, and the output text sequence format is defined as: (event type of event 1; the first argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; the second argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; ...; event type of event 2; the first argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; the second argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; ...)

[0042] After obtaining the event element information extraction results of the text and video, the arguments in the same event are matched, and the corresponding start and end positions of each argument in the text are added to the output video event extraction result sequence to obtain the final target sequence output.

[0043] A text-video cross-modal event element extraction system implements the text-video cross-modal event element extraction method to achieve text-video cross-modal event element extraction.

[0044] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the text-video cross-modal event element extraction method is implemented to achieve text-video cross-modal event element extraction.

[0045] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the text-video cross-modal event element extraction method is implemented to achieve text-video cross-modal event element extraction.

[0046] Compared with the prior art, the present invention has the following significant advantages: 1) It combines scene perception and context understanding to achieve effective fusion of multimodal data and accurate extraction of event information. 2) Based on text and video data, it can extract more event parameters, including the information conveyed by static images and the contextual information of each static image frame. 3) When it is not possible to directly borrow existing image events, with the help of a new multimodal transformer structure, it can dynamically focus on the most relevant parts of the text description and video content, more accurately capture the correlation between multimodal internals, and improve the extraction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the text-video cross-modal event feature extraction method. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] Based on the technology of event extraction in single text mode and single video mode, the present invention introduces a self-supervised model and a multimodal transformer architecture. The self-supervised model is used to find text sentences and video clip pairs describing the same event, extract the sentence-video co-reference clip pairs in the text-video mode, and extract events from them; the multimodal transformer architecture is used to jointly perform event classification and event element extraction from video and text. The specific implementation steps are as follows:

[0050] Step 1: Collect video data and text data (text data is the video introduction related to the video content) in the vertical segmented field that the text-video cross-modal event extraction model plans to solve. For the collected data, filter out long videos longer than 15 minutes to avoid the information explosion that may be caused by this; check the filtered videos and their introduction texts to ensure that the video content contains at least one visually observable event, the text content contains at least one event, and contains at least one text-video event matching pair.

[0051] Use image annotators to annotate text and video data. Annotate text data with event types and event argument roles. Considering that a video often contains more than one event, video data needs to be annotated with all event types it contains and the start and end time of each event in the video. For each event segment in the video, at least three frames need to be selected, and each frame should be annotated with two to three typical entities (such as people, objects, etc.) as the argument roles of the event. The pre-annotated event types and event argument roles corresponding to the event types in the text data and video data should be consistent, that is, which event types are annotated in the text data should also have the corresponding event types annotated in the video data, and which event argument roles are annotated in the text data should also have the corresponding event argument roles annotated in the video data.

[0052] The event types / event argument roles annotated by the text data and the event types / event argument roles annotated by the video data constitute a text-video multimodal co-referential event pair. The text-event co-referential relationship can be marked as 1 to 1, 1 to n, n to 1, and n to n, where 1 to 1 means that a text event is only related to a single video event; 1 to n means that 1 text event is related to multiple video events, n to 1 means that multiple text events are related to 1 video event, and n to n means that multiple text events are related to multiple video events. The event types and event argument roles given here are the event elements that need to be extracted for the text-video cross-modal event extraction of the present invention.

[0053] Step 2: Perform multimodal event coreference resolution. This step can realize the coreference event pairing between any "text-video" data, that is, determine whether the text and video belong to the same event. If the text and video can match and refer to the same event, it is a positive sample pair, otherwise it is a negative sample pair.

[0054] Construct the standard noise contrast loss (NCE) with the following formula:

[0055]

[0056] Among them, x i Represents a sentence; yi Represents the ability to i A video clip to be matched; f and g are two fully connected layer networks that can project sentences and video clips into a common vector space, thereby converting sentences and video clips into feature vectors with consistent latitude; n represents the number of all text-video pairs used for judgment; Represents the negative sample set of "text-video" mismatch; e is the natural base; (x', y') represents the negative sample set Unmatched “text-video” pairs. By using the standard noise contrast loss, we can narrow down the matching “text-video” positive sample pairs (x i ,y i ), pulling away the unmatched negative sample pairs (x', y'), can achieve coarse-grained global event reference resolution between "text-video" data.

[0057] In addition, some local objects appearing in the video will also affect the pairing of "text-video" (for example, a gun appearing in a certain frame of the video is matched with the gun in the text). Therefore, the object-based multi-instance learning loss is introduced.

[0058]

[0059] Where x represents a sentence. z represents an object that appears in a certain frame of the video (actually a portion of an image captured from a certain frame). f and h are fully connected layer networks that project sentences and objects into a common vector space, thus converting sentences and local objects into feature vectors with consistent latitude. Given a video clip, Represents a set of positive “text-object” samples that can be matched with the video clip; represents the negative sample set of "text-object" that has no association and cannot be paired; e is the natural base; (x',z') represents the negative sample set Unmatched text-object pair. It can enable many objects in video instance i to better match text sentences and achieve more fine-grained local event reference resolution.

[0060] Constructing the final multimodal event reference resolution (mmcoref) loss function, that is, achieving global pairing and local pairing sum.

[0061]

[0062] By training the fully connected layer networks f, g, and h in the loss function and providing any text and video data, it is possible to determine whether the two belong to the same event.

[0063] Step 3: For the "text-video" pairs paired in step 2, perform data preprocessing and convert them into corresponding feature vectors.

[0064] For text data, perform text tokenization and text embedding to convert all words in the input sentence into word vectors. If a word is divided into multiple word fragments, use the average of all fragment vectors as its word representation.

[0065] For video data, the following four feature types are used to comprehensively represent global and local information:

[0066] (1) Global event context: extracted through the entire video clip using the ResNet algorithm;

[0067] (2) Frame-level object label: generated by the Fast-R-CNN object detector, indicating the argument role to which the detected object belongs;

[0068] (3) Frame-level object features: Using the ResNet algorithm, the object features are extracted from the object bounding box detected by the Fast-R-CNN object detector to provide fine-grained local object feature information;

[0069] (4) Frame-level object coordinates: provided by the Fast-R-CNN object detector, indicating the location information of the detected object bounding box, which is used for parameter positioning.

[0070] Then, t frames are sampled for each video clip (t=3). For each frame, an object detector is used to select k target entity objects with the highest confidence score. The global level event feature features of the video are extracted through the ResNet architecture. The labels, images, and bounding box coordinates of the frame-level event feature objects are obtained by Fast-R-CNN recognition. The images are extracted through the ResNet architecture, and the labels and bounding box coordinates are converted into word vector feature forms through text tokenization and text embedding.

[0071] Step 4: Based on the vectorized text data and the global and local feature vectors of the video obtained in step 3, unify the vector dimensions of the three in the form of feature vectors through a fully connected layer. The dimension of the text embedding vector is unified to (m, 768), where m represents the number of tokens. The dimension of the global feature vector of the video is unified to (1, 768); the dimension of the local feature vector of the video is unified to a set of (n, 768). Each local feature extracted from the video corresponds to a feature vector of (n, 768), which can be further divided into (n1, 768), (1, 768), and (n2, 768), that is, n = n1 + 1 + n2. The n1 in (n1, 768) represents the number of tokens of the target label; (1, 768) represents the target image feature; and the n2 in (n2, 768) represents the number of tokens of the target bounding box coordinates. In this way, a 768-dimensional space is constructed as the text-video shared vector space.

[0072] Step 5: Based on step 4, the text vector and video vector of uniform dimension are input into the Transformer encoder respectively. The text and video intermediate vectors outputted by the Transformer encoder keep the latitude consistent with the original vector. The output of the text vector uses the ONEIE joint neural network framework to extract text element information; the extraction of video event elements is performed using the T5-base model.

[0073] The ONEIE framework usually extracts information networks from a given sentence in four steps: encoding, recognition, classification, and decoding. The encoding step usually uses a pre-trained BERT encoder to encode all the words in the input sentence into word representations. The text vector has been obtained through the previous steps, so here the intermediate vector output by the Transformer is directly put into the recognition step as input.

[0074] The goal of the recognition step is to identify entity mentions and event triggers in the sentence and use them as nodes in the information network. First, a feedforward network FFN is used to represent each word w i Calculate the corresponding score vector Score vector Each item in represents a word w iFor the score of each label (here, the BIO serialization labeling method is used to label each word in the sentence: B represents the beginning of a named entity, and the word marked as B is at the beginning of a named entity; I represents the inside of a named entity, and the word marked as I is inside a named entity; O represents the outside of a named entity, and the word marked as O does not belong to any named entity. The named entity types used here can be divided into seven types, namely: personnel type PER, organization type ORG, location type LOC, facility type FAC, weapon type WEA, vehicle type VEH, and geopolitical entity GPE. For example, "John Smith" in a paragraph is marked as the personnel type PER in the named entity, and the word "John" is marked as B-PER, and the word "Smith" is marked as I-PER). Then use the conditional random field (CRFs) layer to restrict the dependency between the predicted labels (for example, the I-PER label must follow the B-PER label) and finally obtain the optimal label for each word. After obtaining the label, some words or phrases can be classified as entity mentions or event triggers. The identified entity mentions and event trigger words are used as nodes in the information network. The identified nodes may be single words or phrases composed of multiple words. For multiple words, the vocabulary representations of all words are averaged to obtain the vocabulary representation of the multi-word node. Next, the feedforward network is used to calculate the v of each node. i The score vector for a specific event type t Put the edge ij Defined as the relationship between the i-th node and the j-th node. In order to obtain an edge e ij The score vector of the edge is obtained by concatenating the vocabulary representations of the two nodes. Then, a similar method is used to calculate the score vector of the edge for a specific task t. Finally, the cross entropy loss function is used (N t represents the total number of nodes and edges contained in a specific task t) to narrow the calculated node and edge score vector for a specific task t and the target score vector Through this step, the classification of nodes and edges for tasks is completed, that is, to identify which event the node / edge belongs to.

[0075] Beam search is used to combine each inferred node / edge to complete the event extraction of text data. The text event extraction result is in a structured format: (Event Type 1: [Argument 1 <Word position, entity>, Argument 2 <Word position, entity>, Argument 3 <Word position, entity>]; Event Type 2: [Argument 1 <Word position, entity>, Argument 2 <Word position, entity>, Argument 3 <Word position, entity>]; ...).

[0076] T5-base model:

[0077] The T5 (Text-to-Text Transfer Transformer) model for video event extraction uses a unified encoder-decoder framework to treat the event extraction task in the video modality as a Sequence-to-Sequence task. Its architecture is basically the same as the original Transformer architecture, with the main differences being: (1) eliminating the canonical deviation of the convolutional layer; (2) placing the convolutional layer normalization outside the residual path; and (3) using a different position encoding method.

[0078] The video intermediate vector output by the Transformer encoder is directly put into the decoder of the T5-base model as input to complete the decoding step of the video event extraction end. Since the output of T5 is a series of text sequences, the output text sequence format is defined as: (event type of event 1; the first argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; the second argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; ...; event type of event 2; the first argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; the second argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; ...).

[0079] After obtaining the event extraction results of the text and video, the arguments in the same event are matched. The corresponding start and end positions of each argument in the text are added to the output video event extraction result sequence. The final target sequence output is obtained.

[0080] Step 6: Based on the training process of steps 3-5 above, a text-video cross-modal event extraction model is obtained.

[0081] In order to make the purpose, technical solution and advantages of the present application more clear, the present application is further described in detail below in conjunction with the embodiments and the attached flow chart. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0082] Example

[0083] The above text-video cross-modal event extraction model is deployed. Users can use the model on the deployed system. The system extracts cross-modal events based on the text and video data entered by the user. Taking user A as an example, the cross-modal event extraction process is as follows:

[0084] Step 1:

[0085] First, user A collects open-source sample data about "current news". For example, he selects 500 videos of 1-2 minutes in length from a video portal website using keywords such as "conflict" and "attack", and uses the articles or video descriptions under the videos as text data. The video data and text data together constitute the initial training data set, and the event types and event elements in the initial training data set are manually annotated. In this example, the event types involved are mainly "Attack" attack events and "Die" death events. The event elements included in the "Attack" event are:<Attacker,Target,Place,Instrument> , respectively representing the attacker of the attack event, the target of the attack event, the location of the attack event, and the instrument or equipment used in the attack; the event elements of the "Die" event are<Agent,Victim,Place,Instrument> , respectively representing the implementing agency / personnel that caused the death, the victim of the death, the location where the death occurred, and the equipment used by the implementer. The image annotator is used to annotate the event type and event elements of the video data. At least three frames are selected for annotating each video, and the content of each annotated frame contains at least one event type and three event elements. The text data associated with the video material is annotated with event types and event elements.

[0086] Step 2:

[0087] The text data and video clips obtained in step 1 are paired with co-referential events, and the event reference resolution between the text and video modalities is completed based on global and local pairing to obtain "text-video" matching pairs. For example, a match is achieved between a terrorist attack video data X and the text "Several militants opened fire on the public at the train station, killing at least 10 people."

[0088] Step 3:

[0089] Data preprocessing is performed on the "text-video" data pairs matched based on step 2. The text data is subjected to text tokenization and text embedding and converted into word vector form. For video data, the global feature vector and local feature vector of the video data are extracted. To obtain the global feature vector of the video, the global feature vector of the entire video can be obtained by placing the entire video clip into the ResNet architecture. To obtain the local feature vector, t frames (t=3) are extracted from the video. For each frame, the Fast-R-CNN target detector is used to identify and obtain k objects with the highest confidence scores for sampling. For example, the 90th frame in the video data X is randomly selected, and the entire video screen of the 90th frame is placed into the target detector Fast-R-CNN. The object detector recognizes, detects and frames many objects in the 90th frame, such as "PER person holding a gun", "GUN in the hands of the person", "TRAIN train parked on the platform in the background", "PER person lying on the ground", and "DUSTBIN on the platform", with corresponding confidence levels of 0.98, 0.95, 0.93, 0.93, and 0.85 respectively. The top 4 objects are ranked from high to low according to the confidence level. For each object, the object image is passed through the ResNet architecture to obtain local target image features; the object label and coordinates are converted into word vector form by tokenization and embedding methods similar to text. The image feature vector, label word vector, and coordinate word vector corresponding to each object after processing are used together as the local level event element features of the object.

[0090] Step 4:

[0091] Based on the text embedding vector and the video global feature vector and local feature vector obtained in step 3, the vector dimensions of the three are unified through a fully connected layer to construct a text-video shared vector space.

[0092] Step 5:

[0093] Put the text and video feature vectors whose dimensions have been unified in step 4 into the Transformer encoder. Perform event element extraction on the text side and the video side for the output text and video intermediate vectors respectively. The text intermediate vector is input into the ONEIE joint neural network framework for text element information extraction; the video intermediate vector is input into the decoder of the T5-base model for video modality event element extraction. In this example, two event types "Attack" and "Die" are extracted from the text modality data: "Attack" event<Attacker,Target,Place,Instrument> The event elements are extracted as <several armed elements, civilians, train station, gun>; "Die" event<Agent,Victim,Place,Instrument> The event elements are extracted as <several armed elements, civilians / 10 people, train station, gun>. The objects extracted from the video modality data such as "personnel holding guns PER", "personnel lying on the ground PER", "train train parked on the platform in the background", and "guns GUN in the hands of personnel" are all supplemented for the event extraction results of the pure text modality. Finally, the text and video event extraction result sequences are combined to obtain the final target sequence output.

[0094] Step 6:

[0095] Get the text-video cross-modal event extraction results.

[0096] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A text-video cross-modal event element extraction method, characterized in that: The steps include: Step 1: Collect video data and its video introduction text data, and annotate the event types and corresponding event argument roles of the text and video data respectively. The event argument roles represent entities that play different roles in the event. The event types and event argument roles corresponding to the event types in the text and video data are consistent; Step 2: Perform multimodal event coreference resolution to achieve co-referential event pairing between any "text-video" data, that is, match the text and video referring to the same event to form a set of text-video co-referential event pairs; Step 3: Convert the matched "text-video" data into corresponding feature vectors, where text data is tokenized and embedded into word vectors; the ResNet algorithm is used directly on the video data to obtain global event element features and construct the video global feature vector; the Fast-R-CNN is used to identify local objects in the video data, and the ResNet algorithm is used to obtain local time elements and construct the video local feature vector; Step 4: Unify the vector dimensions of the text word vector, video global feature vector, and local feature vector through a fully connected layer to construct a text-video shared vector space; Step 5: Input the text word vector and the video global feature vector and local feature vector into the Transformer encoder, and then use the ONEIE algorithm to extract event element information in the text mode, and use the T5-base algorithm to extract event element information in the video mode; Step 6: Based on the training process of steps 3-5 above, a text-video cross-modal event extraction model is obtained.

2. The text-video cross-modal event element extraction method according to claim 1 is characterized in that: In step 1, long videos with a duration of more than 15 minutes are filtered out, and the filtered videos and their introduction texts are checked to ensure that the video content contains at least one visually observable event, the text content contains at least one event, and at least one text-video event matching pair.

3. The text-video cross-modal event element extraction method according to claim 1, characterized in that: In step 1, when a video contains multiple event types, the video data needs to be labeled with all the event types it contains and the start and end time of each event in the video. For each event segment in the video, at least three frames need to be selected, and two to three entities are labeled in each frame as the argument roles of the event.

4. The text-video cross-modal event element extraction method according to claim 1, characterized in that: Step 2: Perform multimodal event coreference resolution to achieve co-referential event pairing between any "text-video" data, that is, match the text and video referring to the same event to form a set of text-video co-referential event pairs. The specific method is as follows: Construct the standard noise contrast loss NCE: Among them, y i Represents the ability to be associated with sentence x i A video clip to be matched; f and g are fully connected layer networks that project sentences and video clips into a common vector space to realize the conversion of sentences and video clips into feature vectors with consistent latitude; n represents the number of all text-video pairs used for judgment; Represents the negative sample set of "text-video" mismatch; e is the natural base; (x', y') represents the negative sample set Unmatched "text-video" pairs; Constructing object-based multi-instance learning losses Among them, x represents a sentence; z represents an object that appears in a certain frame of the video; f and h are fully connected layer networks that project sentences and objects into a common vector space to realize the conversion of sentences and local objects into feature vectors with consistent latitude; Represents a set of positive “text-object” samples that can be matched with the video clip; Construct the final multimodal event reference resolution loss function mmcoref: Based on the constructed multimodal event reference resolution loss function mmcoref, the fully connected layer networks f, g, and h in the training loss function are trained. By providing arbitrary text and video data, it is possible to determine whether the two belong to the same event.

5. The text-video cross-modal event element extraction method according to claim 1, characterized in that: Step 3: Convert the matching "text-video" data into corresponding feature vectors, where: For text data, perform text tokenization and text embedding to convert all words in the input sentence into word vectors. If a word is divided into multiple word fragments, use the average of all fragment vectors as the vector representation of the word. For video data, the following four feature types are used to comprehensively represent global and local information: (1) Global event context: extracted from the entire video clip using the ResNet algorithm; (2) Frame-level object label: generated by the Fast-R-CNN object detector, indicating the argument role to which the object belongs; (3) Frame-level region features: Using the ResNet algorithm, the image is extracted from the bounding box detected by the Fast-R-CNN object detector; (4) Frame-level object coordinates: The Fast-R-CNN object detector provides bounding box coordinates for parameter positioning; The entire video clip is placed in the ResNet architecture to obtain the global feature vector of the entire video. To obtain the local feature vector, t frames are extracted from the video. For each frame, the Fast-R-CNN target detector is used to identify and obtain k objects with the highest confidence scores for sampling. For each object, the object image is passed through the ResNet architecture to obtain the local target image features; the object label and coordinates are converted into word vector form by text tokenization and text embedding, and the image feature vector, label word vector, and coordinate word vector corresponding to each object after processing are collectively used as the local feature vector of the object.

6. The text-video cross-modal event element extraction method according to claim 1, characterized in that: Step 4: Unify the vector dimensions of the text word vector and the video global feature vector and local feature vector through a fully connected layer to construct a text-video shared vector space, where: The dimensions of text embedding vectors are unified as (m, 768), the dimensions of video global feature vectors are unified as (1, 768), and the dimensions of video local feature vectors are unified as (n, 768). Based on this, a 768-D space is obtained as the text-video shared vector space.

7. The text-video cross-modal event element extraction method according to claim 1, characterized in that: Step 5: Use the ONEIE algorithm to extract event element information in text mode, and use the T5-base algorithm to extract event element information in video mode, where: The ONEIE algorithm is used to extract event element information in text mode, specifically: Identify entity mentions and event trigger words in sentences as nodes in the information network, that is, first use the feedforward network FFN to generate the intermediate vector w for each text output by the Transformer encoder. i Calculate the corresponding score vector Score vector Each item in represents a word w i For each label, that is, the score of the entity position and entity type to which the word belongs; then use the conditional random field layer to restrict the dependency between the predicted labels and obtain the optimal label for each word; finally, classify the word or phrase as an entity mention or event trigger word as a node in the information network; For the identified nodes that are single words or phrases consisting of multiple words, for the case of multiple words, the lexical representations of all words are averaged to obtain the lexical representation of the multi-word node, and then the feedforward network is used to calculate the v of each node i The score vector for a specific task t Put the edge ij Defined as the relationship between the i-th node and the j-th node, the lexical representations of the two nodes are concatenated to obtain the vector representation of this edge, and the score vector of this edge for a specific task t is calculated for the vector representation of this edge Use the cross entropy loss function to shrink the score vectors of nodes and edges calculated for a specific task t and the target score vector Based on the gap between them, the classification of nodes and edges for tasks is completed, that is, to identify which event the node / edge belongs to; Use beam search to combine each inferred node / edge to complete the event extraction of text data; The T5-base algorithm is used to extract event element information of the video modality, specifically: The video intermediate vector output by the Transformer encoder is directly put into the decoder of the T5-base model as input to complete the event element information extraction of the video modality, and the output text sequence format is defined as: (event type of event 1; the first argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; the second argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; ...; event type of event 2; the first argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; the second argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position box in the frame>; ...) After obtaining the event element information extraction results of the text and video, the arguments in the same event are matched, and the corresponding start and end positions of each argument in the text are added to the output video event extraction result sequence to obtain the final target sequence output.

8. A text-video cross-modal event element extraction system, characterized in that: Implementing the text-video cross-modal event element extraction method described in any one of claims 1 to 7 to realize text-video cross-modal event element extraction includes five modules, specifically: Module 1: Collect video data and its video introduction text data, annotate the event types and corresponding event argument roles of the text and video data respectively, where the event argument roles represent entities that play different roles in the event, and the pre-annotated event types and event argument roles corresponding to the event types in the text data and video data are consistent; Module 2: Perform multimodal event reference resolution to achieve co-referential event pairing between any "text-video" data, that is, match the text and video referring to the same event to form a set of text-video co-referential event pairs; Module 3: Convert the matched "text-video" data into corresponding feature vectors, where text data is tokenized and embedded into word vectors; the ResNet algorithm is used directly on the video data to obtain the global level event element features and construct the video global feature vector; the Fast-R-CNN is used to identify local objects in the video data, and the ResNet algorithm is used to obtain the local level time elements and construct the video local feature vector; Module 4: Unify the vector dimensions of text word vectors, video global feature vectors, and local feature vectors through a fully connected layer to construct a text-video shared vector space; Module 5: Input the text word vector and the video global feature vector and local feature vector into the Transformer encoder, and then use the ONEIE algorithm to extract event element information in the text mode, and use the T5-base algorithm to extract event element information in the video mode.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the text-video cross-modal event element extraction method according to any one of claims 1 to 7 is implemented to realize text-video cross-modal event element extraction.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the text-video cross-modal event element extraction method according to any one of claims 1 to 7 is implemented to achieve text-video cross-modal event element extraction.

Images