A text-video cross-modal event element extraction method

Abstract

The application discloses a text-video cross-modal event element extraction method, collects video data and video introduction text data thereof, respectively labels event types and corresponding event argument roles of the text and video data, wherein the event argument role represents entities playing different roles in events, the event types and the event argument roles corresponding to the event types of the text data and the video data are pre-labeled to be consistent; multi-modal event reference resolution is carried out to realize co-reference event pairing between any "text-video" data, that is, text and video referring to the same event are matched to form a text-video co-reference event pair; the matched "text-video" data is converted into corresponding feature vectors, wherein text tokenization and text embedding are performed on the text data to be converted into a word vector form; ResNet algorithm is directly used on the video data to obtain global level event element features to construct a video global feature vector; Fast-R-CNN is used on the video data to identify local objects, ResNet algorithm is used to obtain local level time elements to construct a video local feature vector; the text word vector and the video global feature vector and the local feature vector are unified in vector dimension through a full connection layer to construct a text-video shared vector space; the text word vector and the video global feature vector and the local feature vector are input into a Transformer encoder, then ONEIE algorithm is adopted to extract event element information of the text mode, and T5-base algorithm is adopted to extract event element information of the video mode. The application can more accurately capture the correlation between the internal of multi-modal, and improves the extraction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information processing, in particular to a text-video cross-modal event element extraction method. BACKGROUND

[0002] The purpose of event extraction is to extract event-related element data from unstructured raw data and structure it. Event extraction mainly covers event type detection, event argument extraction, and event argument role extraction. An event is usually composed of an event trigger word and elements that describe the structure of the event, where the event trigger word refers to the occurrence of the event, and the event argument and argument role are the components of the event. Generally, the difference between the argument and the argument role is that the argument refers to the entity type of the event element, while the argument role refers to the role played by the event element in the event.

[0003] Currently, the common event extraction technology can usually only extract events from text format data, while for information carriers such as images and videos that contain more rich semantic information, the event information they contain is more rich than text format data, and can express event elements that cannot be directly expressed by text data. There are few event extraction technologies for this type of modality, and they can only be used for a single mode. However, information publishing media in the real world usually publish content in multiple modalities, such as a news report composed of text and pictures, or a video with a video-related content introduction. These modal data together constitute the description of the event. Therefore, event extraction from single modal data may not be complete due to the limitations of the data format itself. SUMMARY

[0004] The purpose of the present application is to provide a text-video cross-modal event element extraction method to enrich the event element extraction results of text-video multi-modal event data and improve the completeness of event extraction.

[0005] The technical solution for achieving the purpose of the present application is a text-video cross-modal event element extraction method, comprising the following steps:

[0006] Step 1: Collect video data and its video introduction text data, and label the event type and corresponding event argument role of the text and video data respectively, where the event argument role represents entities playing different roles in the event, and the event type and event argument role of the text data and video data are pre-labeled consistent with the event type and event argument role corresponding to the event type;

[0007] Step 2: Perform multi-modal event resolution to realize co-reference event pairing between any "text-video" data, i.e. matching text and video that refer to the same event to form a set of text-video co-reference event pairs;

[0008] Step 3: Convert the matched "text-video" data into corresponding feature vectors, where text tokenization and text embedding are performed on the text data, transforming it into word vector form; ResNet algorithm is directly used on the video data to obtain global-level event element features, constructing a video global feature vector; Fast-R-CNN is used to identify local objects in the video data, and ResNet algorithm is used to obtain local-level time elements, constructing a video local feature vector;

[0009] Step 4: Unify the vector dimensions of the text word vector and the video global feature vector and local feature vector through a fully connected layer, constructing a text-video shared vector space;

[0010] Step 5: Input the text word vector and the video global feature vector and local feature vector into the Transformer encoder, and then use the ONEIE algorithm for event element information extraction in the text mode and the T5-base algorithm for event element information extraction in the video mode;

[0011] Step 6: Based on the training process of steps 3-5 above, a text-video cross-modal event extraction model is obtained.

[0012] Further, in step 1, long videos with a duration exceeding 15 minutes are filtered out, and the filtered videos and their introduction texts are checked to ensure that the video content contains at least one visually observable event, the text content contains at least one event, and at least one text-video event matching pair.

[0013] Further, in step 1, when a video contains multiple event types, the video data needs to be labeled with all event types it contains and the start and end times of each event in the video. For each event segment in the video, at least three frames need to be selected, and two to three entities need to be labeled as the argument roles of the event.

[0014] Further, in step 2, multi-modal event coreference resolution is performed to realize coreferential event pairing between any "text-video" data, i.e., matching text and video that refer to the same event to form a set of text-video coreferential event pairs. The specific method is:

[0015] Construct a standard noise contrast loss NCE:

[0016]

[0017] where represents the sentence x ia matched video segment; f and g are fully connected layer networks that project sentences and video segments into a common vector space, realizing the conversion of sentences and video segments into latitude-consistent feature vectors; n represents the number of all text-video pairs for judgment; negative sample set representing "text-video" mismatch; e is the natural base; (x', y') represents belonging to the negative sample set "Text-video" pairs that cannot be matched;

[0018] Constructing object-based multi-instance learning loss

[0019]

[0020] wherein x represents a sentence; z represents a certain object appearing in a certain frame of a video; f and h are fully connected layer networks that project sentences and objects into a common vector space, realizing the conversion of sentences and local objects into latitude-consistent feature vectors; positive sample set representing "text-object" that can be matched in relation to a video segment;

[0021] Constructing the final multi-modal event coreference resolution loss function mmcoref:

[0022]

[0023] Based on the constructed multi-modal event coreference resolution loss function mmcoref, the fully connected layer networks f, g, h in the loss function are trained, and any text and video data can be provided to determine whether they belong to the same event.

[0024] Further, step 3: converting the matched "text-video" data into corresponding feature vectors, wherein:

[0025] For text data, text tokenization and text embedding are performed to convert all words in the input sentence into word vector form, and if a word is divided into multiple word segments, the average value of all segment vectors is used as the vector representation of the word;

[0026] (1) Global event context: extracted from the entire video segment using the ResNet algorithm;

[0027] (2) Frame-level object label: generated by the Fast-R-CNN object detector, indicating the argument role to which the object belongs;

[0028] (3) Frame-level region feature: using the ResNet algorithm to extract images from the bounding box detected by the Fast-R-CNN object detector;

[0029] (4) Frame-level target coordinates: the bounding box coordinates provided by the Fast-R-CNN target detector are used for the positioning of parameters;

[0030] The entire video segment is put into the ResNet architecture to obtain the global feature vector of the entire video. For the acquisition of the local feature vector, t frames are extracted from the video. For each frame, the Fast-R-CNN target detector is used to identify and acquire k objects with the highest confidence score for sampling. For each object, the object image is passed through the ResNet architecture to obtain the local target image feature. The label and coordinates of the object are text tokenized and text embedded to convert them into word vector form. The image feature vector, label word vector, and coordinate word vector of each processed object are collectively used as the local feature vector of the object.

[0031] Further, step 4: the text word vector and the global feature vector and the local feature vector of the video are unified in vector dimension through a fully connected layer, and a text-video shared vector space is constructed, wherein:

[0032] The dimension of the text embedding vector is unified to (m, 768), the dimension of the global feature vector of the video is unified to (1, 768), and the dimension of the local feature vector of the video is unified to (n, 768). Accordingly, a 768-D space is obtained as the text-video shared vector space.

[0033] Further, step 5: the ONEIE algorithm is used for event element information extraction of the text mode, and the T5-base algorithm is used for event element information extraction of the video mode, wherein:

[0034] The ONEIE algorithm is used for event element information extraction of the text mode, specifically:

[0035] Discriminate the entity mention and event trigger word in the sentence as the node in the information network, that is, first use the feedforward network FFN to obtain the score vector of each text intermediate vector w i The corresponding score vector is calculated The score vector Each item in the score vector represents the score of the word w i The score of the word belonging to the entity position and entity type; then use the conditional random field layer to limit the dependency relationship between the predicted labels, and obtain the optimal label of each word; finally, classify the word or phrase as an entity mention or an event trigger word as a node in the information network;

[0036] For identified nodes that are single words or phrases composed of multiple words, in the case of multiple words, the lexical representations of all words are averaged to obtain the lexical representation of the multi-word node. Next, a feedforward network is used to calculate the v of each node. i Score vector for a specific task t

[0037] Put the edge e ij The edge is defined as the relationship between the i-th node and the j-th node. The vector representation of this edge is obtained by concatenating the lexical representations of the two nodes. The score vector of this edge for a specific task t is then calculated from its vector representation.

[0038] The cross-entropy loss function is used to reduce the calculated score vector of nodes and edges for a specific task t. and target score vector Based on the differences between them, the nodes and edges are classified for the tasks, that is, the node / edge is identified as belonging to which event;

[0039] Beam search is used to combine each inferred node / edge to complete the event extraction of text data;

[0040] The T5-base algorithm is used to extract event element information from video modalities, specifically as follows:

[0041] The intermediate video vector output from the Transformer encoder is directly fed into the decoder of the T5-base model to extract event element information from the video modality. The output text sequence format is defined as follows: (Event type of event 1; first argument of event 1 <argument type, video in which the argument is located, frame in which the video is located, bounding box in the frame>; second argument of event 1 <argument type, video in which the argument is located, frame in which the video is located, bounding box in the frame>; ...; event type of event 2; first argument of event 2 <argument type, video in which the argument is located, frame in which the video is located, bounding box in the frame>; second argument of event 2 <argument type, video in which the argument is located, frame in which the video is located, bounding box in the frame>; ...)

[0042] After obtaining the event element information extraction results from the text and video, the arguments in the same event are matched, and the start and end positions of each argument in the text are added to the output video event extraction result sequence to obtain the final target sequence output.

[0043] A text-video cross-modal event element extraction system is provided, which implements the text-video cross-modal event element extraction method to achieve text-video cross-modal event element extraction.

[0044] The computer device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the text-video cross-modal event element extraction method and realize text-video cross-modal event element extraction.

[0045] A computer readable storage medium has a computer program stored thereon, wherein the computer program is executed by a processor to implement the text-video cross-modal event element extraction method and realize text-video cross-modal event element extraction.

[0046] Compared with the prior art, the present application has the following advantages: 1) scene perception and context understanding are combined to realize effective fusion of multi-modal data and accurate extraction of event information; 2) based on text and video data, more event parameters can be extracted, including information conveyed by static images and context information of each static image; and 3) when existing image events cannot be directly borrowed, a new multi-modal transformer structure is used to dynamically focus on the most relevant part of the text description and video content, more accurately capture the relevance between multi-modal internals, and improve the extraction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of the text-video cross-modal event element extraction method. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0049] The present application introduces a self-supervised model and a multi-modal transformer architecture on the basis of event extraction technology of single text modality and single video modality. The self-supervised model is used to find text sentences and video segment pairs describing the same event, extract sentence-video co-reference segment pairs in the text-video modality, and extract events therefrom. The multi-modal transformer architecture is used to perform event classification and event element extraction from the video and the text. The specific implementation steps are as follows:

[0050] Step 1: Collect video data and its text data (text data is the video introduction related to the video content) of the vertical subdivision field that the text-video cross-modal event extraction model plans to solve. For the collected data, filter out long videos with a duration of more than 15 minutes to avoid information explosion that may be caused; check the filtered video and its introduction text to ensure that the video content contains at least one visually observable event, the text content contains at least one event, and at least one text-video event matching pair.

[0051] The text and video data are labeled using an image labeler. The text data is labeled with event types and event argument roles. Considering that a video often contains more than one event, the video data needs to be labeled with all event types it contains and the start and end times of each event in the video. For each event segment in the video, at least three frames need to be selected, and each frame needs to be labeled with two to three typical entities (such as people, objects, etc.) as the argument roles of the event. The pre-labeled event types and event argument roles of the text data and the video data should be consistent, that is, the text data labels which event types, the video data should also have corresponding event types labeled, and the text data labels which event argument roles, the video data should also have corresponding event argument roles labeled.

[0052] The event types / event argument roles labeled by the text data and the event types / event argument roles labeled by the video data constitute the text-video multi-modal co-reference event pair. The text-event co-reference relationship can be marked as 1:1, 1:n, n:1 and n:n, where 1:1 means that one text event is related to only one video event; 1:n means that one text event is related to multiple video events, n:1 means that multiple text events are related to one video event, and n:n means that multiple text events are related to multiple video events. The given event types and event argument roles here are the event elements that the text-video cross-modal event extraction needs to extract.

[0053] Step 2: Perform the Multimodal Event Coreference Resolution task. Through this step, co-reference event pairing between any "text-video" data can be achieved, that is, whether the text and the video belong to the same event, the text and the video can match, and refer to the same event, are positive sample pairs, otherwise are negative sample pairs.

[0054] Construct a standard noise contrast loss (NCE), the formula is as follows:

[0055]

[0056] Where x i represents a sentence; yi representing the ability to match with x i representing the ability to match with x representing the ability to match with x representing the ability to match with x i ,y i ), can achieve coarse-grained global event coreference resolution between “text-video” data.

[0057] In addition, some local objects appearing in the video will also affect the matching of “text-video” (for example, a gun appearing in a frame of the video and the word gun in the text are matched). Therefore, an object-based multi-instance learning loss

[0058]

[0059] where x represents the sentence. z represents a certain object appearing in a frame of the video (actually a part of the image cut from a frame). f and h are fully connected layer networks that project the sentence and the object into a common vector space, realizing the conversion of the sentence and the local object into feature vectors with consistent latitude. Given a video segment, representing the ability to match with x representing the ability to match with x representing the ability to match with x can make the objects in the video instance i better match the text sentences, realizing finer-grained local event coreference resolution.

[0060] The final multi-modal event coreference (mmcoref) loss function is constructed, that is, the sum of which realizes global matching, and which realizes local matching.

[0061]

[0062] The fully connected layer networks f, g, h in the training loss function are trained, and given any text and video data, it can be determined whether they belong to the same event.

[0063] Step 3: For the "text-video" pairs that have been paired in Step 2, data preprocessing is performed to convert them into corresponding feature vectors, respectively.

[0064] For text data, text tokenization and text embedding are performed to convert all words in the input sentence into word vector form. If a word is divided into multiple word fragments, the average of all fragment vectors is used as its word representation.

[0065] For video data, the following four feature types are used to comprehensively represent global and local information:

[0066] (1) Global event context: extracted by the entire video segment using the ResNet algorithm;

[0067] (2) Frame-level object label: generated by the Fast-R-CNN object detector, indicating the argument role to which the detected object belongs;

[0068] (3) Frame-level object feature: using the ResNet algorithm, the object feature extracted from the object bounding box detected by the Fast-R-CNN object detector, providing fine-grained local object feature information;

[0069] (4) Frame-level object coordinates: provided by the Fast-R-CNN object detector, indicating the position information of the detected object bounding box, used for parameter positioning.

[0070] Then, t frames are sampled for each video segment (t = 3), and for each frame, the object detector is used to take k target entity objects with the highest confidence score. The global-level event element features of the video are extracted by the ResNet architecture. The label, image, and bounding box coordinates of the frame-level event element object are obtained by the Fast-R-CNN recognition, where the image is extracted by the ResNet architecture, and the label and bounding box coordinates are converted into word vector feature form by text tokenization and text embedding.

[0071] Step 4: Based on the vectorized text data and video global feature vector and local feature vector obtained in step 3, the vector dimensions of the three are unified through a fully connected layer in the form of feature vectors. The dimension of the text embedding vector is unified to (m, 768), where m represents the number of tokens. The dimension of the video global feature vector is unified to (1, 768); the dimension of the video local feature vector is unified to a set of (n, 768). Each local feature extracted from the video corresponds to a (n, 768) feature vector, which can be further divided into (n1, 768), (1, 768), (n2, 768), i.e. n = n1 + 1 + n2. n1 in (n1, 768) represents the number of tokens of the target label; (1, 768) represents the target image feature; n2 in (n2, 768) represents the number of tokens of the target bounding box coordinates. A 768-dimensional space is constructed as a text-video shared vector space.

[0072] Step 5: Based on step 4, the text vector and video vector with unified dimensions are respectively input into the Transformer encoder. The output text and video intermediate vectors after the Transformer encoder have the same latitude as the original vectors. The output text vector uses the ONEIE joint neural network framework for text element information extraction; the extraction of video event elements uses the T5-base model.

[0073] The ONEIE framework usually extracts information networks from a given sentence in four steps: encoding, recognition, classification, and decoding. The encoding step usually uses a pre-trained BERT encoder to encode all the words in the input sentence into word representations. The text vector has been obtained through the previous step, so the intermediate vector output by the Transformer is directly put into the recognition step as input.

[0074] The goal of the recognition step is to identify entity mentions and event triggers in the sentence as nodes in the information network. First, a feedforward network FFN is used to obtain a score vector for each word representation w i The corresponding score vector is calculated Each item in the score vector represents the word w iThe scores for each tag are calculated (here, the BIO serialization tagging method is used to tag each word in the sentence: B represents the beginning of a named entity, and words tagged with B are at the beginning of a named entity; I represents the inside of a named entity, and words tagged with I are inside a named entity; O represents the outside of a named entity, and words tagged with O do not belong to any named entity). Then, a Conditional Random Field (CRF) layer is used to constrain the dependencies between predicted tags (e.g., the I-PER tag must follow the B-PER tag) and finally, the optimal tag for each word is obtained. After obtaining the tags, some words or phrases can be classified as entity mentions or event trigger words. The identified entity mentions and event trigger words are used as nodes in the information network. For the identified nodes, they may be single words or phrases composed of multiple words. For the case of multiple words, the lexical representations of all words are averaged to obtain the lexical representation of the multi-word node. Next, a feedforward network is used to calculate the v of each node. i Score vector for a specific event type t Put the edge e ij This is defined as the relationship between the i-th node and the j-th node. To obtain a certain edge e... ij The score vector is obtained by concatenating the word representations of the two nodes to get the vector representation of the edge. Then, a similar method is used to calculate the score vector of the edge for a specific task t. Finally, the cross-entropy loss function is used. (N t The total number of nodes and edges in a specific task t is used to narrow down the calculated score vector of nodes and edges for that specific task t. and target score vector The difference between them. Through this step, the classification of nodes and edges for tasks is completed, that is, identifying which event a node / edge belongs to.

[0075] Beam search is used to combine each inferred node / edge to complete the event extraction from the text data. The text event extraction result is in a structured format: (Event type 1: [argument 1 <word position, entity>, argument 2 <word position, entity>, argument 3 <word position, entity>]; Event type 2: [argument 1 <word position, entity>, argument 2 <word position, entity>, argument 3 <word position, entity>]; ...).

[0076] T5-base model:

[0077] The T5 (Text-to-Text Transfer Transformer) model is used to perform video event extraction, and the event extraction task of the video modality is regarded as a Sequence-to-Sequence task in a unified encoder-decoder framework. The architecture is basically the same as the original Transformer architecture, and the main differences are as follows: (1) eliminate the specification deviation of the convolutional layer; (2) place the convolutional layer normalization outside the residual path; (3) use a different position encoding method.

[0078] The video intermediate vector output by the Transformer encoder is directly put into the decoder of the T5-base model as input to complete the decoding step of the video event extraction end. Since the output of T5 is a continuous text sequence, the output text sequence format is specified as follows: (event type of event 1; first argument of event 1 <argument type, video where the argument is located, frame in the video where the argument is located, Bbox position box in the frame>; second argument of event 1 <argument type, video where the argument is located, frame in the video where the argument is located, Bbox position box in the frame>; …; event type of event 2; first argument of event 2 <argument type, video where the argument is located, frame in the video where the argument is located, Bbox position box in the frame>; second argument of event 2 <argument type, video where the argument is located, frame in the video where the argument is located, Bbox position box in the frame>; …).

[0079] After obtaining the event extraction results of the text and the video, the arguments in the same event are matched. The start and end positions of each argument in the text are added to the output video event extraction result sequence. The final target sequence output is obtained.

[0080] Step 6: Based on the training process of steps 3-5 above, a text-video cross-modal event extraction model is obtained.

[0081] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present application.

[0082] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A text-video cross-modal event element extraction method, characterized in that, Comprising the following steps: Step 1: Collect video data and its video introduction text data, respectively label the event type and corresponding event argument role of the text and video data, wherein the event argument role represents the entity playing different roles in the event, the pre-labeled event type of the text data and the video data is consistent with the event argument role corresponding to the event type; Step 2: Multi-modal event coreference resolution is performed to realize the co-reference event pairing between any "text-video" data, that is, matching the text and video referring to the same event to form a set of text-video co-reference event pairs; Step 3: Convert the matched "text-video" data into corresponding feature vectors, wherein text tokenization and text embedding are performed on the text data to convert it into a word vector form; ResNet algorithm is directly used on the video data to obtain global level event element features and construct a video global feature vector; Fast-R-CNN is used to identify local objects in the video data, and ResNet algorithm is used to obtain local level time elements and construct a video local feature vector; Step 4: The text word vector and the video global feature vector and the local feature vector are unified in vector dimension through a fully connected layer to construct a text-video shared vector space; Step 5: The text word vector and the video global feature vector and the local feature vector are input into the Transformer encoder, and then the ONEIE algorithm is used for event element information extraction in the text mode, and the T5-base algorithm is used for event element information extraction in the video mode; Step 6: Based on the training process of steps 3-5 above, a text-video cross-modal event extraction model is obtained; Step 2, multi-modal event coreference resolution is performed to realize the co-reference event pairing between any "text-video" data, that is, matching the text and video referring to the same event to form a set of text-video co-reference event pairs, the specific method is: Construct a standard noise contrast loss NCE: where y i represents a video segment that can match the sentence x i ; f and g are fully connected layer networks that project the sentence and the video segment into a common vector space, realizing the conversion of the sentence and the video segment into feature vectors with the same latitude; n represents the number of all text-video pairs used for judgment; represents a negative sample set of "text-video" that does not match; e is the natural base; (x', y') represents a "text-video" pair that cannot match belonging to the negative sample set . Object-based multi-instance learning loss construction Wherein, x represents a sentence; z represents a certain object appearing in a certain frame of a video; f and h are fully connected layer networks for projecting the sentence and the object into a common vector space, so as to convert the sentence and the local object into feature vectors with consistent latitude; represents a set of "text-object" positive samples capable of matching related to a video segment; Construct the final multi-modal event coreference resolution loss function mmcoref: Based on the constructed multi-modal event coreference resolution loss function mmcoref, the fully connected layer networks f, g, h in the loss function are trained, and any text and video data can be provided to determine whether they belong to the same event.

2. The text-video cross-modal event element extraction method of claim 1, wherein, In step 1, long videos with a duration of more than 15 minutes are filtered out, and the filtered videos and their introduction texts are checked to ensure that the video content contains at least one visually observable event, the text content contains at least one event, and at least one text-video event matching pair.

3. The text-video cross-modal event element extraction method of claim 1, wherein, In step 1, when a video contains multiple event types, the video data needs to be labeled with all event types and the start and end time of each event in the video, and at least three frames need to be selected from each event segment in the video, and two to three entities are labeled as argument roles of the event.

4. The text-video cross-modal event element extraction method of claim 1, wherein, Step 3: Convert the matched "text-video" data into corresponding feature vectors, wherein: For text data, text tokenization and text embedding are performed to convert all words in the input sentence into word vector form, and if a word is divided into multiple word fragments, the average value of all fragment vectors is used as the vector representation of the word; For video data, the following four feature types are used to comprehensively represent global and local information: (1) Global event context: extracted from the entire video segment using the ResNet algorithm; (2) Frame-level object label: generated by the Fast-R-CNN object detector, indicating the argument role to which the object belongs; (3) Frame-level region feature: image extracted from the bounding box detected by the Fast-R-CNN object detector using the ResNet algorithm; (4) Frame-level object coordinates: the bounding box coordinates provided by the Fast-R-CNN object detector are used for parameter positioning; Put the entire video segment into the ResNet architecture to obtain the global feature vector of the entire video, and for the acquisition of local feature vectors, t frames are extracted from the video, for each frame, the Fast-R-CNN object detector is used to identify and sample k objects with the highest confidence score, and for each object, the object image is passed through the ResNet architecture to obtain the local object image feature; The label and coordinates of the object are converted into word vector form by text tokenization and text embedding, and the image feature vector, label word vector, and coordinate word vector corresponding to each object after processing are collectively used as the local feature vector of the object.

5. The text-video cross-modal event element extraction method of claim 1, wherein, Step 4: The text word vector and the video global feature vector and local feature vector are unified in vector dimension through a fully connected layer, and a text-video shared vector space is constructed, wherein: The dimension of the text embedding vector is unified to (m, 768), the dimension of the video global feature vector is unified to (1, 768), and the dimension of the video local feature vector is unified to (n, 768). A 768-D space is obtained as the text-video shared vector space.

6. The text-video cross-modal event element extraction method of claim 1, wherein, Step 5: ONEIE algorithm is used for event element information extraction of text modal, and T5-base algorithm is used for event element information extraction of video modal, wherein: ONEIE algorithm is used for event element information extraction of text modal, specifically: distinguishing entity mentions and event triggers in a sentence as nodes in an information network, i.e. first using a feed-forward network FFN to predict scores for each text intermediate vector w i computing a corresponding score vector the score vector each entry in the score vector represents a word w i scores for each label, i.e. entity position and entity type, to which the word belongs; then using a conditional random field layer to restrict the dependencies between the predicted labels, obtaining the optimal label to which each word belongs; finally classifying the words or phrases as entity mentions or event triggers as nodes in an information network; For the identified nodes being either a single word or a phrase of multiple words, for the case of multiple words, the lexical representation of all the words is averaged to obtain the lexical representation of the multi-word node, next the forward network is used to compute the score vector for each node v i The score vector for a particular task t edge e ij defined as the relation between the ith node and the jth node, concatenating the lexical representations of the two nodes to obtain the vector representation of this edge, computing the score vector of this edge for a particular task t on the vector representation of this edge The cross-entropy loss function is used to reduce the calculated score vector of nodes and edges for a specific task t. and target score vector Based on the differences between them, the nodes and edges are classified for the tasks, that is, the node / edge is identified as belonging to which event; Beam search is used to combine each node / edge inferred to complete the event extraction of text data; T5-base algorithm is used for event element information extraction of video modal, specifically: The video intermediate vector output by the Transformer encoder is directly put into the decoder of the T5-base model as input to complete the event element information extraction of the video modality, and the output text sequence format is specified as: (event type of event 1; first argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; second argument of event 1 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; …; event type of event 2; first argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; second argument of event 2 <argument type, video where the argument is located, frame where the video is located, Bbox position frame in the frame>; …) After obtaining the event element information extraction results of the text and the video, the arguments in the same event are matched, and the start and end positions of each argument in the text are added to the output video event extraction result sequence to obtain the final target sequence output.

7. A text-video cross-modal event element extraction system, characterized in that, The text-video cross-modal event element extraction method of any one of claims 1-6 is implemented to realize text-video cross-modal event element extraction, which includes five modules, specifically: Module 1: Collect video data and its video introduction text data, and label the event type and corresponding event argument roles of the text and video data, wherein the event argument role represents an entity playing a different role in an event, and the text data and the video data are pre-labeled with consistent event type and event type corresponding event argument roles; Module 2: Perform multi-modal event anaphora resolution to realize co-reference event pairing between any "text-video" data, that is, matching text and video referring to the same event to form a set of text-video co-reference event pairs; Module 3: Convert the matched "text-video" data into corresponding feature vectors, wherein text tokenization and text embedding are performed on the text data to convert it into a word vector form; ResNet algorithm is directly used on the video data to obtain global level event element features to construct a video global feature vector; Fast-R-CNN is used to recognize local objects from the video data, and ResNet algorithm is used to obtain local level time elements to construct a video local feature vector; Module 4: Uniform the vector dimensions of the text word vector and the video global feature vector and local feature vector through a fully connected layer to construct a text-video shared vector space; Module 5: Input the text word vector and the video global feature vector and local feature vector into the Transformer encoder, and then use the ONEIE algorithm to extract event element information of the text modality and use the T5-base algorithm to extract event element information of the video modality.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, a text-video cross-modal event element extraction method according to any one of claims 1-6 is implemented to realize text-video cross-modal event element extraction.

9. A computer readable storage medium, having stored thereon a computer program, wherein when the computer program is executed by a processor, a text-video cross-modal event element extraction method according to any one of claims 1-6 is implemented to realize text-video cross-modal event element extraction.

Images