A multimedia event extraction method based on multimodal low-dimensional feature representation space
By constructing a multimodal low-dimensional feature representation space and momentum distillation method, the problem of noise influence in multimedia event extraction is solved, achieving efficient extraction of text and visual events and improving the accuracy and recall of multimedia event extraction.
Patent Information
- Application Number
- CN202411213679.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-30
AI Technical Summary
Existing multimedia event extraction methods are severely affected by dataset noise when dealing with multimodal data, leading to a decline in model extraction performance. Furthermore, single-modal feature fusion methods cannot effectively improve the accuracy of multimedia event extraction.
We adopt a method based on multimodal low-dimensional feature representation space. By constructing a Transformer encoder and momentum distillation model, we learn the matching of images and text. We use multimodal low-dimensional feature representation space to map text and images to a unified dimension, and introduce momentum distillation method to reduce the influence of noise. Combined with contrastive learning, we improve the matching ability of the model.
It effectively reduced the impact of dataset noise on the model, improved the performance of multimedia event extraction, increased the F1 score of text event extraction by 1.0% to 1.7%, and improved various indicators of visual event extraction to varying degrees, with significant improvements in recall and accuracy.
Smart Images

Figure CN118762261B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of multimedia event extraction, and in particular relates to a multimedia event extraction method based on a multimodal low-dimensional feature representation space. Background Art
[0002] The event extraction task is an important and challenging information extraction research. The event extraction task is to identify the event type and event arguments of an event from unstructured data and form structured data, providing basic data for event-related applications such as information retrieval, intelligent recommendation, and event knowledge graph construction. The event type is generally a verb that causes the event to occur, also known as a trigger word. Event arguments include the time and place of the event, as well as the participants related to the event, such as the initiator, receiver, and tool.
[0003] Previous research on event extraction only focused on a single modality. Whether extracting event information from text data or image data, it ignored the complementary role of event information provided by multiple modal data. Research on multimedia news event extraction started late. Due to the high cost of data annotation, there is still a lack of data sets that can support supervised training of models. There is only a small multimedia event extraction data set M2E2 for testing. This data set is not enough to support model training. Therefore, current research uses weak supervision methods to train models. Specifically, the model construction process is to collect image captions through the network to build a feature representation space that can represent text and images at the same time. The purpose is to achieve alignment of the two modalities of text and image. At the same time, the text data set and image data set are used to train text events and image events to extract models respectively. After a simple fusion of text and image with the help of feature representation space, they are used for text event extraction and image event extraction respectively.
[0004] Although researchers continue to explore new event extraction methods to improve the accuracy of multimedia event extraction, weakly supervised methods are inevitably affected by noise in the data. First, the image titles collected from the Internet have a lot of noise problems in the dataset, and weakly related image titles often have a negative impact on subsequent extraction. Secondly, from the perspective of text, multimedia event extraction research pays more attention to whether a certain word in the article is a trigger word or event argument of the event type. From the perspective of images, it pays more attention to whether a certain visual area in the image involves an event type or event argument. Simple vector fusion methods often fuse a piece of text with an image as a vector. This method not only fails to improve the performance of the model, but also introduces noise from another modality, thereby affecting the extraction of event information. Summary of the invention
[0005] In response to the problems existing in the above-mentioned background technology, the present invention provides a multimedia event extraction method based on a multimodal low-dimensional feature representation space, which aims to weaken the impact of data set noise on model extraction performance during model construction during multimedia event extraction, and improve the performance of the model in multimedia event extraction.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A multimedia event extraction method based on a multimodal low-dimensional feature representation space comprises the following steps:
[0008] S1. Constructing a multimodal low-dimensional feature representation space
[0009] A multimodal low-dimensional feature representation space is constructed using the VOA image-caption pair dataset; when training the multimodal low-dimensional feature representation space model, for a given image-caption pair, a Transformer-based encoder is constructed for the image set and the text set respectively, and two queues are constructed at the same time for storing the latest image-caption pairs. A momentum encoder is created based on the image and text encoders, and the momentum distillation method is used to learn the matching of images and texts from the pseudo targets generated by the momentum distillation model. The momentum encoder is used to calculate the similarity between images and texts; thereby mapping the matched text and image to similar areas of the multimodal low-dimensional feature representation space; through the multimodal low-dimensional feature representation space, the text and image of the test data are mapped to multimodal low-dimensional features of unified dimension, so as to realize the fusion of different modal features and facilitate the calculation of the similarity between text and image;
[0010] While using contrastive learning to improve the model's ability to match text and images, the momentum distillation method is introduced to reduce the impact of image captions on the noise in the dataset on the model; the multimodal low-dimensional feature representation space can represent text and image features as a unified dimension, thereby achieving feature fusion of text and image;
[0011] S2. Text Event Extraction
[0012] The ACE2005 text event dataset is used as training data for the text event extraction model; five images that best match each text description in the ACE2005 text event dataset are selected from the VOA image title pair dataset as additional image information of the text event; each word in the text description is initialized with feature representation through DeBERTaV3 to obtain a vector representation of each word; the image involved in the additional image information is encoded into an image low-dimensional vector representation through a multimodal low-dimensional feature representation space, and the image low-dimensional vector representation is weightedly fused according to the similarity weight to obtain an image representation that integrates the image information, and then the vector representation of each word is spliced with the image representation that integrates the image information and the low-dimensional vector representation of the sentence to obtain a word representation that incorporates the image information, thereby completing the text event extraction;
[0013] S3. Image event extraction
[0014] The image-related text description, i.e., image description, is generated through the ViECap image description technology. After the image events, related text events and the generated image description are encoded in a multimodal low-dimensional feature representation space, the similarity scores of the image events, related text events and image descriptions with the pre-defined visual event types and visual event arguments are calculated using cosine similarity. The visual event types and visual event arguments with the highest similarity scores are obtained through weighted fusion. After obtaining the event arguments, the image is forward-transferred, gradient-based attention integration and correlation graph calculation are performed, so as to frame the relevant event arguments in the image.
[0015] Furthermore, in step S1, in order to obtain a multimodal low-dimensional feature representation space that can unify the dimensional representation of the text and image of the test data, it is first necessary to use the CLIP ViT-B / 32 version to initialize the parameters of the multimodal low-dimensional feature representation space model and the momentum distillation model respectively, and then use the VOA image caption to pre-train the multimodal low-dimensional feature representation space of the text and image of the dataset, and use the pseudo target generated by the momentum distillation model to supervise the basic model; finally, use the imSitu dataset to fine-tune the pre-trained dataset, and use the momentum distillation model to supervise the fine-tuning model.
[0016] Furthermore, in step S2, the method for obtaining the vector representation of the word is: each sentence representation in the sentence collection of the multimedia event is passed through the DeBERTaV3 encoder to obtain the type embedding representation, subword embedding representation and absolute position embedding representation of each word, the type embedding representation and the subword embedding representation are fused as the content representation vector of the word, and then the attention and relative position information between words are calculated with the absolute position embedding representation through the attention decoupling mechanism to obtain the vector representation of each word.
[0017] Furthermore, in step S2, the similarity weight is calculated by cosine similarity, and the formula is as follows:
[0018]
[0019] In the formula, I j Represents a collection of images I The j images i j The low-dimensional vector representation after multimodal low-dimensional feature representation space encoding, S i It is a collection of sentences S The i Sentences i i The low-dimensional vector representation after multimodal low-dimensional feature representation space encoding, α i,j is an image i j With sentence i i The similarity weight of the image information is then represented by H Image The formula is as follows:
[0020] .
[0021] Furthermore, in step S3, the method of generating image description using the ViECap image description technology is as follows:
[0022] Each image in the image collection is converted into a visual embedding through the image encoder of the ViECap model, and the visual embedding is used to generate soft prompts using the converter; at the same time, the ViECap model uses a CLIP-based entity classifier to perform zero-shot image entity retrieval, and selects certain entities by calculating the correlation between the image and the entity to construct entity-aware hard prompts; finally, the soft prompts and hard prompts are connected in sequence and input into the language model for image description prediction to obtain image descriptions of the multimedia image collection.
[0023] Furthermore, in step S3, the method for identifying the visual event type is as follows: the active verb set and the visual event type in the imSitu dataset are made into a mapping table, and the cosine similarity of all active verbs and the image is calculated through the common space to obtain the active verb that is most similar to the image; then the cosine similarity of the active verb and the text event and the image description is calculated respectively; finally, the three cosine similarity scores are assigned different weights for weighted fusion to obtain the active verb that matches the visual image, text event and image description text at the same time, and the visual event type of the active verb can be obtained according to the mapping table of the active verb set and the visual event type.
[0024] Furthermore, in step S3, the method for identifying the visual event argument is as follows: the semantic role set and the visual event argument in the imSitu dataset are made into a mapping table, and then the cosine similarities of all semantic roles and visual images, text events and image descriptions are calculated through a common space, and finally the three cosine similarity scores are assigned different weights for weighted fusion to obtain a semantic role that simultaneously matches the visual image, text event and image description text, and the visual event argument of the semantic role can be obtained according to the mapping table between the semantic role set and the visual event argument.
[0025] Compared with the shortcomings and deficiencies of the prior art, the present invention has the following beneficial effects:
[0026] (1) The present invention extracts multimedia events based on a multimodal low-dimensional feature representation space, and improves the performance of event extraction by integrating relevant event information of different modalities. In terms of text event extraction, by fusing the low-dimensional representation vector of the relevant image with each word, the F1 values of event type and event argument extraction are increased by 1.0% and 1.7% respectively. By introducing text descriptions and image descriptions of events, various evaluation indicators of visual event extraction have also been improved to varying degrees. Among them, the recall rate of event type and event argument extraction has been improved most significantly, which effectively increases the number of correct examples predicted by the model and reduces the number of incorrect predictions;
[0027] (2) Based on the CLIP model parameters, the present invention uses the VOA dataset to perform contrastive learning pre-training, and then uses the imSitu dataset to fine-tune the model. During pre-training and fine-tuning, the momentum distillation model is introduced for supervision. After adding momentum distillation, the image-text matching ability of the model can be effectively improved.
[0028] (3) The present invention effectively weakens the impact of data set noise on model extraction performance during the model construction process, and improves the performance of the model in multimedia event extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a flowchart of a multimedia event extraction method based on a multimodal low-dimensional feature representation space provided by an embodiment of the present invention;
[0030] Figure 2 It is a framework diagram for constructing a multimodal low-dimensional feature representation space provided by an embodiment of the present invention;
[0031] Figure 3 is an event extraction framework diagram provided by an embodiment of the present invention;
[0032] Figure 4It is the influence of different multimodal low-dimensional feature representation space construction strategies provided by the experimental examples of the present invention on visual event type detection, visual event argument detection and visual event argument framing. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0034] Example
[0035] A multimedia event extraction method based on multimodal low-dimensional feature representation space, the flow chart is as follows Figure 1 As shown, the following steps are included:
[0036] S1. Constructing a multimodal low-dimensional feature representation space
[0037] Contrastive learning:
[0038] The VOA image caption pair dataset is used to construct a multimodal low-dimensional feature representation space. N For image-title pairs, for image collections I =[ I 1 , I 2 , I 3 ,..., I N ] and text collection T =[ T 1 , T 2 , T 3 , ... , T N ] respectively build Transformer-based encoders.
[0039] Define the following similarity calculation function:
[0040] (1)
[0041] (2)
[0042] in, f v ( I p )and f w ( Tq ) respectively represent the image I p With text T q ( ) is mapped into a unified and standardized low-dimensional feature representation; Represents the similarity calculation function obtained after the text passes through the ordinary encoder, Represents the similarity calculation function obtained after the image passes through the ordinary encoder. These two similarity calculation functions can make the positions of similar images and texts in the vector space closer, that is, the similarity scores of the two are higher. At the same time, two queues are constructed to store the latest N For image-caption pairs. Create a momentum encoder based on the image and text encoders, using fˊ v ( Iˊ p )and fˊ w ( T ˊ q ) represent the normalized features of image and text after momentum encoder, and define , ,in, represents the similarity calculation function between the ordinary encoder text and the momentum distillation encoder image, Represents the similarity calculation function between the ordinary encoder image and the momentum distillation encoder text; for each image and text, after passing through the encoder, the similarity between the image and the text can be calculated using the following formula (Formula (3)) and the similarity from text to image (Formula (4)):
[0043] (3)
[0044] (4)
[0045] in, τ is a temperature parameter that can be learned.
[0046] The loss function L for contrastive learning of images and texts itc Depend on s and p The cross entropy H is defined as:
[0047] (5) .
[0048] Momentum Distillation:
[0049] In contrastive learning of image captions, one-hot tags are used. Images and texts are either completely matched or considered mismatched. However, real image caption data is often weakly correlated. There may be entities in the image that are not mentioned in the text, and the text may be a negation of the image. These factors will affect the contrastive learning model to identify the two as unrelated. Therefore, the momentum distillation method is used to learn the matching of images and texts from the pseudo targets generated by the momentum distillation model. The momentum encoder is used to calculate and ,in, Represents the similarity calculation function from image to text, Represents the similarity calculation function from text to image. Referring to formula (3) and formula (4), the soft pseudo target representation can be calculated: q i2t as well as q t2i Contrastive learning loss function under momentum distillation The definition is as follows:
[0050] (6)
[0051] Among them, the weight α is set to 0.4, and KL(||) represents the calculation of the relative entropy between the true similarity and the soft pseudo target similarity.
[0052] Model training process:
[0053] First, the CLIP ViT-B / 32 version model is used to initialize the parameters of the multimodal low-dimensional feature representation space model, and the momentum distillation model is also initialized using its parameters. Then, the dataset is pre-trained with the help of VOA image captions, and the momentum distillation model is used to generate pseudo targets to supervise the basic model, so that the model can obtain more relevant knowledge in the news field. Finally, the imSitu dataset is used to fine-tune the pre-trained dataset. Because the dataset involves two types of image-related text content, active verbs and semantic roles, a model is used to simultaneously train the similarity matching capabilities of images and active verbs, and images and semantic roles. Similarly, the momentum distillation model is used to supervise the fine-tuning model.
[0054] While using contrastive learning to improve the model's ability to match text and images, the momentum distillation method is introduced to reduce the impact of image captions on the noise in the dataset on the model; the multimodal low-dimensional feature representation space can represent text and image features as a unified dimension, thereby achieving feature fusion of text and image.
[0055] S2. Build a text event extraction model
[0056] Text event type prediction:
[0057] After obtaining the word representation integrated with the image information, determine whether each word is an event trigger word:
[0058] (7)
[0059] in, W t and B t These are model parameters that need to be obtained through model training. H ˊ i is a fusion feature vector containing the first i Words w i The word feature vector obtained by the DeBERTaV3 encoder, the sentence feature vector and the image feature vector obtained by the multimodal low-dimensional feature representation space of the text sentence and the related image respectively. O i It is the probability output vector of all possible event types for the word, while words that are not event trigger words are represented by None. O i The event type with the highest probability is the event type of the text event.
[0060] Text event argument prediction:
[0061] Referring to the idea of text event type prediction, the following method is used to predict the corresponding element role of the entity in the text event:
[0062] (8)
[0063] in, O j is the probability vector of the element role, W a and B a are model parameters that need to be obtained through model training. H ˊ trigger Indicates the event trigger word predicted by the text event. H ˊ j The first j If an entity involves multiple words, the average value is used.
[0064] Training and testing:
[0065] The model training phase of text event extraction uses the ACE2005 text event dataset as the model training data. However, there is currently no matching image data. With the help of the VOA image-caption pair dataset, the five images that best match each text description in the ACE2005 text event dataset can be selected as image data support. Finally, the ACE2005 text events and matching images are converted into feature vectors of uniform dimensions and feature fusion and model training are performed.
[0066] During the testing phase, M 2 E 2 To test the model, all image information related to each sentence is weighted and integrated into the text for model prediction. The details are as follows:
[0067] When given a collection of sentences of multimedia events S ={ s 1 , s 2 , s 3 ,..., s m}, for each sentence in the sentence collection s i ={ w 1 , w 2 , w 3 ,..., w t}, obtain the type embedding representation of each word through the encoder of DeBERTaV3 E T ={ E T,1 , E T,2 , E T,3 ,..., E T,t}、Subword embedding representation E S ={ E S,1 , E S,2 , E S,3 ,..., E S,t} and absolute position embedding representation E A ={ E A,1 , E A,2 ,E A,3 ,..., E A,t}, type embedding representation E T With subword embedding representation E S Fusion as the word content representation vector, and then with the absolute position embedding representation E A The attention decoupling mechanism calculates the attention and relative position information between words to obtain the final vector representation of each word H Word = { H 1 , H 2 , H 3 ,..., H t}.
[0068] After obtaining the vector representation of each word, it needs to be fused with the image features. I ={ i 1 , i 2 , i 3 ,..., i n The multimodal low-dimensional feature representation space is encoded into a low-dimensional vector representation of the image, and the similarity weight of the low-dimensional vector representation of the image is calculated by cosine similarity:
[0069] (9)
[0070] In the formula, I j Represents a collection of images I The j images i j The low-dimensional vector representation after multimodal low-dimensional feature representation space encoding, S i It is a collection of sentences S The i Sentences i i The low-dimensional vector representation after multimodal low-dimensional feature representation space encoding, α i,j is an image i j With sentence i iThe similarity weight is the normalized value calculation of the image and the sentence. The low-dimensional vector representation of the image is weighted and fused according to the similarity weight to obtain the image representation that integrates the image information. H Image :
[0071] (10)
[0072] Finally, the vector representation of each word H Word Image representation integrated with information H Image And the low-dimensional vector representation of the sentence S i To splice: , obtain word representations that incorporate image information Hˊ Word = { Hˊ 1 , H ˊ 2 , Hˊ 3 ,..., Hˊ t}.
[0073] S3. Build an image event extraction model
[0074] The multimedia event image collection I={ i 1 , i 2 , i 3 ,..., i 1 Each image in the image is converted into a visual embedding by the image encoder of the ViECap model, and the visual embedding is used to generate soft prompts by the converter; at the same time, the ViECap model uses a CLIP-based entity classifier to perform zero-shot image entity retrieval, and selects certain entities by calculating the correlation between the image and the entity to construct entity-aware hard prompts; finally, the soft prompts and hard prompts are sequentially connected and input into the language model for image description prediction to obtain a multimedia image collection I Image Description S I2T ={ s 1 I2T , s 2 I2T , s 3 I2T ,..., sn I2T}.
[0075] Visual event type prediction:
[0076] For images in multimedia events, it is necessary to determine the visual event type of each image. Due to the lack of direct annotation of visual event types, the active verb set in the imSitu dataset is V ={ v 1 , v 2 , v 3 ,..., v n} and event types to make a mapping table. Calculate the cosine similarity of all active verbs and images through the common space , we can get the most similar activity verb to the image; among them, Verbs that indicate activity v p The public space low-dimensional vector representation, I q is an image i q Considering that there are many activity verbs in the imSitu dataset that cannot be mapped to visual event types (such as "camouflaging", "unplugging", etc.), it is also necessary to calculate the cosine similarity between the activity verb and the text event and the image description separately. and .in T r and T t I2T Text events s r , Image Description s t I2T The low-dimensional vector representation after the public space is finally as shown in formula (11): (11)
[0077] The three cosine similarity scores are assigned different weights λ 1 , 2 With λ 3, Only those that simultaneously match the visual image, textual events, and the activity verbs of the image description text can obtain a higher similarity score.
[0078] In the imSitu dataset, there are multiple active verbs corresponding to one visual event type. The Top-K method is used to select the top 10 active verbs with the highest image similarity scores, and the frequency of occurrence of the visual event types mapped by these 10 active verbs is counted. The visual event type with the highest frequency is the event type of the visual event.
[0079] Visual event argument prediction:
[0080] Once the visual event type is obtained, the event argument set related to it can be filtered out (the event argument set related to the visual event type is predefined). Using the same method as the visual event type prediction, the active verb set in the imSitu dataset is V ={ v 1 , v 2 , v 3 ,..., v n} is replaced by the set of semantic roles in the same dataset R ={ r 1 , r 2 , r 3 ,..., r m}, then calculate the cosine similarity between the semantic role and the visual image, text event, and image description, and finally obtain the semantic role contained in the image. Through the semantic role and event argument mapping table, the event argument of the visual event can be obtained.
[0081] Visual event argument framing:
[0082] Event arguments refer to objects or participants that play a key role in an event. The task of extracting visual event arguments requires framing the event arguments that have been predicted in the image, that is, locating the key objects or participants in the image with boxes. The boxes of event arguments are located in three steps: forward pass, gradient-based attention integration, and correlation map calculation. The semantic role and the image are taken as input to obtain the similarity score; in the forward pass, the attention map of each attention layer is A Averaging to generate a gradient-enhanced attention map :
[0083] (12)
[0084] in, E hd represents the expected value of the attention weights of all attention heads in the image encoder,C is the similarity score between the active character and the image, and e represents the Hadamard product. After obtaining the attention map, the attention map can be converted into a correlation map. Definition R tt , R ii , R ti They are the text-to-text, image-to-image, and text-to-image correlation matrices respectively, and then the following update rule is used for the self-attention part of the text:
[0085] (13)
[0086] in, Indicates x The correlation matrix calculated by the self-attention layer, Indicates x +1 correlation matrix of the self-attention layer. Then, in the self-attention calculation process, contextual feature information of different modalities is added to the correlation map:
[0087] (14)
[0088] in, Indicates x The text-to-image correlation matrix of other modalities included in the self-attention layer, Indicates x +1 text-to-image correlation matrix of other modalities included in the self-attention layer. For cross-modal attention between text and image, the following update rule is used to normalize the self-attention map:
[0089] (5)
[0090] in, , and Respectively represent the normalized 、 and ;
[0091] The last layer of cross-modal attention R ti is used as a correlation map to predict argument bounding boxes. To generate the bounding box of the character, R ti Set the adaptive value to 0.75* p The threshold value of p is the local peak of the correlation graph, and the minimum bounding box containing the local peak of the correlation graph is calculated as the prediction box.
[0092] Experimental example
[0093] In order to verify the technical effect of the method of the present invention, the following experimental verification is carried out:
[0094] Experimental use of M 2 E 2 The dataset is used to evaluate the model. The dataset consists of 6167 sentences and 1014 images from 245 multimedia news documents, covering 8 event types (selected from the ACE 2005 dataset) and 15 event arguments (expanded from the ACE 2005 dataset and the imSitu dataset). The event instances in the dataset can be divided into 1105 pure text events, 188 pure image events, and 395 multimedia events.
[0095] During the experiment, the results of text event type extraction, text event argument extraction, visual event type detection, visual event argument detection, and visual event argument framing were calculated. Precision, recall, and F1-score were used as indicators for model evaluation.
[0096] In the stage of constructing the multimodal low-dimensional feature representation space, the CLIP model is used to initialize the parameters of the contrastive learning model and the momentum distillation model. The text encoders of both models are Transformer encoders with 12 layers, 512 hidden units, and 8 attention heads. The image encoders are ViT-Transformer encoders with a patch size of 16×16, 12 layers, 768 hidden units, and 12 heads. In the pre-training and fine-tuning stages, 40 rounds of training are set, and the batch size of each round of training is set to 40. In the first 5000 iterations, the learning rate is preheated to 1e -5 After cosine scheduling, it decays to 1e -7 The momentum parameter of the momentum distillation model is set to 0.995, and the distillation weight increases linearly from 0 to 0.4. During the training of the text event extraction model, the maximum text input length is set to 200, and a total of 20 rounds of training are conducted. The batch size of each round of training is 20, and the learning rate is 1e -5 . The Top-K method is used in visual event extraction. K The value is set to 10, and the weight λ 1 , 2 With λ 3 They are 0.7, 0.15, and 0.15 respectively. In terms of image description generation, GPT-2base is used as the text generation model for image description, and the soft prompt length is set to 10. Visual event argument extraction threshold p is 0.7.
[0097] In order to verify the effectiveness of the present invention, several popular and high-performance models were selected as benchmark models for comparative experiments. The experimental results of the following models are quoted from the original paper:
[0098] JMEE is a text event extraction method that uses attention graph convolutional neural networks to combine syntactic information with graph representation learning;
[0099] The GAIL method uses reinforcement learning to extract text events;
[0100] VAD combines visual knowledge with text words and phrases, and uses images to enhance text semantics, thereby improving the extraction performance of text events.
[0101] WASE-T and WASE-V are single-modality extractors of the WASE multimedia event extraction model. Both are trained with only a single modality (WASE-T uses text data, and WASE-V uses image data) and are used for text and visual event extraction, respectively;
[0102] WASE att with WASE obj Weakly supervised methods are used to align text and visual events, and two single-modal event extractors are used to achieve joint extraction of multimedia events; att and obj represent two different methods used in image frame timing:
[0103] Flat att With Flat obj It is in WASE att with WASE obj A variant of the model that removes graph convolutional networks can represent text and visual information by connecting text and visual features;
[0104] CLIP-event aligns multimedia events and extracts information with the help of optimized transmission;
[0105] UniCL builds a unified framework under which text and visual event information can be represented uniformly, and text and visual events can be extracted based on inter-modal information enhancement.
[0106] Table 1 shows the results of text event extraction (top), visual event extraction (middle), and multimedia event extraction (bottom). Train Dataset Type indicates whether various extraction methods (Method) use text data (T), visual data (V), or two data fusions (T+V) for model training. The model in this experimental example is represented by Ours. It can be seen that the results of this experimental example model are better than most baseline results.
[0107] Table 1 Comparison of extraction results using different multimedia event extraction models
[0108]
[0109] The method proposed in the present invention has certain improvements in text event extraction, visual event extraction and multimedia event extraction. In terms of event type extraction, with the help of supplementary enhancement of different modal information, the performance of text and visual event type extraction has been improved by 0.8% and 1.0% respectively, and multimedia event type extraction has been improved by 1.9%. Event argument extraction has been improved by 0.6%, 1.7% and 5.4% respectively. At the same time, the method of the present invention also shows that the performance of event extraction can be improved by integrating relevant event information of different modalities. In terms of text event extraction, by fusing the low-dimensional representation vector of the relevant image with each word, the F1 value of event type and event argument extraction has been improved by 1.0% and 1.7% respectively. By introducing the text description and image description of the event, the various evaluation indicators of visual event extraction have also been improved to varying degrees. Among them, the recall rate of event type and event argument extraction has been improved most significantly, by 3.4% and 1.7% respectively. This shows that by introducing event text and image description information, the model can better judge whether each activity verb and semantic role is related to the event, and reduce the probability of irrelevant words appearing in the Top-10 (using the Top-K method and taking the first 10 words for related calculations), thereby increasing the number of correct examples predicted by the model and reducing the number of incorrect predictions.
[0110] In order to evaluate the rationality of the model improvement of the present invention, multiple groups of ablation experiments were set up for verification.
[0111] The effect of introducing momentum distillation technology on multimedia event extraction is as follows Figure 4As shown in the figure, the five rectangular columns in visual event type detection, visual event argument detection, and visual event argument framing represent the methods CLIP, CLIP+VOA, CLIP+VOA+MoD, CLIP+VOA+imSitu, and CLIP+VOA+iSmitu+MoD from left to right. Visual event extraction directly uses the common space to calculate the similarity between images and event types and event arguments, which can better reflect the performance of the common space. Therefore, the three tasks of visual event type detection, visual event argument detection, and visual event argument framing are selected for comparison. Here, only image data is used instead of text data to reduce the influence of irrelevant factors. The experiment mainly compares the F1 values of the three tasks; CLIP means that the common space is directly extracted using the parameters provided by the CLIP model; CLIP+VOA means that the common space is pre-trained with the VOA dataset based on the CLIP model parameters; CLIP+VOA+MoD is supervised by the momentum distillation model during VOA pre-training; CLIP+VOA+imSitu is to fine-tune the common space with the imSitu dataset after pre-training with the VOA dataset, and train a fine-tuned model using active verbs and semantic roles; CLIP+VOA+imSitu+MoD is our final method, which uses the VOA dataset for contrastive learning pre-training based on the CLIP model parameters, and then uses the imSitu dataset to fine-tune the model. The momentum distillation model is introduced for supervision during pre-training and fine-tuning. By comparison, it can be seen that the F1 values of the CLIP+VOA+imSitu+MoD method in event type detection and event argument detection are 56.9 and 39.8, respectively, which achieve the highest extraction effect. Although the visual event argument frame is 15.9, which is lower than the CLIP+VOA+MoD method, it is verified that the addition of momentum distillation can effectively improve the model's image-text matching ability. Overall, the extraction method of the present invention achieves the best performance in the three tasks.
[0112] Analysis of the impact of the introduction of image description technology on visual event types and visual event argument detection: By introducing text events and text descriptions of images, the overall effect of visual event extraction can be improved. Therefore, the present invention also explores the model performance of image events, text events and image text descriptions under different weights. 2 E 2The dataset contains two types of image events, one with only the event-related image but no matching text event (Only Image), and the other with both image events and directly related text events (Multimedia Image). There is no corresponding text event in Only Image, so only the impact of different weights of image (Image) and image description (I2T) is analyzed. Multimedia Image needs to analyze the impact of image (Image), text event (Text), and image description (I2T). The different proportions of image and image description in Only Image are divided into four situations: only image information (Image: 1.0 and I2T: 0.0), image information has a large weight (Image: 0.85 and I2T: 0.15), image information has a large weight (Image: 0.7 and I2T: 0.3), and weight is balanced (Image: 0.5 and I2T: 0.5). Similarly, the different proportions of images, text events, and image descriptions in Multimedia Image are divided into the following five cases: only image information (Image: 1.0; Text: 0; I2T: 0), only image information and text events (Image: 0.7; Text: 0.3; I2T: 0), only image information and image description (Image: 0.7; Text: 0; I2T: 0.3), image information has a large weight while text events and image descriptions have small weights (Image: 0.7; Text: 0.15; I2T: 0.15), and weight balance (Image: 0.4; Text: 0.4; I2T: 0.4). By calculating the correlation between image events, text events, and related image descriptions and active verbs and semantic roles respectively, the obtained correlation scores are weighted and summed using different weights, and finally the detection results of image event types and image event arguments are obtained. The specific proportions and results are shown in Table 2. Whether it is event type or event argument detection, when the weights of image events, text events and image descriptions are balanced, the detection performance will decrease (such as the 5th, 11th and 13th rows of data in Table 2). At the same time, the Multimedia Image data cannot achieve the best detection performance without text events or image descriptions (such as the 2nd, 3rd, 7th and 8th rows of data in Table 2). When the image description is not integrated into the OnlyImage data, the proportion of the three types of information in the Multimedia Image data is 0.7, 0.15, and 0.15 (such as the 4th row of data in Table 2), the best performance of visual event type detection can be achieved. For visual event argument detection, the proportion of image and image description in the Only Image data is 0.7 and 0.3, and the proportion of image, text and image description in the Multimedia Image data is 0.7, 0.15, and 0.15, which can achieve the best detection performance.Finally, for visual event type detection and visual event argument detection, using Only Image (Image: 1.0 and I2T: 0.0) and Multimedia Image (Image: 0.7; Text: 0.15; I2T: 0.15) as parameters, the best detection results can be achieved, with F1 values of 58.6 and 42.9 respectively.
[0113] Table 2 The impact of different weights of image events, text events and image descriptions on the detection of visual event types and visual event arguments
[0114]
[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A multimedia event extraction method based on a multimodal low-dimensional feature representation space, characterized in that: The following steps are involved: S1. Constructing a multimodal low-dimensional feature representation space Use the VOA image caption dataset to construct a multimodal low-dimensional feature representation space; When training the multimodal low-dimensional feature representation space model, for a given image-caption pair, a Transformer-based encoder is constructed for the image set and the text set respectively, and two queues are constructed for storing the latest image-caption pairs. A momentum encoder is created based on the image and text encoders, and the momentum distillation method is used to learn the matching of images and texts from the pseudo targets generated by the momentum distillation model. The momentum encoder is used to calculate the similarity between images and texts; thereby mapping the matched texts and images to similar areas of the multimodal low-dimensional feature representation space; through the multimodal low-dimensional feature representation space, the texts and images of the test data are mapped to multimodal low-dimensional features of unified dimensions, so as to realize the fusion of features of different modalities and facilitate the calculation of the similarity between texts and images; S2. Text Event Extraction The ACE2005 text event dataset is used as training data for the text event extraction model; The five images that best match each text description in the ACE2005 text event dataset are selected from the VOA image-caption pair dataset as additional image information of the text event; each word in the text description is initialized with feature representation through DeBERTaV3 to obtain a vector representation of each word; the image involved in the additional image information is encoded into an image low-dimensional vector representation through a multimodal low-dimensional feature representation space, and the image low-dimensional vector representation is weighted fused according to the similarity weight to obtain an image representation that integrates the image information, and then the vector representation of each word is spliced with the image representation that integrates the image information and the low-dimensional vector representation of the sentence to obtain a word representation that incorporates the image information, thereby completing the text event extraction; S3. Image event extraction Generate image-related text descriptions, i.e., image descriptions, through ViECap image description technology. After encoding image events, related text events, and generated image descriptions through multimodal low-dimensional feature representation space, use cosine similarity to calculate the similarity scores between image events, related text events, and image descriptions and pre-defined visual event types and visual event arguments. Obtain the visual event type and visual event argument with the highest similarity score through weighted fusion. After obtaining the event arguments, the image is forward-transferred, gradient-based attention integration, and correlation map calculation are performed to frame the relevant event arguments in the image.
2. The multimedia event extraction method based on multimodal low-dimensional feature representation space according to claim 1, characterized in that: In step S1, in order to obtain a multimodal low-dimensional feature representation space that can unify the dimensions of the text and image of the test data, it is first necessary to use the CLIP ViT-B / 32 version to initialize the parameters of the multimodal low-dimensional feature representation space model and the momentum distillation model respectively, and then use the VOA image caption to pre-train the multimodal low-dimensional feature representation space of the text and image of the dataset, and use the pseudo targets generated by the momentum distillation model to supervise the basic model; finally, the imSitu dataset is used to fine-tune the pre-trained dataset, and the momentum distillation model is used to supervise the fine-tuning model.
3. The multimedia event extraction method based on multimodal low-dimensional feature representation space according to claim 1, characterized in that: In step S2, the method for obtaining the vector representation of the word is: each sentence representation in the sentence collection of the multimedia event is passed through the DeBERTaV3 encoder to obtain the type embedding representation, subword embedding representation and absolute position embedding representation of each word, the type embedding representation and the subword embedding representation are fused as the content representation vector of the word, and then the attention and relative position information between words are calculated with the absolute position embedding representation through the attention decoupling mechanism to obtain the vector representation of each word.
4. The multimedia event extraction method based on multimodal low-dimensional feature representation space according to claim 1, characterized in that: In step S2, the similarity weight is calculated by cosine similarity, and the formula is as follows: In the formula, I j Represents a collection of images I The j images i j The low-dimensional vector representation after multimodal low-dimensional feature representation space encoding, S i It is a collection of sentences S The i Sentences i i The low-dimensional vector representation after multimodal low-dimensional feature representation space encoding, α i,j is an image i j With sentence i i The similarity weight of the image information is then represented by H Image The formula is as follows: 。 5. The multimedia event extraction method based on multimodal low-dimensional feature representation space according to claim 1, characterized in that: In step S3, the method of generating image description by the ViECap image description technology is as follows: each image in the image collection is converted into a visual embedding through the image encoder of the ViECap model, and the visual embedding is used to generate soft prompts using the converter; at the same time, the ViECap model uses a CLIP-based entity classifier to perform zero-sample image entity retrieval, and selects certain entities by calculating the correlation between the image and the entity to construct entity-aware hard prompts; finally, the soft prompts and hard prompts are connected in sequence and input into the language model for image description prediction to obtain image descriptions of the multimedia image collection.
6. The multimedia event extraction method based on multimodal low-dimensional feature representation space according to claim 1, characterized in that: In step S3, the method for identifying the visual event type is as follows: the active verb set in the imSitu data set and the visual event type are made into a mapping table, and the cosine similarity of all active verbs and the image is calculated through the common space to obtain the active verb that is most similar to the image; then the cosine similarity of the active verb and the text event and the image description is calculated respectively; finally, the three cosine similarity scores are assigned different weights for weighted fusion to obtain the active verb that matches the visual image, text event and image description text at the same time, and the visual event type of the active verb can be obtained according to the mapping table of the active verb set and the visual event type.
7. The multimedia event extraction method based on multimodal low-dimensional feature representation space according to claim 1, characterized in that: In step S3, the method for identifying the visual event argument is as follows: the semantic role set and the visual event argument in the imSitu dataset are made into a mapping table, and then the cosine similarities of all semantic roles and visual images, text events and image descriptions are calculated through a common space, and finally the three cosine similarity scores are assigned different weights for weighted fusion to obtain a semantic role that simultaneously matches the visual image, text event and image description text, and the visual event argument of the semantic role can be obtained according to the mapping table between the semantic role set and the visual event argument.
