A multi-modal event detection method and system based on a dual-Transformer fusion model

Abstract

The application provides a multi-modal event detection method and system based on a double-Transformer fusion model, relates to the technical field of computer application, and uses a Chinese pre-training ALBERT model and an S-CBAM-VGG model to extract text features and image features from social media text content and image content respectively; a double-layer Transformer feature interaction fusion network is used to capture the heterogeneity difference between different modes and the cross-modal correlation, to fuse the text features and the image features, and to obtain multi-modal features; based on the obtained multi-modal features, a correlation event is identified, and event invariance features are captured to classify the identified correlation event; the application designs a double-Transformer fusion model based on images and language, effectively fuses the data of different modes, detects the multi-modal data, and makes the screening and classification results of the correlation event more accurate.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, and in particular relates to a multimodal event detection method and system based on a dual Transformer fusion model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Over the past few decades, event detection based on multimodal data has been an important research task in fields such as computer image processing, speech recognition, and natural language processing. It aims to fuse various information extracted from multiple sensory modalities to better understand and identify current multimodal events. In other words, the goal of event detection based on multimodal data is to identify and understand the events described in multimodal data (such as images, videos, speech, and text) and classify them. Events can be known or unknown, but the description of the events must be accurate, comprehensive, and consistent.

[0004] Multimodal data-based event detection has wide applications in monitoring, healthcare, and security. For example, in disaster detection and classification, data from social media platforms is widely used in disaster assessment and management. When a disaster occurs, users post a large number of tweets to share real-time information and seek help. These tweets contain key information such as casualties, infrastructure damage, notifications, and emergency assistance. Although obtaining large-scale real-time data from social media platforms is no longer difficult, filtering out and classifying tweets related to disaster events from a large number of irrelevant, redundant, or false tweets remains a significant challenge.

[0005] Textual and image information are the two main forms of information on social media platforms. Most previous research focused on classifying social media information using either text or image alone. However, this unimodal event classification is one-sided and inefficient. Text-based classification methods may miss some useful information, while image-based methods may be ambiguous. Therefore, many studies have emerged that approach disaster events from a multimodal perspective, achieving progress in various tasks. Jin et al. proposed a novel recurrent neural network (att-RNN) with an attention mechanism, which can fuse textual and image features to classify multimodal data on social network platforms. Ofli et al. used a pre-trained VGG-16 model to extract image features from multimodal data, and a CNN model with five hidden layers to extract text features from the multimodal data. They then input the two feature vectors into a shared representation for event classification based on multimodal data. Zou et al. used a pre-trained VGG-16 model to extract image features from social media platforms, used the FastText framework to extract text features, and then used a multimodal fusion module with three fully connected layers and one SoftMax layer to combine image and text features to classify relevant disaster images. They conducted experiments on the real-world disaster dataset CrisisMMD. Even though these studies use both text and image information for event detection, they tend to extract features that cannot be shared between different events. These specific features are helpful for classifying past events but are not as effective for detecting and classifying newly occurring events.

[0006] Therefore, most existing methods either have overly complex model structures, low classification accuracy, or extract multimodal features that depend on specific disaster events and cannot be well generalized to identify new disaster events. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, this invention provides a multimodal event detection method and system based on a dual Transformer fusion model. It designs a dual Transformer fusion model (VLDT) based on images and text, which effectively fuses data from different modalities to detect multimodal data, making the screening and classification results of related events more accurate.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of this invention provides a multimodal event detection method based on a dual Transformer fusion model;

[0010] A multimodal event detection method based on a dual Transformer fusion model is proposed. This method uses a trained dual Transformer fusion model to perform multimodal event detection on input social media text and image content. The dual Transformer fusion model includes a feature extraction module, a multimodal fusion module, an event detection module, and an event classification module. The specific processing steps are as follows:

[0011] The feature extraction module uses pre-trained ALBERT and S-CBAM-VGG models to extract text features and image features from social media text content and image content, respectively.

[0012] The multimodal fusion module utilizes a two-layer Transformer feature interaction fusion network to capture the heterogeneity differences between different modalities and the cross-modal correlations, and fuses text features and image features to obtain multimodal features.

[0013] The event detection module identifies relevant events based on the obtained multimodal features.

[0014] The event classification module captures event invariance features based on the obtained multimodal features and classifies related events accordingly.

[0015] The S-CBAM-VGG model uses the scaling factor of Batch Normalization as the contribution factor to the channel weights to suppress insignificant features and obtain image features.

[0016] Furthermore, before extracting text features, the text content of social media is preprocessed, specifically by removing forwarding titles, user handles, and stop words from the text content; and by using Bert tokenizer to perform word segmentation on the text content.

[0017] Furthermore, before extracting image features, the image content of social media is preprocessed, specifically by: adjusting the image size to a preset size; and using random horizontal and vertical flip operations for augmentation.

[0018] Furthermore, the S-CBAM-VGG model includes a modified VGG-19 and an S-CBAM attention module;

[0019] The modified VGG-19, based on VGG-19, replaces average pooling with global average pooling and replaces the last three fully connected layers with one fully connected layer.

[0020] The S-CBAM attention module includes a parameterless channel attention module and a spatial attention module.

[0021] Furthermore, the multimodal fusion module calculates the fused image features and fused text features respectively through a two-layer Transformer feature interaction fusion network DTFIF-Net, and then performs weighted fusion of the two features to obtain multimodal features.

[0022] Furthermore, the DTFIF-Net is built based on Transformer. The Transformer encoder serves as an auxiliary module to obtain auxiliary features that can be fused, while the Transformer decoder serves as the main module. Through multi-head self-attention and normalization processing, it obtains main features that can be fused. The main features and auxiliary features are fused through attention feature fusion to obtain the fused features.

[0023] Furthermore, the loss function of the dual Transformer fusion model includes two parts: the loss function of the event detection module and the loss function of the event classification module;

[0024] Adversarial training is introduced. By training the feature extraction module and the event detection module adversarially, the loss of the event detection module is minimized, thereby improving the performance of detecting relevant events. By training the feature extraction module and the event classification module adversarially, the loss of the event classification module is maximized, thereby learning feature representations that are invariant to relevant events.

[0025] A second aspect of the present invention provides a multimodal event detection system based on a dual Transformer fusion model.

[0026] A multimodal event detection system based on a dual Transformer fusion model includes a feature extraction module, a multimodal fusion module, an event detection module, and an event classification module.

[0027] The feature extraction module is configured to extract text features and image features from social media text content and image content using the ALBERT model and the S-CBAM-VGG model, respectively.

[0028] The multimodal fusion module is configured to: utilize a two-layer Transformer feature interaction fusion network to capture the heterogeneity differences between different modalities and the cross-modal correlations, and fuse text features and image features to obtain multimodal features;

[0029] The event detection module is configured to identify relevant events based on the obtained multimodal features;

[0030] The event classification module is configured to: capture event invariant features based on the obtained multimodal features, and classify related events;

[0031] The S-CBAM-VGG model uses the scaling factor of Batch Normalization as the contribution factor to the channel weights to suppress insignificant features and obtain image features.

[0032] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a multimodal event detection method based on a dual Transformer fusion model as described in the first aspect of the present invention.

[0033] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a multimodal event detection method based on a dual Transformer fusion model as described in the first aspect of the present invention.

[0034] The above one or more technical solutions have the following beneficial effects:

[0035] (1) In order to effectively extract image features, this invention proposes a new feature extraction model S-CBAM-VGG, which is composed of a modified VGG-19 and an S-CBAM attention module. S-CBAM is an upgrade of the CBAM attention module and can be seamlessly integrated into any CNN architecture. It can suppress non-salient features and improve the attention effect.

[0036] (2) In order to effectively reduce the heterogeneity differences between different modalities and capture cross-modal correlations, a new two-layer Transformer Feature Interaction Fusion Network (DTFIF-Net) is proposed to fuse text features and image features to obtain multimodal interaction fusion representations of text and images.

[0037] (3) The VLDT model proposed in this invention is applicable to various event detection fields based on multimodal data. Extensive experiments were conducted on disaster-related public datasets. The experimental results show that the model can effectively detect and classify disaster-related events and is significantly better than the baseline model in several evaluation metrics.

[0038] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0039] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0040] Figure 1 This is a diagram of the VLDT architecture of the first embodiment.

[0041] Figure 2 This is a structural diagram of the first embodiment, the S-CBAM-VGG model.

[0042] Figure 3 This is a diagram of the S-CBAM architecture of the first embodiment.

[0043] Figure 4 This is a diagram of the two-layer Transformer feature interaction fusion network structure of the first embodiment. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] Example 1

[0046] This embodiment discloses a multimodal event detection method based on a dual Transformer fusion model.

[0047] A multimodal event detection method based on a dual Transformer fusion model is proposed. This method uses a trained dual Transformer fusion model to perform multimodal event detection on input social media text and image content, such as... Figure 1 As shown, the dual Transformer fusion model includes a feature extraction module, a multimodal fusion module, an event detection module, and an event classification module. The specific processing steps are as follows:

[0048] Step S1: Using the feature extraction module, extract text features and image features from social media text content and image content using the Chinese pre-trained ALBERT model and the S-CBAM-VGG model, respectively.

[0049] The feature extraction module includes an image feature extraction module and a text feature extraction module. The two modules are described in detail below.

[0050] (1) Text Feature Extraction Module

[0051] This paper uses a pre-trained ALBERT model for Chinese text extraction. ALBERT addresses many issues associated with the BERT model, which sacrifices performance for increased parameter count, such as higher computational requirements, longer training times, and performance degradation with a large number of parameters. Therefore, this embodiment uses ALBERT as the primary component to extract text features R. t .

[0052] To meet the input requirements of the ALBERT model, the text content is preprocessed. First, forwarding titles, user handles, and stop words are removed. Then, the BERT tokenizer is called to perform word segmentation on the text content. Finally, the preprocessed text content is input into the ALBERT model to extract text feature representations. To ensure that the features have the same dimension as the image features, a fully connected layer is added to the last layer of the ALBERT model, as shown in formula (1).

[0053] R t =σ(W t ·P t (1)

[0054] Among them, R t Let P represent text features, σ represent the ReLU activation function in the ALBERT model, and P represent text features. t W represents the output of the last layer of the ALBERT model. t This represents the weight matrix of the fully connected layer in the text feature extractor.

[0055] (2) Image feature extraction module

[0056] For image modal content on social media platforms, this invention proposes a novel image feature extraction model, S-CBAM-VGG, to extract image features R. v To meet the input requirements of the model, the images first need to be preprocessed. The image size is uniformly adjusted to 224×224, and augmentation is performed using operations such as random horizontal and vertical flipping. This can enhance the generalization ability of the model while expanding the number of samples, enabling it to more accurately capture deeper information and focus on the important parts of relevant events. The preprocessed images are then input into the S-CBAM-VGG model for feature extraction.

[0057] Figure 2 This is a structural diagram of the S-CBAM-VGG model, as shown below. Figure 2 As shown, the S-CBAM-VGG model consists of a modified VGG-19 and an S-CBAM attention module.

[0058] The modified VGG-19 is obtained by making the following improvements to VGG-19: global average pooling is used instead of average pooling, and a single fully connected layer is used instead of the last three fully connected layers.

[0059] The S-CBAM attention module, based on the lightweight and general-purpose CBAM module, can be seamlessly integrated into any CNN architecture. S-CBAM helps the model focus on key regions of relevant events and suppress background noise. While CBAM utilizes the mutual information between different dimensions of features, it lacks consideration of the contribution factor of channel weights, which could further suppress insignificant features. Therefore, this invention aims to improve the attention effect by utilizing the contribution factor of channel weights. It directly uses the scaling factor of batch normalization to represent the importance of weights, avoiding the need for fully connected and convolutional layers like CBAM, while also achieving higher computational efficiency.

[0060] Figure 3 This is a structural diagram of the S-CBAM attention module, such as... Figure 3 As shown, S-CBAM is divided into two independent sub-modules: a parameterless channel attention module and a spatial attention module. The parameterless channel attention module learns weight information along the channel dimension and generates channel-wise attention features M. c The spatial attention module learns weight information along the spatial dimension to generate spatial attention features M. s .

[0061] The parameterless channel attention module takes a modified VGG-19 extracted feature map F as input and primarily explores the relationships between feature maps of different channels to focus on more important channels. When the input is an H×W×C feature map F, a scaling factor from batch normalization is used. The scaling factor reflects the magnitude of change in each channel and also indicates the importance of that channel; a larger scaling factor indicates a more significant change in that channel, thus containing richer information and greater importance. The input feature F is processed by the parameterless channel attention module to obtain feature M. c (F), as shown in formula (2):

[0062] M c (F) = sigmoid(W) η (BN(F))) (2)

[0063] Where η is the scaling factor for each channel, W n BN represents the weight of each channel; the input feature F passes through the parameterless channel attention module to obtain the output F' of the parameterless channel attention module, as shown in formula (3):

[0064]

[0065] in, This indicates element-wise multiplication.

[0066] Unlike the parameterless channel attention module, the spatial attention module focuses on which parts of the input image are more important, complementing the parameterless channel attention module. It takes the output F' of the parameterless channel attention module as input, performs average pooling and max pooling along the channel dimension to obtain two H×W×1 feature maps, and then concatenates these two feature maps by channel. Finally, a 7×7 convolutional layer and a sigmoid function are used to obtain the spatial weight coefficients M. s (F'), as shown in formula (4):

[0067]

[0068] Among them, f 7×7 F represents a 7×7 convolution kernel. avg F represents the output of average pooling. max This represents the output of max pooling; after the input feature F' passes through the spatial attention module, the output feature map F' is obtained, which is the final output feature of S-CBAM, as shown in formula (5):

[0069]

[0070] To ensure that it has the same dimensionality as the text features, a fully connected layer is added to the last layer of the model, as shown in formula (6):

[0071] R v =σ(W v ·F”) (6)

[0072] Among them, R v It is the final image feature, W v σ is the weight matrix of the fully connected layer, and σ represents the ReLU activation function in the S-CBAM-VGG model.

[0073] Step S2: Through the multimodal fusion module, a two-layer Transformer feature interaction fusion network is used to capture the heterogeneity differences between different modalities and the cross-modal correlations, and to fuse text features and image features to obtain multimodal features.

[0074] Heterogeneous modal data content exhibits varying degrees of heterogeneity and cross-modal correlations. To effectively reduce heterogeneity and capture cross-modal correlations, a two-layer Transformer Feature Interaction Fusion Network (DTFIF-Net) is used to fuse text and image features, resulting in a multimodal interactive fusion representation of the text and an image. Finally, these two representations are fused to obtain the final multimodal features.

[0075] Specifically, the two-layer Transformer feature interaction fusion network includes DTFIF-Net1 and DTFIF-Net2. The inputs of DTFIF-Net1 and DTFIF-Net2 are reversed, but the calculation process is the same. Both DTFIF-Net1 and DTFIF-Net2 take text features and image features as inputs. After internal Transformer feature interaction fusion, they respectively obtain two vectors. One of them learns the correlation coefficient between text features and image features to obtain the multimodal interactive fusion representation of text, and the other learns the correlation coefficient between image features and text features to obtain the multimodal interactive fusion representation of image.

[0076] For DTFIF-Net1, input 1 is text features, and input 2 is image features; for DTFIF-Net2, input 1 is image features, and input 2 is text features. Each DTFIF-Net consists of two Transformer units (a Transformer encoder and a Transformer decoder), with different input information for each unit. The left Transformer encoder, as an auxiliary module, is a standard encoder structure where the Q, K, and V values ​​are all derived from input 1, designed to learn the intra-modal features of input 1. The right Transformer decoder, as the main module, introduces a novel attention feature fusion method, namely a master-slave attention mechanism. The DTFIF-Net proposed in this invention can better fuse text and image information. The encoder-decoder design enhances the interactivity of different modal data, enabling the model to further extract multimodal fusion features.

[0077] Taking DTFIF-Net1 as an example, the processing procedure of DTFIF-Net is explained as follows: Figure 4 As shown, the specific process is as follows:

[0078] First, image features are input as input 1 to the encoder. Through multi-head self-attention and addition & normalization, auxiliary features S suitable for fusion are obtained. Text features are input as input 2 to the decoder. Through multi-head self-attention and addition & normalization, the main features P suitable for fusion are obtained. To further optimize the feature distribution of P, a master-slave attention mechanism is adopted, and fine-tuning is performed by fusing the auxiliary features S. The attention matrix M is formally defined. ij M represents the correlation between the i-th eigenvector in P and the j-th eigenvector in S; ij The calculation method is shown in formula (7):

[0079] M ij =PS (7)

[0080] Where P represents the primary feature and S represents the auxiliary feature.

[0081] Then, the Softmax function is used to normalize each row in M, as shown in formula (8):

[0082]

[0083] Where n represents the number of rows in the attention matrix M.

[0084] Furthermore, the fine-grained attention output is obtained through formula (9):

[0085] F att =MS (9)

[0086] Among them, F att This represents the output of fine-grained attention, where M represents the attention matrix and S represents the auxiliary features.

[0087] Next, a single-layer fully connected neural network is used to optimize the main feature distribution, and the calculation method is shown in Equation (10):

[0088] F i =Tanh((F att-i +P i W f +b f (10)

[0089] Among them, P i F represents the i-th vector of the main features. att-i This represents the i-th vector of the fine-grained attention matrix. W f and b f These are trainable parameters.

[0090] Finally, the auxiliary features and the optimized main features are pooled and concatenated to obtain the fused text features. That is, the multimodal interactive fusion representation of text.

[0091] DTFIF-Net2 uses the opposite inputs of DTFIF-Net1, but the computation process is the same: input 1 is text features, and input 2 is image features. Finally, after a series of similar operations, the fused image features are obtained. That is, a multimodal interactive fusion representation of images.

[0092] Furthermore, DTFIF-Net1 and DTFIF-Net2 do not share weights. Ultimately, the output of DTFIF-Net, the multimodal feature F M As shown in formula (11):

[0093] F M =αF T +βF V (11)

[0094] Where α and β are the multimodal interaction fusion representations of text (FT) and image (F), respectively. V The weights are α+β=1.

[0095] The multimodal fusion module is expressed by the formula f. F =(X; θ) F X represents the input feature, θ F These are the parameters for multimodal fusion.

[0096] Step S3: Identify relevant events based on the obtained multimodal features using the event detection module.

[0097] First, let's explain the concept of relevant events. Relevant events refer to events that are related to a certain field. For example, if a public safety incident occurs in a certain region, a number of disorganized messages will appear on social media platforms. By collecting and analyzing these messages, we can identify information related to the public safety incident and notify relevant departments to take emergency measures.

[0098] The event detection module is a neural network consisting of two fully connected layers and one softmax layer, used to identify relevant events from predicted multimodal data. The multimodal event detection module is represented by the formula f. D (F M ;θ D ), F M It is the input multimodal feature, θ D This represents all the parameters that need to be learned.

[0099] Combining the formulas from the multimodal fusion module and the multimodal event detection module, we obtain the complete formula:

[0100] p(x j )=f D (f F (x j ;θ F );θ D (12)

[0101] Where, x j Let p(x) represent the j-th feature. j ) represents the probability that the j-th feature is not related to any event.

[0102] The purpose of the event detection module is to identify relevant events from multimodal data. Cross-entropy loss can be used as the loss function, and the formula for the loss function is shown below:

[0103]

[0104] Where y represents the true probability of the event occurring, p(x) represents the predicted probability, and (θ) F ,θ D ) represents the parameters that need to be learned.

[0105] By finding the optimal parameters To minimize the loss function of the multimodal event detection module Expressed as a formula:

[0106]

[0107] Step S4: Using the event classification module, based on the obtained multimodal features, capture the event invariance features and classify the relevant events;

[0108] The event classification module is a neural network consisting of two fully connected layers and an activation function. Its purpose is to classify multimodal data into one of N events. This module can be represented as f C (F M ;θ C ), where F M It is the input multimodal feature, θ C This represents all the parameters that need to be learned.

[0109] The loss of the multimodal event classification module can be defined as:

[0110]

[0111] Among them, y j Let N represent the true probability of the j-th event occurring, and let N represent the number of events.

[0112] Optimal parameters The loss is defined as in, It is used to evaluate the similarity between different events; the greater the loss, the more similar the representations of different events, and the distribution of the representations will also eliminate the uniqueness of each event; therefore, in order to learn the event invariant features, VLDT seeks the optimal parameters. To maximize losses

[0113] Adversarial training is introduced to eliminate the uniqueness of multimodal data. Adversarial training is a minimization strategy among the feature extraction module, multimodal event detection module, and multimodal event classification module. On the one hand, the multimodal fusion module f... F (·;θ F Event detection module f D (·;θ D Cooperation, to reduce losses Minimize and improve the performance of detecting relevant events; on the other hand, the multimodal fusion module f F (·;θ F Attempt to fool the event classification module to cause losses The goal is to maximize and learn feature representations that are invariant to relevant events; furthermore, the multimodal event classification module attempts to minimize the loss. To identify each relevant event.

[0114] The final adversarial loss function for the three modules is defined as follows:

[0115]

[0116] The hyperparameter *v* controls the trade-off between the losses of the event detection module and the event classification module; a larger *v* indicates that event classification is relatively more important, while a smaller *v* means that the event detection module plays a more prominent role in the detection task; for the minimization strategy, the optimal parameters for VLDT are... As shown below:

[0117]

[0118]

[0119] To verify the effectiveness of the proposed model, this embodiment conducted extensive comparative and ablation experiments, and will introduce the dataset and experimental setup of the proposed model, and analyze the performance of VLDT.

[0120] Dataset

[0121] To fairly evaluate the performance of the proposed model, this embodiment conducted extensive experiments on the Crisis MMD dataset. This dataset was obtained by crawling and filtering real tweets from users on social media. Each tweet contains data in both image and text modalities. The structure and partitioning of the dataset are shown in Table 1. Task 1 aims to determine whether the tweets in the dataset contain information related to disaster events. Task 2 aims to further mine the disaster types in the tweets, merging the seven disaster events in the original dataset into four types (e.g., merging Hurricane Harvey, Hurricane Maria, and Hurricane Irma, and merging Mexico Earthquake and Iran Earthquake). The merged dataset includes four disasters: hurricanes, wildfires, earthquakes, and floods.

[0122] Table 1 Crisis MMD Dataset Information

[0123]

[0124] Parameter settings

[0125] For the feature extraction module, pre-trained ALBERT and S-CBAM-VGG models were used as the main components, with the hidden layer size of the fully connected layer set to 32. For the multimodal event detection and classification modules, the hidden layer size of the first fully connected layer was set to 64, and the hidden layer size of the second layer was set to 32. For the entire VLDT model training process, a batch size of 128 was used for 500 epochs, and the Adam optimizer was used to train the model. The initial learning rate was set to 0.001, the weight decay rate was set to 0.0001, and the decay period was 50 epochs. In addition, the dropout probability of each module in the model was 0.5.

[0126] Comparative experiment

[0127] To verify the performance of the proposed VLDT model, representative models in the field of natural disaster detection in recent years were selected as benchmark models for comparative experiments. These models and their descriptions are shown below:

[0128] MANN is a multimodal adversarial neural network proposed by Gao et al., designed to identify tweets related to emerging disaster events. The model consists of three modules: a feature extraction module, a tweet detection module, and a disaster recognition module. The feature extraction module uses BERT and VGG-19 to learn textual and visual feature representations from tweets. The tweet detection module and the disaster recognition module are both composed of fully connected layers and softmax layers.

[0129] DICFM is a multimodal data fusion model proposed by Zou et al., designed to classify disaster tweets. It uses a pre-trained VGG-16 model and the FastText framework to extract image and text features. Then, it uses three fully connected layers and one SoftMax layer to fuse the image and text features in a multimodal manner.

[0130] CABD is a multimodal fusion method proposed by Mahdi et al., which introduces a cross-attention mechanism and aims to detect major events around the world in real time. The model uses BERT and DenseNet to extract text features and image features respectively, uses a cross-attention module to avoid the transfer of negative knowledge between modalities, and uses random shared embeddings to alleviate overfitting of the data.

[0131] CAMM is a novel cross-attention multimodal deep neural network proposed by A. Khattar and SMK Quadri. It utilizes attention masks of the text modality to highlight features of the visual modality and aims to classify multimodal disaster data.

[0132] Table 2 Comparison of experimental results

[0133]

[0134] Table 2 presents the performance comparison results of all the above models. It can be observed that the performance of the VLDT model in this embodiment is significantly better than that of other models in related works. Specifically, compared with the benchmark models of the past two years, in Task 1, the accuracy of the VLDT model improved by 1% to 4%, and the F1 score improved by 1% to 3%; in Task 2, the accuracy of the VLDT model improved by 2% to 3%, and the F1 score improved by 2% to 3%. This fully demonstrates the effectiveness of the VLDT model proposed in this embodiment. It can effectively extract the features of text modality and image modality in tweets and perform high-quality fusion. After fusion, VLDT can effectively utilize the event detection module and the event classification module to improve the accuracy of disaster identification.

[0135] ablation experiment

[0136] To further analyze the contributions of each module component in the VLDT model, ablation experiments were designed on the Crisis MMD dataset. Specifically, to investigate the functions of the feature extraction module, multimodal fusion module, and event classification module in the VLDT model, the following comparative method was adopted:

[0137] Text: ALBERT is used to learn text features from tweets, and a fully connected layer with softmax is added to predict whether a tweet is related to a disaster and to classify informative tweets.

[0138] Visual: This method uses VGG-19 to learn image features from tweets and adds a fully connected layer with softmax to predict whether a tweet is related to a disaster, and classifies informative tweets.

[0139] Visual-All: Learns image features using the complete image feature extraction module (S-CBAM-VGG) and adds a fully connected layer with softmax for prediction and classification.

[0140] VLDT-: This is a variant of the VLDT module that only includes the feature extraction module, multimodal fusion module, and event detection module.

[0141] Table 3 Ablation Experiment Results

[0142]

[0143] Table 3 presents the results of the ablation experiments. A comparison between the Visual method and VLDT shows that the method of using VGG-19 alone to learn image features and predict tweet accuracy is significantly worse than the image feature extraction method S-CBAM-VGG proposed in this embodiment, demonstrating the effectiveness of S-CBAM-VGG. A comparison of Text, Visual, Visual-All, and VLDT shows that with only single-modal data input, the performance is significantly worse than with multi-modal input. This proves that the proposed DTFIF-Net can better integrate text and image information, effectively reducing heterogeneity differences between different modalities and positively impacting the fusion learning effect. The model performance of VLDT- is significantly worse than VLDT, indicating that the features extracted by VLDT- are mostly disaster-specific, making it difficult to learn enough shared features across different disasters. Conversely, VLDT retains disaster-invariant features using the event classification module. Based on the above analysis of the ablation experiments, it is easy to conclude that each module in VLDT plays a crucial role in the overall model performance.

[0144] Detecting and classifying events from multimodal data on social media platforms is a challenging task due to the abundance of irrelevant or fake multimodal data online. This work proposes a novel model, VLDT, capable of detecting and classifying multimodal data. VLDT consists of four modules: a feature extraction module, a multimodal fusion module, an event detection module, and an event classification module. The feature extraction module extracts textual and image features; the multimodal fusion module is responsible for better integrating textual and image information; the adversarial training and event detection module detects relevant events; and the adversarial training and event classification module captures event-invariant features and classifies relevant events. Experiments on the large-scale disaster dataset Crisis MMD validate the effectiveness of VLDT. Future work plans to use this method to fuse other multimodal features for disaster identification, such as fusing audio and video features from social media platforms.

[0145] Example 2

[0146] This embodiment discloses a multimodal event detection system based on a dual Transformer fusion model.

[0147] A multimodal event detection system based on a dual Transformer fusion model includes a feature extraction module, a multimodal fusion module, an event detection module, and an event classification module.

[0148] The feature extraction module is configured to extract text features and image features from social media text content and image content using the ALBERT model and the S-CBAM-VGG model, respectively.

[0149] The multimodal fusion module is configured to: utilize a two-layer Transformer feature interaction fusion network to capture the heterogeneity differences between different modalities and the cross-modal correlations, and fuse text features and image features to obtain multimodal features;

[0150] The event detection module is configured to identify relevant events based on the obtained multimodal features;

[0151] The event classification module is configured to: capture event invariant features based on the obtained multimodal features, and classify related events;

[0152] The S-CBAM-VGG model uses the scaling factor of Batch Normalization as the contribution factor to the channel weights to suppress insignificant features and obtain image features.

[0153] Example 3

[0154] The purpose of this embodiment is to provide a computer-readable storage medium.

[0155] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a multimodal event detection method based on a dual Transformer fusion model as described in Embodiment 1 of this disclosure.

[0156] Example 4

[0157] The purpose of this embodiment is to provide an electronic device.

[0158] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a multimodal event detection method based on a dual Transformer fusion model as described in Embodiment 1 of this disclosure.

[0159] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal event detection method based on a dual Transformer fusion model, characterized in that, The trained dual Transformer fusion model performs multimodal event detection on input social media text and image content. The dual Transformer fusion model includes a feature extraction module, a multimodal fusion module, an event detection module, and an event classification module. The specific processing steps are as follows: Step S1: Using the feature extraction module, extract text features and image features from social media text content and image content respectively using the pre-trained ALBERT model and S-CBAM-VGG model; Step S2: Through the multimodal fusion module, a two-layer Transformer feature interaction fusion network is used to capture the heterogeneity differences between different modalities and the cross-modal correlations, and to fuse text features and image features to obtain multimodal features; Step S3: Based on the obtained multimodal features, identify relevant events using the event detection module; Step S4: Using the event classification module, based on the obtained multimodal features, capture the event invariance features and classify the relevant events; The S-CBAM-VGG model includes a modified VGG-19 and an S-CBAM attention module. The modified VGG-19, based on VGG-19, uses global average pooling and replaces the last three fully connected layers with a single fully connected layer. The S-CBAM attention module includes a non-parametric channel attention module and a spatial attention module. The modified VGG-19 extracted feature map F is obtained by the non-parametric channel attention module. , ,in, η It is the scaling factor for each channel. It is the weight of each channel. BN Indicates batch standardization, input features After passing through the parameterless channel attention module, the output of the parameterless channel attention module is obtained. , ,in, This indicates element-wise multiplication; The dual-layer Transformer feature interaction fusion network includes DTFIF-Net1 and DTFIF-Net2, where each DTFIF-Net consists of a Transformer encoder and a Transformer decoder. The Transformer encoder serves as an auxiliary module to obtain auxiliary features that can be fused, while the Transformer decoder serves as the main module to obtain main features that can be fused through multi-head self-attention and normalization processing. The main features and auxiliary features are fused through attention feature fusion to obtain the fused features.

2. The multimodal event detection method based on a dual Transformer fusion model as described in claim 1, characterized in that, Before extracting text features, the text content of social media is preprocessed, specifically by removing forwarding titles, user handles, and stop words; and by using Bert tokenizer to perform word segmentation on the text content.

3. The multimodal event detection method based on a dual Transformer fusion model as described in claim 1, characterized in that, Before extracting image features, the image content of social media is preprocessed, specifically by: adjusting the image size to a preset size; and using random horizontal and vertical flipping operations for augmentation.

4. The multimodal event detection method based on a dual Transformer fusion model as described in claim 1, characterized in that, The multimodal fusion module calculates the fused image features and fused text features respectively through a two-layer Transformer feature interaction fusion network DTFIF-Net, and then performs weighted fusion of the two features to obtain multimodal features.

5. The multimodal event detection method based on a dual Transformer fusion model as described in claim 1, characterized in that, The loss function of the dual Transformer fusion model includes two parts: the loss function of the event detection module and the loss function of the event classification module. Adversarial training is introduced. By training the feature extraction module and the event detection module adversarially, the loss of the event detection module is minimized, thereby improving the performance of detecting relevant events. By training the feature extraction module and the event classification module adversarially, the loss of the event classification module is maximized, thereby learning feature representations that are invariant to relevant events.

6. A system for a multimodal event detection method based on a dual Transformer fusion model as described in any one of claims 1-5, characterized in that, It includes a feature extraction module, a multimodal fusion module, an event detection module, and an event classification module. The feature extraction module is configured to extract text features and image features from social media text content and image content using the ALBERT model and the S-CBAM-VGG model, respectively. The multimodal fusion module is configured to: utilize a two-layer Transformer feature interaction fusion network to capture the heterogeneity differences between different modalities and the cross-modal correlations, and fuse text features and image features to obtain multimodal features; The event detection module is configured to identify relevant events based on the obtained multimodal features; The event classification module is configured to: capture event invariant features based on the obtained multimodal features, and classify related events; The S-CBAM-VGG model uses the scaling factor of Batch Normalization as the contribution factor to the channel weights to suppress insignificant features and obtain image features. The S-CBAM-VGG model includes a modified VGG-19 and an S-CBAM attention module; The modified VGG-19, based on VGG-19, replaces average pooling with global average pooling and replaces the last three fully connected layers with one fully connected layer. The S-CBAM attention module includes a parameterless channel attention module and a spatial attention module.

7. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-5.

8. A storage medium, characterized in that, The computer-readable instructions are stored non-transitory, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-5 are executed.

Images