Multi-modal public hotline rumor detection method based on interactive attention mechanism and adaptive feature fusion

Abstract

The invention discloses a multi-modal public hotline rumor detection method based on an interactive attention mechanism and adaptive feature fusion, and the method comprises the following steps: firstly, extracting global visual features of an image through a pre-trained ResNet50 model, and extracting context semantic features described by an original text and an image text through a pre-trained BERT model; secondly, performing deep association on text features and image text description features by utilizing an interactive attention mechanism, and enhancing the expression ability of key semantic information; then, dynamically adjusting feature weights through a two-stage adaptive feature fusion module, and optimizing complementarity of global and local features; and finally, inputting the fused features into a multi-layer perceptron for prediction. According to the method, semantic association is captured through an interactive attention mechanism, feature expression is optimized through adaptive feature fusion, and the rumor detection efficiency and effect are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and natural language processing, and particularly to a multimodal public hotline rumor detection method based on an interactive attention mechanism and adaptive feature fusion. Background Art

[0002] Single-modal information is often insufficient to fully reflect the authenticity of an event, and the fusion processing of multimodal information becomes the key to rumor detection.

[0003] Research on multimodal rumor detection technology aims to automatically identify false information from massive amounts of information, helping the public and government agencies make informed decisions. Traditional rumor detection methods primarily rely on single-modal data, such as text or images. However, single-modal information often fails to fully capture the characteristics of a rumor, resulting in low detection accuracy. In recent years, multimodal rumor detection methods have gained increasing attention. By combining information from multiple modalities, such as text and images, they can more accurately identify rumors. By comprehensively utilizing data from different modalities, multimodal rumor detection technology can capture richer information features, thereby improving the accuracy and reliability of rumor detection.

[0004] Existing multimodal rumor detection methods mainly focus on the following aspects:

[0005] 1) Feature fusion-based methods: These methods simply concatenate or weightedly fuse text and image features before inputting them into a classifier for rumor detection. While this method can comprehensively utilize multimodal information, the feature fusion process often ignores the correlation and complementarity between modalities, resulting in poor fusion results.

[0006] 2) Deep learning-based methods: These methods use deep learning models (such as CNN and RNN) to automatically extract and fuse multimodal features. These models can learn complex patterns in data, but they are computationally complex when processing large amounts of data and are prone to overfitting.

[0007] 3) Graph Neural Network-based Methods: By constructing a propagation graph in social networks, graph neural networks (GNNs) are used to capture rumor propagation patterns and user interaction characteristics. This type of method performs well in modeling social relationships, but still has shortcomings when dealing with multimodal feature fusion.

[0008] 4) Attention-based methods: Dynamically adjust the weights of features from different modalities through the attention mechanism, enabling the model to focus on key information. While the attention mechanism can effectively capture the correlation between features, the semantic alignment between modalities remains a challenge when dealing with complex scenarios.

[0009] While existing multimodal rumor detection methods have made some progress, they still face several limitations. First, feature fusion methods are often overly simplistic and fail to fully exploit the deep connections between modalities. Second, deep learning models suffer from low computational efficiency when processing large amounts of data and rely heavily on labeled data. Third, graph neural network methods are limited in their ability to fuse multimodal features, making it difficult to fully capture the multidimensional characteristics of rumors. Finally, attention mechanisms need further optimization for cross-modal semantic alignment to improve model accuracy and robustness.

[0010] To overcome these limitations, the present invention proposes a multimodal public hotline rumor detection method based on an interactive attention mechanism and adaptive feature fusion. By introducing the interactive attention mechanism, the method effectively captures the deep connections between different semantic features, enhancing the model's ability to represent key features. Simultaneously, the adaptive feature fusion module dynamically adjusts feature weights, further improving model performance. Furthermore, the proposed method has been optimized in terms of computational efficiency and robustness, making it more suitable for practical application scenarios. Summary of the Invention

[0011] Building on existing research, this paper proposes a multimodal public hotline rumor detection method based on a mutual attention mechanism and adaptive feature fusion. This method aims to capture the deep connections between text and image features through the mutual attention mechanism and enhance the model's ability to express key features through adaptive feature fusion, thereby improving the model's rumor detection performance in complex scenarios.

[0012] Technical Solution: To achieve the above-mentioned purpose, the present invention proposes a multimodal public hotline rumor detection method based on interactive attention mechanism and adaptive feature fusion, which includes the following steps:

[0013] (1) Multimodal feature extraction: Multimodal feature extraction is divided into image feature extraction, original text feature extraction and image text description feature extraction.

[0014] Image feature extraction: Use the pre-trained ResNet50 model to extract features from the input image and output the image feature vector Extract global visual features of the input image.

[0015] Original text feature extraction: Use the pre-trained BERT model to extract features from the input original text and output the text feature vector Extract contextual semantic features from the original text.

[0016] Image text description feature extraction: Use the pre-trained BERT model to extract features from the text description of the input image and output a text feature vector Extract contextual semantic features from text descriptions of images.

[0017] (2) Interactive attention mechanism: The text feature T is used as the query vector (Query), and the image text description feature D is input into the multi-head attention module as the key vector (Key) and the value vector (Value). The multi-head attention mechanism captures the correlation between features by mapping the features into multiple subspaces, and each subspace is independently calculated to obtain the weight, and finally obtains the deep text feature after interaction through the residual network and normalization operation. feature The interactive text features can focus on the key information of the original text features and the text description features of the image, thereby enhancing the expressiveness of the features.

[0018] (3) Adaptive feature fusion: Adaptive feature fusion is divided into the primary adaptive fusion stage and the secondary adaptive fusion stage.

[0019] Initial adaptive fusion stage: perform linear transformation on the original text feature T and image feature I respectively to obtain the transformed text feature t feature and image features i feature , the transformed text feature t feature and image features i feature Spliced ​​together to form a joint feature vector, then the joint features are transformed by linear transformation to obtain an intermediate representation. The intermediate representation is then processed using the sigmoid activation function to generate the fusion weight ti weight , the value of the fusion weight is limited to between 0 and 1, and the weight reflects the model's judgment on the relative importance of text features and image features. weight , the text feature t feature and image features i feature Perform weighted fusion and finally obtain the feature ti after the initial adaptive fusion feature =ti weight *t feature +(1-ti weight )*i feature .

[0020] Secondary adaptive fusion stage: deep interactive text features obtained through interactive attention mechanism feature Perform linear transformation, and then deep transform the interactive features feature and the feature ti after the initial adaptive fusion feature The sigmoid activation function is then used to process the intermediate representation to generate the fusion weight f. weight, then the feature ti after the initial adaptive fusion feature And the transformed interaction feature deep feature Perform feature weighted fusion and finally obtain the features after secondary adaptive fusion feature =f weight *ti feature +(1-f weight )*deep feature .

[0021] (4) Classification prediction: The final fusion features are input into the multi-layer perceptron for classification and description to produce a judgment result to determine whether the input information is a rumor.

[0022] Beneficial effects: The present invention can effectively solve the shortcomings of existing rumor detection methods in terms of multimodal feature interaction and fusion, give full play to the semantic association mining ability of the interactive attention mechanism and the dynamic weight adjustment advantages of adaptive feature fusion, and optimize feature expression by introducing an innovative interactive attention mechanism and a two-stage fusion strategy, thereby significantly improving the accuracy and efficiency of multimodal rumor detection. First, the interactive attention mechanism can deeply mine the semantic association between the original text features and the text description features of the image. The multi-head attention module independently calculates the attention weights in different subspaces to capture the complex relationship between features, thereby enhancing the model's ability to recognize key semantic information. Secondly, the adaptive feature fusion module balances the contributions of different modal features by dynamically adjusting the feature weights. The initial adaptive fusion stage combines the original text features and image features, and the secondary adaptive fusion stage integrates the text features after interaction, so that the model can optimize feature expression according to data characteristics, improve sensitivity to key features, and effectively complement global features and local features, thereby optimizing feature expression and improving the model's expression ability and generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall framework diagram of the method of the present invention; DETAILED DESCRIPTION

[0024] Below in conjunction with accompanying drawing, further illustrate the present invention, it should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.After reading the present invention, those skilled in the art will all fall within the scope limited by the appended claims of this application to the modification of various equivalent forms of the present invention.

[0025] This paper proposes a multimodal public hotline rumor detection method based on interactive attention mechanism and adaptive feature fusion, which aims to capture the deep correlation between text and image features through interactive attention mechanism and enhance the model's ability to express key features through adaptive feature fusion, thereby improving the model's rumor detection performance in complex scenarios. Figure 1As shown in Figure 1, the complete process of the present invention includes four stages: multimodal feature extraction, interactive attention mechanism processing, adaptive feature fusion, and rumor classification. The specific implementation method is described as follows:

[0026] The multimodal feature extraction stage corresponds to step (1) of the technical solution. The specific implementation methods are: image feature extraction, original text feature extraction and image text description feature extraction.

[0027] Image feature extraction: Given an input image I raw ∈R H×W×3 , use the pre-trained ResNet50 model to extract features from the image. The ResNet50 model encodes the image through a multi-layer convolutional neural network and outputs the global visual feature vector I of the image:

[0028] I feature =ResNet50(I raw )

[0029] Among them, I feature is the global visual feature vector of the image.

[0030] Original text feature extraction: Given the input original text T = {t1, t2, ..., t n}, where T is a text containing n tokens. Use the pre-trained BERT model to extract features from the original text and output the contextual semantic feature vector of the text:

[0031] T feature =BERT(T)=BERT(t1,t2,…,t n )={x1,x2,…,x n}

[0032] Among them, T feature is the contextual semantic feature vector of the original text T, x i Represents the i-th token representation, with a dimension of d.

[0033] Image text description feature extraction: Given an input image text description D = {d1, d2, ..., d m}, where D is a text containing m tokens. Use the pre-trained BERT model to extract features from the text description of the image and output the contextual semantic feature vector of the text description of the image:

[0034] D feature =BERT(D)=BERT(d1,d2,…,d m )={v1,v2,…,v m}

[0035] Among them, Dfeature is the contextual semantic feature vector of the text description D of the image, v i Represents the i-th token representation, with a dimension of d.

[0036] The interactive attention mechanism stage corresponds to step (2) of the technical solution. The specific implementation method is: the original text feature T feature As the query vector, the text description feature D of the image feature It is used as both a key vector and a value vector and is input to the multi-head attention module. The multi-head attention mechanism captures the correlation between features by mapping features into multiple subspaces and independently calculating the attention weight for each subspace. The specific formula is as follows:

[0037]

[0038] Among them, Q, K, V are query, key and value matrices respectively, d k is the dimension of the key vector. For the multi-head attention mechanism, the calculation process is:

[0039] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O

[0040] Where h is the number of attention heads, W O is the output weight matrix.

[0041] After processing by the multi-head attention mechanism, the text features after deep interaction are obtained. feature , the formula is:

[0042] deep feature =MultiHead(Q=T feature ,K=D feature ,V=D feature )

[0043] In order to enhance the expressiveness and stability of the model, a residual connection is applied to the output of the multi-head attention mechanism. This residual connection helps the gradient to be better transmitted in the network, prevents the gradient disappearance problem, and improves the learning ability of the model. The specific calculation formula is:

[0044] deep feature =deep feature +T feature

[0045] Perform layer normalization on the output of the residual connection to ensure the stability and convergence speed of model training. The specific formula for layer normalization is:

[0046]

[0047] where μ and σ are the mean and standard deviation of the input features, γ and β are learnable scaling and bias parameters, and ε is a small constant to prevent division by zero.

[0048] Therefore, after multi-head attention mechanism, residual connection and layer normalization processing, the text feature deep after interaction is obtained. feature :

[0049] deep feature =LayerNorm(deep feature )

[0050] This interactive attention mechanism can deeply explore the semantic association between the original text features and the text description features of the image. By independently calculating the attention weights in different subspaces through the multi-head attention module, it can capture the complex relationship between features and thus enhance the model's ability to recognize key semantic information.

[0051] The adaptive feature fusion stage corresponds to step (3) of the technical solution. The specific implementation method is: the first adaptive feature fusion stage and the second adaptive feature fusion stage.

[0052] The specific implementation method of the initial adaptive feature fusion stage is: the original text feature T feature and image features I feature Perform linear changes respectively to map the original text features and image features into a unified semantic space to obtain the transformed text features t feature and image features i feature , the calculation formula is as follows:

[0053] t feature =W t T+b t

[0054] i feature =W i T+b i

[0055] Among them, W t and W i is the learnable weight matrix, b t and b i is the bias term.

[0056] In order to capture the text features t feature and image features i feature The potential correlation between the two, the text feature t feature and image features i feature Spliced ​​together to form a joint feature vector:

[0057] ti concat =Concat(t feature ,i feature )

[0058] The joint features are then processed through a linear transformation module to generate an intermediate representation that reflects the complex semantic relationship between text and image features. The calculation formula is as follows:

[0059] ti middle =W m ·ti concat +b m

[0060] In order to capture the relative importance of text and image features in the current context, the weight value is ensured to be between 0 and 1. Therefore, the intermediate representation features are processed using the sigmoid activation function to generate feature fusion weights, which dynamically adjust the relative contributions of image features and text features. The calculation formula is as follows:

[0061]

[0062] ti weight =sigmoid(ti middle )

[0063] The Sigmoid function takes the input value ti middle Compressed to the range of 0 to 1, the generated weight ti weight It can be directly interpreted as the “relative importance of text and image features in the current context.” This probabilistic weighting not only ensures the rationality of the values ​​but also introduces dynamic adjustment capabilities through nonlinear mapping, enabling the model to better adapt to different input data and contexts.

[0064] According to the generated feature weight ti weight , perform weighted fusion of text features and image features to obtain the feature ti after the initial adaptive fusion feature :

[0065] ti feature =ti weight ·t feature +(1-ti weight )·i feature

[0066] This process not only balances the contributions of different modal features, but also enhances the model's sensitivity to key information.

[0067] The specific implementation of the secondary adaptive feature fusion stage is: using the deep interactive features obtained by the interactive attention mechanism feature, the deep interactive features are mapped to the same feature space as the initial feature fusion features through linear transformation, and the transformed text features are obtained feature , the calculation formula is as follows:

[0068] deep feature =W deep deep feature +b deep

[0069] Among them, W deep is the learnable weight matrix, b deep is the bias term.

[0070] In order to capture higher-level feature associations, the deep-level interactive features are concatenated with the features after the initial adaptive fusion to form a new joint feature vector. The calculation formula is as follows:

[0071] f concat =Concat(deep feature +ti feature )

[0072] In order to further explore the deep relationship between features, the joint features are processed through the linear transformation module to generate a new intermediate representation. The calculation formula is as follows:

[0073] f middle =W m ·f concat +b m

[0074] After that, the sigmoid activation function is used again to generate a new feature fusion weight f weight , this weight dynamically adjusts the relative contribution of the initial fusion features and the interaction features. The calculation formula is as follows:

[0075] f weight =sigmoid(f middle )

[0076] In order to further optimize the feature expression so that the model can capture key information more accurately and achieve more efficient rumor detection in complex scenarios, the feature weight f is used. weight , perform weighted fusion of the initial adaptive fusion features and the interactive features to obtain the final fusion feature feature. The calculation formula is as follows:

[0077] feature=f weight ·ti feature +(1-f weight )·deep feature

[0078] Through this carefully designed two-stage adaptive feature fusion strategy, the model designed in this paper not only dynamically adjusts the weights of different modal features, but also optimizes the integration of features at different semantic levels. This innovative feature fusion method significantly improves the model's expressiveness and generalization performance, enabling it to perform well in multimodal rumor detection tasks.

[0079] The classification prediction stage corresponds to step (4) of the technical solution. The specific implementation method is: input the final adaptive fusion feature feature into the fully connected layer to generate the logits of the classification result.

[0080] Logits(X)=W f feature+b f

[0081] Among them, W f is the weight matrix of the fully connected layer, b f is the bias term, and Loogits(f) is the score for each category.

[0082] Finally, the logits are converted into probability distributions of various categories through the softmax activation function, and the category with the maximum probability is selected as the prediction result. The calculation formula is as follows:

[0083]

[0084] Among them, p(y k |X) is category y k The predicted probability, C is the total number of categories, Logits k (X) is category y k The corresponding logit value, if the category with the maximum probability is y max , then the final classification label of the text is y max .

[0085] During the training process, in order to optimize the performance of the model, the cross entropy loss function is used To measure the difference between the prediction and the true label, the calculation formula is:

[0086]

[0087] Among them, y k is the 0 or 1 encoding of the true label, (p(y k |X) is the predicted category probability, and the optimization goal is to minimize the loss function And use the back propagation algorithm to update the model parameters W f , W, b f and b, etc., to train the model.

[0088] This paper uses the model's accuracy and F1 value to measure the correctness of the model's predictions. Specifically, the accuracy is a basic indicator for measuring the correctness of the model's predictions. It represents the ratio of the number of samples correctly predicted by the model to the total number of samples. The specific formula is:

[0089]

[0090] Among them, θ(predicted label i =True label i ) is an indicator function that takes the value of 1 when the model predicts the same label as the true label for the i-th sample, and 0 otherwise.

[0091] In order to further evaluate the performance of the model, especially when the data set is unbalanced, the F1 value is introduced. The F1 value is the harmonic mean of the precision and recall rate, which can comprehensively consider the precision and recall ability of the model. The specific formula is:

[0092]

[0093] Accuracy provides an intuitive reflection of the overall accuracy of a model, but can be misleading when working with imbalanced datasets. The F1 score, by combining precision and recall, provides a more comprehensive assessment of model performance, particularly when the ratio of positive to negative samples is imbalanced. Therefore, this paper uses both accuracy and F1 to measure the accuracy of model predictions, ensuring reliable evaluation results across diverse data distributions and contexts.

[0094] The present invention proposes a multimodal public hotline rumor detection method based on an interactive attention mechanism and adaptive feature fusion. In order to test the effectiveness of this method, this patent uses a rumor detection dataset of social media comments open sourced from Weibo. This patent is evaluated on the Weibo open source rumor detection multimodal dataset, which contains 7,639 pieces of data, including 2,425 pieces of non-rumor information, 1,775 pieces of rumor information, and 3,439 pieces of information to be verified. This patent divides the training set, validation set, and test set into 8:1:1 ratios, with 6,159 pieces of training set, 769 pieces of validation set, and 711 pieces of test set, and is compared with other multimodal rumor detection methods.

[0095] The hyperparameter configuration used in the method described in this invention is detailed in Table 1. For model inference, the BERT model based on the hfl-chinese-macbert-base was used, consisting of 12 Transformer encoder layers. A fixed learning rate optimization strategy was employed, with the learning rate set to 2e-5. AdamW was used as the optimization algorithm. Furthermore, the hidden layer vector dimension of the model was set to 768 to ensure sufficient information representation.

[0096] Table 1 Model hyperparameter settings

[0097]

[0098] Table 2 shows the comparative experimental results of different multimodal rumor detection algorithms on the Weibo dataset. The experimental results show that the proposed multimodal public hotline rumor detection method based on an interactive attention mechanism and adaptive feature fusion achieves the best accuracy on the Weibo dataset. Therefore, the comparative experimental results demonstrate the effectiveness of the proposed algorithm in multimodal rumor detection and recognition tasks. The comparative experimental models used in this paper are: the LSTM+VGG19 model, the CNN+VGG19 model, the BERT+ReNet50 model, the BRCA model with BERT+ReNet50 and an interactive attention network, and the BRSF model with BERT+ResNet50 and an adaptive fusion network.

[0099] Table 2 Multimodal rumor detection weibo dataset (results report accuracy and F1 value)

[0100]

[0101] The present invention has achieved remarkable experimental results on the open source dataset of multimodal rumor detection on Weibo, with an accuracy rate of 92.62% and an F1 value of 92.86%, both of which are better than other multimodal rumor detection methods, proving the effectiveness of the method proposed in this patent in the multimodal public hotline rumor detection method. By introducing the interactive attention mechanism, the present invention can deeply explore the semantic association between interactive text features and image text description features, capture the dependency of features in different subspaces, and enhance the model's ability to recognize key information. At the same time, the adaptive feature fusion module optimizes the expression of global features and local features through a two-stage dynamic weight adjustment strategy, enabling the model to dynamically adjust feature weights according to the characteristics of the input data, significantly improving the feature recognition ability and generalization performance.

[0102] The innovation of this invention lies in proposing a multimodal public hotline rumor detection method based on a mutual attention mechanism and two-stage adaptive feature fusion. The mutual attention mechanism captures the association between textual description features of text and images, enhancing the expressiveness of key semantic information. Two-stage adaptive feature fusion dynamically adjusts feature weights, optimizing the complementarity of global and local features, significantly improving the accuracy and efficiency of rumor detection. Experimental results demonstrate that this method performs well in multimodal rumor detection tasks.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal public hotline rumor detection method based on interactive attention mechanism and adaptive feature fusion, characterized by: The method comprises the following steps: (1) Extract the global visual features of the image through the pre-trained ResNet50 model, and extract the contextual semantic features of the original text and the text description of the image through the pre-trained BERT model; (2) Using the interactive attention mechanism to deeply interact and associate text features with image text description features, thus enhancing the ability to express key semantic information; (3) Dynamically adjust feature weights through a two-stage adaptive feature fusion module to optimize the complementarity of global and local features; (4) The fused features are input into a multi-layer perceptron for prediction to determine whether the input information is a rumor.

2. The method according to claim 1, characterized in that In the step (2), the multi-head attention module captures the correlation between features by mapping features to multiple subspaces and independently calculating the attention weight for each subspace, thereby deeply interacting the representation of the original text features and the text description features of the image, and enhancing the expression ability and stability of the model through residual connections and layer normalization operations.

3. The method according to claim 1, characterized in that In the step (3), the initial adaptive fusion stage maps the original text features and image features to a unified semantic space through linear transformation, generates a joint feature vector after splicing, and generates a fusion weight through a sigmoid activation function to dynamically adjust the relative contributions of the text features and image features.

4. The method according to claim 1, wherein In step (3), the secondary adaptive fusion stage concatenates the deep interactive features obtained by the interactive attention mechanism with the features after the initial adaptive fusion, generates a new intermediate representation through linear transformation, and generates a new fusion weight through the sigmoid activation function to further optimize the feature expression.

5. The method according to claim 1, characterized in that In step (4), the final fusion feature is input into the multi-layer perceptron, the logits of the prediction result are generated through the fully connected layer, and the logits are converted into probability distribution through the softmax activation function, and the category with the maximum probability is selected as the prediction result.

Images