Multimodal rumor detection method and system based on hierarchical attention network

Abstract

The application discloses a kind of multi-modal rumor detection method and system based on layered attention network, text feature vector is extracted by layered attention model, visual feature vector in news picture is extracted by VGG19 model, and news sample is judged as rumor or non-rumor according to the text feature vector and visual feature vector;Layered attention model includes word encoder and sentence encoder based on self-attention mechanism, the importance between current word and context word and the importance between current sentence and context sentence are measured, word vector representation and sentence vector representation that fuse context information are obtained, so that the features of news document hierarchy can be captured;The importance of the feature vector of each picture different area relative to text is measured by attention mechanism, which can make the model focus more on the picture area with higher importance, improve the detection accuracy of the model, and multi-modal detection is more suitable for multi-modal form news, to get more comprehensive and accurate rumor prediction.

Description

A Multimodal Rumor Detection Method and System Based on Hierarchical Attention Networks Technical Field

[0001] This invention relates to a rumor detection method and system, and more particularly to a multimodal rumor detection method and system based on a hierarchical attention network. Background Technology

[0002] The portability and low cost of social media have fostered collective wisdom, but they have also bred various false rumors, often containing fabricated or even forged content to mislead readers. Given the vast amount of online content, manually debunking rumors is extremely time-consuming; therefore, building an effective and practical automatic rumor detection system that mines supplementary information from different angles is essential. With the increasing rich media nature of social media, news is gradually shifting from single-text formats to multimodal formats such as images and videos. Existing multimodal rumor detection systems use convolutional neural networks and long short-term memory networks to extract features from news text, failing to distinguish the importance of contextual words to the current word. When sentences are long, they exhibit forgetting characteristics. Furthermore, the processing of visual features relies on information from the entire image, ignoring the importance of different regions within the image for semantic understanding, leading to a decline in multimodal detection performance. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a multimodal rumor detection method that fully captures the hierarchical features of news documents and considers the importance of words, sentences, and image regions; the second purpose of this invention is to provide a multimodal rumor detection system.

[0004] Technical solution: The multimodal rumor detection method based on hierarchical attention networks described in this invention includes the following:

[0005] The word vectors in the news text are obtained by using the GloVe pre-trained model, the text feature vectors are extracted by the hierarchical attention model, and the visual feature vectors in the news images are extracted by the VGG19 model. The news sample is judged to be a rumor or not based on the text feature vectors and the visual feature vectors.

[0006] The method for extracting text feature vectors using a hierarchical attention model is as follows: based on the importance between the current word and the surrounding words in the sentence, the context word information is aggregated to obtain the word vector representation of the word; based on the importance of all word vector representations in the sentence to the sentence, the word information is aggregated to obtain the sentence vector; based on the importance between the sentence vector and the surrounding sentences in the text, the context sentence information is aggregated to obtain the sentence vector representation of the sentence; based on the importance of all sentence vector representations in the text to the text, the sentence information is aggregated to obtain the text feature vector.

[0007] Furthermore, the method for obtaining word vectors from news text using a GloVe pre-trained model is as follows:

[0008] The GloVe pre-trained model is used to learn word vector initialization on a corpus, and the word vector loss function J is...

[0009]

[0010]

[0011] Where V is the size of the vocabulary, and X is a co-occurrence matrix constructed based on the corpus, with each element X in the matrix representing a co-occurrence. pq The number of times word p and its context word q co-occur within a context window of a specific size; w p and Let b represent any two word vectors in the corpus, which are the word vectors to be pre-trained and solved. p and It is a bias term; f(X) pq ) is the weighting function, α is the threshold, and X is the weighting function. max This represents the maximum number of times the two elements appear together.

[0012] Furthermore, the method for aggregating contextual word information to obtain the word vector representation of a word based on the importance between the current word and its contextual words in the sentence is as follows:

[0013] Based on the context word w ig Compared to the current word w ij Importance weight α gj Calculate the weighted sum as the word vector representation of the current word.

[0014]

[0015]

[0016] Where q ij =W Q w ij k ij =W K w ij v ij =W V w ij W Q W k and W V Let N be the parameter matrix in the attention mechanism, where N is the number of words.

[0017] Furthermore, the method for aggregating word information to obtain a sentence vector based on the importance of all word vectors in the sentence is as follows:

[0018] Based on word vector representation Importance weight β of the current sentence ij Calculate the weighted sum as the sentence vector s i ;

[0019]

[0020]

[0021] Among them, u ij yes The hidden representation, W w Let b be the parameter matrix. w For bias terms; u w The context vectors for words are randomly initialized and jointly learned during training.

[0022] Furthermore, based on the importance between the sentence vector and its surrounding sentences in the text, the method for aggregating contextual sentence information to obtain the sentence vector representation is as follows:

[0023] Based on the context of sentence s l Relative to the current sentence s i Importance weight α l Calculate the weighted sum as the sentence vector representation of the current sentence.

[0024]

[0025]

[0026] in, and Let M be the parameter matrix in the attention mechanism, and M be the number of sentences.

[0027] Furthermore, the method for aggregating sentence information to obtain text feature vectors based on the importance of all sentence vectors in the text is as follows:

[0028] According to sentence vector representation Importance weight β relative to text t i Calculate the weighted sum as the text feature vector t;

[0029]

[0030]

[0031] Among them, u i yes The hidden representation, W s Let b be the parameter matrix. s For bias terms; u s The sentence context vector is randomly initialized and jointly learned during training.

[0032] Furthermore, the method for extracting visual feature vectors from news images using the VGG19 model is as follows:

[0033] The feature maps of news images are represented as feature vector sequences. Where v h This represents the feature vector of the image region corresponding to the h-th image patch, based on v. h The importance weights relative to text t are calculated and weighted to form a visual feature vector v;

[0034]

[0035]

[0036]

[0037] Among them, u h It is v h The hidden representation of u h =tanh(W H v h +b H )v h b H W H W H Let b be the parameter matrix. H H is the bias term; H is the number of image patches, and P is the dimension of the visual feature vector.

[0038] Furthermore, the method for determining whether a news sample is a rumor or not based on the text feature vector and visual feature vector is as follows:

[0039] The text feature vector and visual feature vector are concatenated to obtain the multimodal representation x of the news. x is then input into a fully connected layer and an activation layer to obtain the probability distribution p = soft max(W) of a sample being predicted as a rumor. C x+bc ), where W C Let b be the parameter matrix. C This is the bias term; if p exceeds the threshold, the sample is judged as a rumor; otherwise, it is judged as not a rumor.

[0040] The multimodal rumor detection system based on hierarchical attention networks described in this invention includes:

[0041] The preprocessing module is used to obtain word vectors from news texts using a GloVe pre-trained model;

[0042] The text feature extraction module is used to extract text feature vectors through a hierarchical attention model. Based on the importance between the current word and its context words in the sentence, it aggregates context word information to obtain the word vector representation. Based on the importance of all word vector representations in the sentence to the sentence, it aggregates word information to obtain the sentence vector. Based on the importance of the sentence vector to its context sentences in the text, it aggregates context sentence information to obtain the sentence vector representation. Based on the importance of all sentence vector representations in the text to the text, it aggregates sentence information to obtain the text feature vector.

[0043] The visual feature extraction module is used to extract visual feature vectors from news images using the VGG19 model.

[0044] The rumor detection module is used to determine whether a news sample is a rumor or not based on the text feature vector and the visual feature vector.

[0045] The computer-readable storage medium of the present invention stores a computer program, characterized in that, when the computer program is executed by a processor, it implements a multimodal rumor detection method based on a hierarchical attention network.

[0046] Beneficial effects: Compared with the prior art, the advantages of the present invention are: (1) Using a word encoder and sentence encoder based on a self-attention mechanism, the importance between the current word and all context words in the sentence, and the importance between the current sentence and all context sentences in the text, to obtain word vector representation and sentence vector representation that integrate context information, and then to obtain text feature vector, so that when constructing document representation, it can distinguish between content with high importance and content with low importance, thereby capturing the features of the hierarchical structure of news documents; (2) By measuring the importance of feature vectors of different regions of each image relative to the text through an attention mechanism, and aggregating the representations of all regions of the image to obtain the visual feature vector representation of the whole image; by measuring the importance of different regions of each image relative to the text, the model focuses on the image regions with higher importance, and obtains visual feature clues more effectively; (3) Capturing the features of the hierarchical structure of news documents, while making full use of visual modal information features, greatly improving the accuracy of rumor detection; Combining text and image features to comprehensively detect rumors, making the final detection results more applicable to today's multimodal news, and obtaining more comprehensive and accurate rumor prediction. Attached Figure Description

[0047] Figure 1 is a diagram of the multimodal rumor detection framework based on hierarchical attention network of the present invention.

[0048] Figure 2 shows the results of the elimination analysis experiment in the embodiment of the present invention. Detailed Implementation

[0049] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0050] As shown in Figure 1, the multimodal rumor detection method based on hierarchical attention networks includes the following steps:

[0051] S1: Preprocess the news dataset, including the following steps:

[0052] A1: For news data with missing text or images, delete the data to ensure the quality of the dataset;

[0053] A2: Divide the data obtained in step A1 into the final training set and test set in a 7:3 ratio.

[0054] S2: Use a GloVe pre-trained model to obtain vector representations of words in the news text data, including the following steps:

[0055] Unlike traditional word vectors that are directly and randomly initialized, we use the GloVe pre-trained language model to initialize word vectors. Through learning on a large-scale corpus, it can learn certain syntactic and common-sense knowledge.

[0056] Word vector learning requires optimal prediction of the co-occurrence frequency of any two words in the corpus. To this end, a novel weighted least squares regression model is proposed, and the following loss function J is given:

[0057]

[0058] Where V is the size of the vocabulary, and X is a co-occurrence matrix constructed based on the corpus, with each element X in the matrix representing a co-occurrence. pq This represents the number of times word p and its context word q co-occur within a context window of a specific size. p and Let represent any two word vectors in the corpus, which are the word vectors to be pre-trained and solved. The superscript T indicates the transpose of the vector. p and It is a bias term. The weighting function f(X) pq The following piecewise function was used:

[0059]

[0060] In this embodiment, α is taken as 0.75, while X max The value is 100.

[0061] For any news text t, assuming it contains M sentences and each sentence contains N words, the total number of words in the text is M×N. Finally, GloVe pre-training is used to embed the words into vectors, resulting in all pre-trained word vectors w for each text t. p p∈[1,M×N].

[0062] In the subsequent hierarchical attention network, since two-layer attention mechanisms are applied at the word and sentence levels to extract features, we further divide the text t into two hierarchical structures. The first is the sentence-level structure, that is, dividing the text t into M sentences, s i ,i∈[1,M] represents the vector of the i-th sentence in the text; then comes the word hierarchy structure, that is, each sentence is further divided into N words, w ij ,i∈[1,M],j∈[1,N] represent the word vector of the jth word in the i-th sentence.

[0063] S3: Based on the pre-training results obtained in step S2, an improved hierarchical attention model is used to extract features from the text, including the following steps:

[0064] Traditional hierarchical attention models use bidirectional GRU recurrent neural networks to obtain contextual information about words by summarizing bidirectional information. This method has significant drawbacks: it cannot distinguish the importance of contextual words in a sentence to the current word, and it is prone to forgetting when sentences are long. Therefore, we introduce a word encoder based on a self-attention mechanism.

[0065] B1: Word encoder based on self-attention mechanism: given a word w ij Sentences s where i∈[1,M],j∈[1,N] i For i∈[1,M], we introduce a self-attention mechanism model to measure the current word w. ij and all context words in the sentence w ig The importance of g ∈ [1, N] and all context words w ig The information is aggregated to obtain a word vector representation that incorporates contextual information.

[0066] Specifically, we introduce the parameter matrix W Q W k W V For each word w ij :

[0067] q ij =W Q w ij

[0068] k ij =W K w ij

[0069] v ij =W V w ij

[0070] Then we base our current word w ij The corresponding q ij and all context words in the sentence w ig ,k corresponding to g∈[1,N] ig The similarity between g∈[1,N] is used to measure the importance of all context words relative to the current word, and the normalized importance weight α is obtained through the softmax function. gj :

[0071]

[0072] Where α gj Indicates the context word w ig Compared to the current word wij Importance weights. The superscript T indicates the transpose of the vector.

[0073] Finally, calculate the weights of all context words w. ig The corresponding v ig The weighted sum is used as the word vector representation of the current word. Right now:

[0074]

[0075] B2: Word Attention: Not all words contribute equally to the meaning of a sentence. Therefore, an attention mechanism model is introduced to measure the meaning of all word vectors in a sentence. Relative to the current sentence vector s i The importance of this is emphasized, and the representations of all words in the sentence are aggregated to obtain the sentence vector s. i .

[0076] Specifically, firstly, a linear transformation is used on the word vectors. Perform the transformation and obtain the hyperbolic tangent function. Hidden representation u ij :

[0077]

[0078] Among them W w Let b be the parameter matrix. w This is a bias term.

[0079] Then, the word context vector u is introduced. w It can be viewed as a high-level representation of the fixed query "what are the information words?", which is randomly initialized and jointly learned during training. Then, based on u... ij ,j∈[1,N] and u w The similarity is used to measure the importance of all words in a sentence relative to the current sentence, and the normalized importance weight β is obtained through the softmax function. ij :

[0080]

[0081] Where β ij Word vectors Relative to the current sentence s i Importance weights. The superscript T indicates the transpose of the vector.

[0082] Finally, all word vectors are calculated based on the obtained weights. The weighted sum is used as the sentence vector si :

[0083]

[0084] B3: Sentence Encoder Based on Self-Attention Mechanism: Similarly, a self-attention mechanism is used instead of the traditional bidirectional GRU recurrent neural network model to obtain the contextual information of the sentence to represent the current sentence. Given any sentence vector s i For i∈[1,M], we also introduce a self-attention mechanism model to measure the current sentence s. i and all context sentences s in text t l The importance of l∈[1,M], and all context sentences s l The information is aggregated to obtain a sentence vector representation that incorporates contextual information.

[0085] Specifically, a parameter matrix is ​​introduced. For each sentence s i :

[0086]

[0087]

[0088]

[0089] Similarly, according to the current sentence s i The corresponding q i and all context sentences s in text t l The corresponding k l The similarity between l∈[1,M] is used to measure the importance of all context sentences relative to the current sentence, and the normalized importance weight α is obtained through the softmax function. l :

[0090]

[0091] Where α l Indicates the context of the sentence s l Relative to the current sentence s i The importance weights are indicated by the superscript T, which represents the transpose of the vector.

[0092] Finally, all context sentences s are calculated based on the obtained weights. l The corresponding v l The weighted sum of l∈[1,M] is used as the sentence vector representation of the current sentence. Right now:

[0093]

[0094] B4: Sentence Attention: To reward sentences that serve as clues for correctly classifying documents, an attention mechanism model is used again to measure the vector representations of all sentences in the text. The importance of the text t is determined by aggregating the representations of all sentence information in the text to obtain the final text vector t.

[0095] Specifically, firstly, a linear transformation is applied to the sentence vector. Perform the transformation and obtain the hyperbolic tangent function. Hidden representation u i :

[0096]

[0097] Among them W s Let b be the parameter matrix. s This is a bias term.

[0098] Introducing sentence context vector u s It is also randomly initialized and jointly learned during the training process. Then, based on u i i∈[1,M] and u s The similarity is used to measure the importance of all sentences relative to text t, and the normalized importance weight β is obtained through the softmax function. i :

[0099]

[0100] Where β i Sentence vector Importance weights relative to the text t. The superscript T indicates the transpose of the vector.

[0101] Finally, calculate the vectors of all sentences based on the obtained weights. The weighted sum is used as the final text vector t:

[0102]

[0103] Where t is the text feature vector that summarizes all sentence information in the document.

[0104] S4: Use a pre-trained VGG19 to obtain visual information from the news image data, including the following steps:

[0105] To effectively extract visual features, we first fine-tune the VGG19 model pre-trained on ImageNet. Considering the inconsistent information density and importance of different spatial regions in an image—that is, different regions of the image have different importance for semantic understanding—fusing all entity information might lead to information redundancy or even introduce noise. Therefore, we introduce an attention mechanism model. For input image I, we can obtain a 7×7×512 feature map from the last convolutional layer of the VGG19 network, and further represent it as a sequence of feature vectors. H = 49, where v h Let v represent the feature vector of the image region corresponding to the h-th image patch. Through an attention mechanism, we measure the feature vector v of different regions in each image. h Compared to the importance of text t, the representations of information from all regions of the image are aggregated to obtain the visual feature vector representation of the entire image.

[0106] Specifically, we first use a linear transformation on the feature vector v of the image region. h Perform the transformation and obtain v using the hyperbolic tangent function. h Hidden representation u h :

[0107] u h =tanh(W H v h +b H )v h b H W H

[0108] Where W is H Parameter matrix, b H This is a bias term.

[0109] Then we use the feature vector v of each image region h The importance of an image region relative to the text is measured by the similarity to the text feature t, and a normalized importance weight γ is obtained through the softmax function. h :

[0110]

[0111] Where γ h The image region feature vector v h Importance weights relative to the text t. The superscript T indicates the transpose of the vector.

[0112] Finally, based on the obtained weights, the feature vector v of all image regions is calculated. hThe weighted sum is used as a visual feature vector representation.

[0113]

[0114] Furthermore, we added a fully connected layer to the last layer of the VGG19 network, adjusting the dimension of the final visual feature representation to P. Therefore, the final visual vector features can be represented as... P = 32.

[0115] S5: Combining text and image features to comprehensively detect and classify rumors, including the following steps:

[0116] C1: Concatenate the text feature vector t obtained in step S3 and the visual feature vector v obtained in step S4 to obtain the final multimodal representation of the news article:

[0117]

[0118] C2: After obtaining the multimodal representation x of the input news, we input it into a fully connected layer, and then pass the output of the fully connected layer through a softmax layer to generate the distribution of classification labels:

[0119] p = soft max(W) C x+b c )

[0120] Among them, W C Let b be the parameter matrix. C This is a bias term.

[0121] C3: We use cross-entropy as the loss function L for the model:

[0122] L=-∑[y log p+(1-y)log(1-p)]

[0123] Where y is the true label of the sample, 1 indicates that the sample is a rumor, and 0 indicates that the sample is not a rumor; p represents the probability that the sample is predicted to be a rumor. In this embodiment, if p exceeds the threshold of 0.5, the sample is judged to be a rumor, otherwise it is judged to be not a rumor.

[0124] Based on the above, to verify the practical effect of the method of the present invention, this embodiment applies the above scheme to a case study, selecting the FakeNewsNet dataset. This dataset is specifically collected for detecting rumors. FakeNewsNet contains tagged news from two websites: politifact.com and gossipcop.com. The news content includes text and image information, all tweets and retweets for each news item, and information about the corresponding Twitter users. We crawled the text and image information of each news item. The selected dataset information is shown in Table 1.

[0125] Table 1 Data Description

[0126]

[0127] The number is in units of "samples," representing the number of news data samples. T indicates that the sample is a rumor, and F indicates that the sample is not a rumor.

[0128] The method of this invention is compared with rumor detection models that utilize different information. Based on a single text modality, CSI uses LSTM to encode news content information for rumor detection. SAFE uses TextCNN to encode news text information. We also directly apply MLP to word2vec-encoded news text embeddings. Based on a single visual modality, VGG19 is widely used as a visual feature extractor; this invention uses a pre-trained VGG19 model fine-tuned on the dataset presented in this paper.

[0129] We also compared several multimodal methods used in recent years, including those based on text and images. EANN uses TextCNN and a pre-trained VGG19 algorithm for text and visual modal feature extraction for rumor detection, respectively. MVAE uses bidirectional LSTM and a pre-trained VGG19 algorithm for feature extraction.

[0130] Table 2 lists the results of the comparative experiment, from which we can draw the following conclusions:

[0131] 1) Methods based on a single text modality outperform those based on a single visual modality, indicating that rumor detection primarily relies on textual cues. Multimodal methods outperform single-modal methods with identical sub-network structures, demonstrating that text and image modalities can provide complementary cues for rumor detection tasks.

[0132] 2) In single-text modality-based methods, pre-trained language models outperform traditional text modeling methods such as CNN and RNN, indicating that pre-trained language models effectively learn linguistic knowledge from a large amount of pre-training corpus, and also demonstrating the necessity of GloVe pre-training.

[0133] 3) None of the methods showed significant improvement in prediction results on the POL dataset. This is likely because the POL dataset has too few samples, preventing any model from obtaining sufficient information for effective modeling. In contrast, the GOS dataset has a sufficient number of samples, and our method significantly outperformed other comparative methods on it, demonstrating that our proposed multimodal model based on hierarchical attention networks can indeed effectively improve rumor detection performance. This indicates that our model can detect rumors missed by existing methods by fully mining multimodal semantic cues.

[0134] Table 1 Performance comparison of different methods

[0135]

[0136] Where ACC represents the accuracy of the predicted classification, and F1 represents the F1 score of the predicted classification.

[0137] To verify the impact of different model components on the experimental results, we designed three model variants and performed elimination analysis on them. Furthermore, since the GOS dataset has sufficient samples to effectively assess model performance, we only conducted experiments on the GOS dataset.

[0138] 1) Remove the GloVe-based word2vec pre-training. Randomly initialize all word vectors in the news text before feeding them into a hierarchical attention network.

[0139] 2) Remove the hierarchical attention network. After pre-training all word vectors in the news text with GloVe, only the most basic bidirectional GRU is fed into it to encode the text information.

[0140] 3) Remove VGG19-based image feature extraction. Extract features from the text only using GloVe pre-training and a hierarchical attention network, ignoring image information in the news samples.

[0141] Figure 2 shows the experimental results of the elimination analysis, from which two conclusions can be drawn:

[0142] 1) Removing any part of the model will cause the classification accuracy of the model to decrease to some extent, which shows the effectiveness of each element of the model.

[0143] 2) Based on the decrease in model classification accuracy after removal, the importance of each model component can be ranked as follows: Hierarchical Attention > GloVe pre-training > VGG19. This indicates that for rumor detection tasks, text plays a more important role than images. Furthermore, because hierarchical attention can distinguish between highly important and unimportant content when constructing document representations, it can capture the features of the hierarchical structure of news documents, playing the greatest role in rumor detection tasks.

[0144] Based on the same inventive concept, the multimodal rumor detection system based on hierarchical attention networks described in this invention includes:

[0145] The preprocessing module is used to obtain word vectors from news texts using a GloVe pre-trained model;

[0146] The text feature extraction module is used to extract text feature vectors through a hierarchical attention model. Based on the importance between the current word and its context words in the sentence, it aggregates context word information to obtain the word vector representation. Based on the importance of all word vector representations in the sentence to the sentence, it aggregates word information to obtain the sentence vector. Based on the importance of the sentence vector to its context sentences in the text, it aggregates context sentence information to obtain the sentence vector representation. Based on the importance of all sentence vector representations in the text to the text, it aggregates sentence information to obtain the text feature vector.

[0147] The visual feature extraction module is used to extract visual feature vectors from news images using the VGG19 model.

[0148] The rumor detection module is used to determine whether a news sample is a rumor or not based on the text feature vector and the visual feature vector.

[0149] Based on the same inventive concept, the computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the multimodal rumor detection method based on a hierarchical attention network.

[0150] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer.

[0151] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

Claims

1. A multimodal rumor detection method based on hierarchical attention networks, characterized in that, The process includes the following: Using a GloVe pre-trained model to obtain word vectors from news text, extracting text feature vectors through a hierarchical attention model, extracting visual feature vectors from news images using a VGG19 model, and determining whether a news sample is a rumor or not based on the text and visual feature vectors. The method for extracting text feature vectors using the hierarchical attention model involves aggregating contextual word information to obtain the word's word vector representation based on the importance of the current word and its surrounding words in the sentence; aggregating word information to obtain a sentence vector based on the importance of all word vectors in the sentence to the sentence; aggregating contextual sentence information to obtain the sentence's sentence vector representation based on the importance of the sentence vector to its surrounding sentences in the text; and aggregating sentence information to obtain a text feature vector based on the importance of all sentence vectors in the text to the text. The method for aggregating contextual word information to obtain the word vector representation based on the importance of the current word and its surrounding words in the sentence involves: aggregating contextual word information based on the importance of the current word and its surrounding words in the sentence to obtain the word vector representation; ... based on the importance of all sentence vectors in the text to the text; and a method for aggregating contextual word information based on the importance of the current word and its surrounding words in the sentence to obtain the word vector representation based on the importance of the surrounding words in the sentence. Relative to the current word Importance weight Calculate the weighted sum as the word vector representation of the current word. ； ； ;in ， ， ， 、 and Here, N is the parameter matrix in the attention mechanism, and N is the number of words. The method for aggregating word information to obtain the sentence vector based on the importance of all word vectors in the sentence is as follows: based on the word vector representation... Importance weight of the current sentence Calculate the weighted sum as the sentence vector ； ； ;in, yes The hidden representation, ， For the parameter matrix, For bias terms; The word context vectors are randomly initialized and jointly learned during training. The method for aggregating context sentence information to obtain the sentence vector representation is based on the importance of the sentence vector to its surrounding sentences in the text. Relative to the current sentence Importance weight Calculate the weighted sum as the sentence vector representation of the current sentence. ； ； ;in, ， ， ， 、 and Here, M is the parameter matrix in the attention mechanism, and M is the number of sentences. The method for aggregating sentence information to obtain text feature vectors based on the importance of all sentence vectors in the text is as follows: based on the sentence vector representation... Relative to text Importance weight Calculate the weighted sum as a text feature vector ； ； ;in, yes The hidden representation, ， For the parameter matrix, For bias terms; The sentence context vector is randomly initialized and jointly learned during training.

2. The multimodal rumor detection method based on hierarchical attention networks according to claim 1, characterized in that, The method for obtaining word vectors from news text using a GloVe pre-trained model is as follows: Word vectors are initialized using a GloVe pre-trained model learned on a corpus, and the word vector loss function J is: ； Where V is the size of the vocabulary, It is a co-occurrence matrix constructed based on the corpus, where each element... This represents the number of times word p and its context word q co-occur within a context window of a specific size. and These represent any two word vectors in the corpus, which are the word vectors to be pre-trained and solved. and It is a bias term; For weighted functions, For the threshold, This represents the maximum number of times the two elements appear together.

3. The multimodal rumor detection method based on hierarchical attention networks according to claim 1, characterized in that, The method for extracting visual feature vectors from news images using the VGG19 model is as follows: The feature map of the news image is represented as a sequence of feature vectors. ,in Indicates the first The image region feature vector corresponding to each image patch, based on The importance weights relative to text t are calculated and weighted to form a visual feature vector v; ； ； ;in, yes The hidden representation, ， For the parameter matrix, H is the bias term; H is the number of image patches, and P is the dimension of the visual feature vector.

4. The multimodal rumor detection method based on hierarchical attention networks according to claim 1, characterized in that, The method for determining whether a news sample is a rumor or not based on the text feature vector and visual feature vector is as follows: The text feature vector and visual feature vector are concatenated to obtain a multimodal representation x of the news; x is then input into a fully connected layer and an activation layer to obtain the probability distribution of the sample being predicted as a rumor. ,in, For the parameter matrix, This is the bias term; if p exceeds the threshold, the sample is judged as a rumor; otherwise, it is judged as not a rumor.

5. A multimodal rumor detection system based on a hierarchical attention network according to the method of claim 1, characterized in that, include: The preprocessing module is used to obtain word vectors from news texts using a GloVe pre-trained model; the text feature extraction module is used to extract text feature vectors using a hierarchical attention model; based on the importance between the current word and its context words in the sentence, the context word information is aggregated to obtain the word vector representation; based on the importance of all word vector representations in the sentence to the sentence, the word information is aggregated to obtain the sentence vector; based on the importance between the sentence vector and its context sentences in the text, the context sentence information is aggregated to obtain the sentence vector representation; based on the importance of all sentence vector representations in the text to the text, the sentence information is aggregated to obtain the text feature vector; the visual feature extraction module is used to extract visual feature vectors from news images using a VGG19 model; and the rumor detection module is used to determine whether a news sample is a rumor or not based on the text feature vectors and visual feature vectors.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal rumor detection method based on a hierarchical attention network according to any one of claims 1-4.

Images