A rumor detection method and system based on multimodal interaction

Abstract

The present invention provides a rumor detection method and system based on multimodal interaction, including: obtaining a number of tweets to be detected; extracting sequence-enhanced text features of each tweet to be detected; extracting sequence-enhanced visual features of each tweet to be detected; based on the sequence-enhanced text features and the sequence-enhanced visual features, using an interactive attention mechanism to obtain visual features with text attention and text features with visual attention, and generating visual features and text features based on the visual features with text attention and the text features with visual attention; performing feature fusion on the visual features with text attention, the text features with visual attention, the visual features and the text features, and obtaining a result of whether each tweet to be detected is a rumor based on the fused features. The method can effectively fuse the multimodal data in the tweets to be detected and generate a joint representation, thereby improving the accuracy of rumor detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a rumor detection method and system based on multimodal interaction. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the improvement of hardware technology and the progress of network technology, social media has achieved unprecedented development and has become an indispensable part of people's daily life. Due to its low cost, easy access and rapid transmission speed, social media has become an important source of news and supports traditional media. Information on social media has been used in many fields such as marketing, public event discussion and public opinion analysis, and rumor detection. Users can easily publish opinions, obtain news, and communicate with each other on social media platforms. At the same time, false news and rumors also spread rapidly through social media, misleading the public and even causing adverse social impacts. Given that manual detection is expensive and slow, automatic rumor detection methods are particularly important.

[0004] Previous studies have focused on using manual features or single-modal features based on deep learning for rumor detection. The disadvantage is that they ignore the complementarity of multimodal data in tweets. Compared with the text content in tweets, visual elements such as videos and images in tweets are more likely to attract readers' attention. Therefore, there are also some methods that comprehensively analyze the multimodal content in tweets for rumor detection. However, most of these methods fuse data in heterogeneous modal spaces, which cannot avoid the numerical differences between multimodal data and make it difficult to capture the dependencies between heterogeneous modal data. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a rumor detection method and system based on multimodal interaction, which can effectively fuse the multimodal data in the tweets to be detected and generate a joint representation, thereby improving the accuracy of rumor detection.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a rumor detection method based on multimodal interaction, comprising:

[0008] Obtain several tweets to be detected that contain text and visual content;

[0009] Based on the text content of each tweet to be detected, extract the sequence enhanced text features of each tweet to be detected;

[0010] Based on the visual content of each tweet to be detected, extract the sequence enhanced visual features of each tweet to be detected;

[0011] Based on the sequence enhanced text features and the sequence enhanced visual features, an interactive attention mechanism is used to obtain visual features with text attention and text features with visual attention, and based on the visual features with text attention and the text features with visual attention, visual features and text features are generated;

[0012] Fusing visual features with text attention, text features with visual attention, and visual features and text features to obtain fused features;

[0013] Based on the fused features, we can get the result of whether each tweet to be detected is a rumor.

[0014] Furthermore, the specific method of generating visual features and text features is:

[0015] The text features with visual attention are used as the input matrix, and the visual features with text attention are used as the target matrix. The visual features are obtained through Transformer mapping.

[0016] The visual features with text attention are used as the input matrix, and the text features with visual attention are used as the target matrix, and the text features are obtained through Transformer mapping.

[0017] Furthermore, the specific method of the feature fusion is:

[0018] After concatenating the visual features with text attention with the visual features, the first early fusion result is obtained through a fusion recognition block composed of fully connected layers;

[0019] After concatenating the text features with visual attention with the text features, the second early fusion result is obtained through a fusion recognition block composed of a fully connected layer;

[0020] The first early fusion result and the second early fusion result are fused by assigning weights to obtain fused features.

[0021] Furthermore, the specific method for extracting the sequence enhanced text features of each tweet to be detected is:

[0022] Perform text preprocessing on the text content of each tweet to be detected, and obtain the primary text feature representation of each tweet to be detected based on the preprocessed text content;

[0023] Based on the primary text feature representation of each tweet to be detected, a text feature with context information of each tweet to be detected is obtained;

[0024] The self-attention mechanism is used to enhance the text features with contextual information and obtain sequence-enhanced text features.

[0025] Furthermore, the text preprocessing includes: deleting stop words, user handles and punctuation marks in each tweet to be detected.

[0026] Furthermore, the specific method for extracting the sequence enhanced visual features of each tweet to be detected is:

[0027] Perform image preprocessing on the visual content of each tweet to be detected, and obtain the regional features of each tweet to be detected based on the preprocessed visual content;

[0028] Based on the regional features of each tweet to be detected, the sequence enhanced visual features are obtained through the sequence attention block.

[0029] Furthermore, the image preprocessing includes: image size adjustment, random horizontal flipping and pixel value standardization.

[0030] A second aspect of the present invention provides a rumor detection system based on multimodal interaction, comprising:

[0031] A data acquisition module is configured to: acquire a number of tweets to be detected that contain text content and visual content;

[0032] A sequence enhanced text feature extraction module is configured to: extract a sequence enhanced text feature of each tweet to be detected based on the text content of each tweet to be detected;

[0033] A sequence enhanced visual feature extraction module, configured to: extract a sequence enhanced visual feature of each tweet to be detected based on the visual content of each tweet to be detected;

[0034] An interactive module, configured to: obtain visual features with text attention and text features with visual attention based on the sequence-enhanced text features and the sequence-enhanced visual features by using an interactive attention mechanism, and generate visual features and text features based on the visual features with text attention and the text features with visual attention;

[0035] A feature fusion module is configured to: fuse visual features with text attention, text features with visual attention, and visual features and text features to obtain fused features;

[0036] The detection module is configured to: obtain the result of whether each tweet to be detected is a rumor based on the fused features.

[0037] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the rumor detection method based on multimodal interaction as described above.

[0038] A fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the rumor detection method based on multimodal interaction as described above are implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention provides a rumor detection method based on multimodal interaction, which obtains time-series enhanced single-modal features and performs cross-modal mapping of different modal data, which can effectively reduce the impact of heterogeneous differences; and adopts an interactive attention mechanism to capture the dependency relationship between multimodal data, thereby improving the accuracy of rumor detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0043] Figure 2 is a flow chart of text feature extraction according to the first embodiment of the present invention;

[0044] Figure 3 is a comparison chart of early detection performance on a microblog data set according to the first embodiment of the present invention;

[0045] Figure 4 4 is a comparison chart of early detection performance on the Twitter dataset of Example 1 of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] Embodiment 1

[0050] This embodiment provides a rumor detection method based on multimodal interaction. Figure 1 As shown, the specific steps include:

[0051] Step 1: Get several tweets to be detected that contain text and visual content.

[0052] The obtained tweets to be detected can be expressed as Tw = {T, V}, where T and V represent the text content and visual content (image) in all tweets to be detected, respectively. That is, each tweet to be detected includes multimodal content (text content and visual content).

[0053] Among them, the text content of all tweets to be detected is T = {T1, T2, ..., T N}, T i ∈T represents the text content of the i-th tweet to be detected, N represents the total number of tweets to be detected, T i ={t1,t2,...,t L} represents the character sequence of the i-th tweet to be detected, L represents the length of the sequence; the visual content V in all tweets to be detected is V = {V1, V2, ..., V N},V i ∈V represents the visual content of the i-th tweet to be detected.

[0054] As an implementation method, the tweet to be detected may come from Weibo.

[0055] Step 2: Input several tweets to be detected into a multimodal rumor detection model (TIMF model) that integrates text content and visual content to detect whether each tweet to be detected is a rumor. The output of the TIMF model is the result of rumor detection: rumor (N) or non-rumor (NR). The specific steps include:

[0056] Step 201: Figure 2 As shown, based on the text content T of each tweet to be detected i ={t1,t2,...,t L}, extract the sequence enhanced text features F of each tweet to be detected T={t1,t2,...,t m}, that is, the text content of all tweets to be detected is input into the text block, which can be expressed as a function F T =f T (T i θ T ), where F T represents the text features of the text block output, θ T Parameters representing a text block:

[0057] (1) For each tweet to be detected, the text content T i ={t1,t2,...,t L} to perform text preprocessing, and based on the preprocessed text content, obtain the primary text feature representation H = {h(0), h(1), ..., h(L1)} of each tweet to be detected, including:

[0058] ① Remove stop words, user handles, and punctuation from each tweet to be tested;

[0059] ② Call Bert tokenizer to segment the processed text and obtain the primary text feature representation H = {h(0), h(1), ..., h(L1)} of each tweet to be detected.

[0060] Specifically, we use a fine-tuned pre-trained Bert model to extract text features. The Bert model based on Transformer (Trm) and Attention mechanism is one of the best models in various NLP tasks. In addition, Bert is more suitable for short texts such as Weibo because of its predefined length. Bert's input consists of three parts: Token Embedding, Segment Embedding, and Position Embedding. i ), E(seg i ), E(pos i ) represent the vector representation of the character, the index of the sentence and the position index of the character respectively.

[0061] Bert’s input BI = {I(0), I(1), ..., I(L)}, where I(i) is the sum of the elements of the i-th character of each tweet to be detected after preprocessing, that is, I(i) = E(tok i )+E(seg i )+E(pos i ), where E(tok i ), E(seg i ), E(pos i ) represent the vector representation of the character, the index of the sentence and the position index of the character respectively. Figure 2 In the example, cls and sep are tags added to the beginning and end of a sentence to indicate the start and end of a sentence, ti represents the vector representation of the ith character, and A represents the sentence index. Since the Bert task in the present invention is feature extraction rather than question answering, A is used to refer to all characters belonging to a sentence, and L represents the number of characters in the sentence. Then, the fine-tuned Bert is used to extract the primary text feature representation H = {h(0), h(1), ..., h(L1)} of the text content, that is,

[0062] H=Bert ft (BI)

[0063] (2) In order to better capture global information, the primary text feature representation H = {h(0), h(1), ..., h(L1)} of each tweet to be detected is input into the sequential attention block (SA module), which consists of Bi-GRU and Self-Attention.

[0064] Based on the primary text feature representation H = {h(0), h(1), ..., h(L1)} of each tweet to be detected, Bi-GRU is used to extract the temporal attributes of the text features of each tweet to be detected to obtain a text feature representation G = {g(0), g(1), ..., g(L1)} with contextual information

[0065] G=Bi-GRU(H)

[0066] The self-attention mechanism can be used to enhance the text features with contextual information, and combined with the addition and normalization operation (Add&Norm) and the full connected layer (Full Connected Layer (2)), the sequence-enhanced text features F T ={t1,t2,...,t m}:

[0067] q i =w q ×g(i),k i =w k ×g(i),p i =w p ×g(i)

[0068]

[0069] att_G={att_g(0),att_g(1),...,att_g(L1)}

[0070] F' T=Add&Norm(G,att_G)

[0071] F T =σ(w fc gF′ T +b fc )

[0072] Among them, q i ,k i ,v i Represents the linear transformation matrix (intermediate matrix) in the attention mechanism, w q ,w k ,w v represents the weight matrix, g(i) is the i-th eigenvalue in the text feature representation with contextual information, and w fc represents the weight matrix of the fully connected layer, b fc represents the bias matrix of the fully connected layer, and σ represents the activation function ReLU.

[0073] Step 202: Based on the visual content V of each tweet to be detected i , extract the sequence enhanced visual features F of each tweet to be detected V ={v1,v2,...,v m}, that is, the visual content of all tweets to be detected is input into the visual block, which can be expressed as a function F V =f V (V i θ V ), where F V represents the visual features output by the visual module, θ V Represents the parameters of the vision module:

[0074] (1) The visual content of each tweet to be tested is preprocessed. Specifically, the image size is adjusted to 224×224. In order to increase the number of training samples and improve the generalization performance of the model, the image is augmented by random horizontal flipping. The pixel values ​​are normalized in the range of [0-1].

[0075] (2) The ResNet50 model pre-trained on ImageNet is used to extract the regional features res_F of the pre-processed visual content V ={v′1, v′2,...,v′ m}, change the last fully connected layer so that the output sequence length is consistent with the text features.

[0076] res_F V =ResNet ft (V)

[0077] (3) Similar to the text module, the SA module is used to obtain the sequence enhanced visual features F V = {v1, v2, ..., v m}:

[0078] F V =SA(res_F V )

[0079] Step 203: Input the sequence enhanced text features and the sequence enhanced visual features into the multimodal fusion block to obtain the rumor detection result, which can be expressed as a function

[0080] Result = f F ({F V , F T};θ F )

[0081] Among them, Result represents the output of the module (R or NR), θ F Represents the parameters of the multimodal fusion block.

[0082] (1) Cross-modal feature mapping

[0083] (101) There are more or less numerical differences and dependencies between data of different modalities. In order to achieve better fusion effect, cross-modal feature mapping is needed to capture the dependencies between heterogeneous modal data and reduce the numerical differences. First, for heterogeneous modal data, the sequence enhanced text feature F T ={t1, t2, ..., t m} and sequence enhanced visual features F V = {v1, v2, ..., v m}, using the interactive attention mechanism (Attention(1)) to obtain visual features with text attention

[0084]

[0085] q i =tanh(w q ·t i +b q )

[0086] k i =tanh(w k ·v i +b k )

[0087] p i =tanh(w p ·v i +b p )

[0088]

[0089] Similarly, using the interactive attention mechanism (Attention(2)) can obtain text features with visual attention This step is to highlight the regions in the image and the characters in the text and initially capture the dependencies between the modalities.

[0090] (102) Transformer is used for cross-modal feature sequence mapping. Transformer is essentially an encoding-decoding framework, which is mainly used to deal with sequence-to-sequence problems in the field of NLP. It consists of six encoding layers and six decoding layers. It is used as a medium for heterogeneous modal feature sequence mapping. Since the sequence-enhanced feature matrix has been obtained through the above SA-module, for the first Transformer (Transformer (1)), the text feature with visual attention is used as input to generate the visual feature gen_F V = Transformer{src:att_F T ,tgt:att_F V}, where src represents the source domain and tgt represents the target domain. The generated visual features are generated by the source visual features att_F T As input matrix, take att_F V is the target matrix, which is obtained through Transformer mapping. Therefore, it reduces the heterogeneity difference between the source visual features and the generated visual features and further captures the dependency between heterogeneous modal data, which can achieve better fusion effect.

[0091] Similarly, we can obtain the generated text feature gen_F using the visual feature with text attention as the input of the second Transformer (Transformer (2)). T Specifically, the visual features with text attention are used as the input matrix, the text features with visual attention are used as the target matrix, and the text features are obtained through Transformer mapping.

[0092] (2) Integration

[0093] Early fusion includes three fully connected layers (the first fully connected layer Fc-1, the second fully connected layer Fc-2, and the third fully connected layer Fc-3), and Dropout layers (the first Dropout layer Dropout-1 and the second Dropout layer Dropout-2) are added between the fully connected layers to prevent overfitting. The source features are concatenated with the generated features and sent to the fully connected layer to obtain the early fusion result:

[0094] After concatenating the visual features with text attention with the visual features, the first early fusion result is obtained through a fusion recognition module composed of fully connected layers, where EF refers to early fusion:

[0095] Result-1=EF{att_F V ,gen_F V}

[0096] After concatenating the text features with visual attention with the text features, the second early fusion result is obtained through a fusion recognition module composed of a fully connected layer:

[0097] Result-2 = EF{att_F T ,gen_F T}

[0098] Late fusion fuses the results from early fusion by assigning weights. The first early fusion result and the second early fusion result are fused by assigning weights, where LF refers to late fusion, to obtain the fused features, i.e., the late fusion result:

[0099] Result=LF{Result-1,Result-2}

[0100] The loss of the module is defined using cross entropy, and Y is used to represent the event label, and Result(i) represents the prediction result of the i-th tweet:

[0101]

[0102] By seeking the optimal parameters To minimize the module loss function L F :

[0103]

[0104] The present invention can effectively and comprehensively analyze the multimodal content in the text to be detected; can map the multimodal content to the same data domain to generate an effective joint representation; and has a good detection effect even in the case of missing modalities and small amounts of early data.

[0105] The proposed model is validated on Weibo and Twitter datasets. Weibo dataset is a multimodal dataset, in which rumor tweets are obtained by crawling all verified false rumor posts from May 2012 to January 2016. The dataset has objective ground truth labels. The non-rumor tweets are taken from tweets verified by authoritative news organizations. The Twitter dataset comes from a part of MediaEval's task to detect false content on Twitter, and tweets without any text or images are deleted.

[0106] Table 1. Dataset details

[0107]

[0108] As shown in Table 2, corresponding ablation experiments are carried out on the TIMF model in the present invention, which are various modules and simplified variants in TIMF.

[0109] Among them, Visual: only the pre-trained ResNet50 model in the visual module is used for verification on single-modal visual content; Visual-SA: the complete visual module including the SA-module is used for verification on single-modal visual content; Texual: only the fine-tuned Bert model in the text module is used for verification on single-modal text content; Texual-SA: the complete text module including the SA-module is used for verification on single-modal text content; EF-1: an early recognition model that only uses the fusion of visual source features and visual generated features; EF-2: an early recognition model that only uses the fusion of text source features and text generated features; TMIF: the complete model proposed in this paper. Accuracy: accuracy; Precision: precision; Recall: recall rate; F1: F1 index.

[0110] Table 2. Ablation experiment

[0111]

[0112] As shown in Table 3, the TIMF model in the present invention is compared with representative algorithms in related work, where K represents knowledge information and S represents social context. Considering the differences in task settings, the main network structures of these models are adopted and necessary adjustments are made for the tasks. Among them, att-RNN is a multimodal rumor detection model (Multimodal fusion with recurrent neural networks for rumor detection on microblogs) proposed in the journal The 25th ACM international conference on Multimedia in 2017. Its input is the text, visual and social context information in the tweet; EANN is a rumor detection network (Eann: Event adversarial neural networks for multi-modal fake news detection) proposed in the journal The 24th acm sigkdd international conference on knowledge discovery & data mining in 2018. The time discriminator is deleted, and only the multimodal feature extractor and fake news detector are retained; MAVE is a multimodal variational autoencoder (Mvae: Multimodal variational autoencoder for fakenews detection) proposed in the journal The world wide web conference in 2019. For the task, its complete structure is used, including encoder, decoder and detector; MFN is a rumor detection network (Eann: Event adversarial neural networks for multi-modal fake news detection) proposed in the journal Pacific-Asia conference on knowledge discovery and data mining in 2018. An unsupervised rumor detection model proposed in mining (Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection). For the task, Text-CNN is used to capture text features of different granularities, and the pre-trained VGG-19 model is used to capture image features. Finally, the attention mechanism and the latent subject memory network are used to generate a joint representation.MKEMN is a rumor detection method (Multi-modalknowledge-aware event memory network for social media rumor detection) proposed in the journal Proceedings of the 27th ACMInternational Conference on Multimedia in 2019. It uses its complete structure to fuse information from three modalities: text, image, and knowledge information for rumor detection. ;

[0113] Table 3. Comparative experiments with related algorithms

[0114]

[0115] like Figure 3 , Figure 4 As shown, a comparative study is conducted on the early detection performance of the TIMF model in the present invention. For this purpose, 20% of the test samples are fixed, and the early detection capability of the model is observed by gradually increasing the training samples.

[0116] Embodiment 2

[0117] This embodiment provides a rumor detection system based on multimodal interaction, which specifically includes the following modules:

[0118] A data acquisition module is configured to: acquire a number of tweets to be detected that contain text content and visual content;

[0119] A sequence enhanced text feature extraction module is configured to: extract a sequence enhanced text feature of each tweet to be detected based on the text content of each tweet to be detected;

[0120] A sequence enhanced visual feature extraction module, configured to: extract a sequence enhanced visual feature of each tweet to be detected based on the visual content of each tweet to be detected;

[0121] An interactive module, configured to: obtain visual features with text attention and text features with visual attention based on the sequence-enhanced text features and the sequence-enhanced visual features by using an interactive attention mechanism, and generate visual features and text features based on the visual features with text attention and the text features with visual attention;

[0122] A feature fusion module is configured to: fuse visual features with text attention, text features with visual attention, and visual features and text features to obtain fused features;

[0123] The detection module is configured to: obtain the result of whether each tweet to be detected is a rumor based on the fused features.

[0124] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0125] Embodiment 3

[0126] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the rumor detection method based on multimodal interaction as described in the above-mentioned embodiment 1 are implemented.

[0127] Embodiment 4

[0128] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the rumor detection method based on multimodal interaction as described in the first embodiment above are implemented.

[0129] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A rumor detection method based on multimodal interaction, characterized in that: include: Obtain several tweets to be detected that contain text and visual content; Based on the text content of each tweet to be detected, extract the sequence enhanced text features of each tweet to be detected; Based on the visual content of each tweet to be detected, extract the sequence enhanced visual features of each tweet to be detected; Based on the sequence enhanced text features and the sequence enhanced visual features, an interactive attention mechanism is used to obtain visual features with text attention and text features with visual attention, and based on the visual features with text attention and the text features with visual attention, visual features and text features are generated; Fusing visual features with text attention, text features with visual attention, and visual features and text features to obtain fused features; The specific method of feature fusion is: After concatenating the visual features with text attention with the visual features, the first early fusion result is obtained through a fusion recognition block composed of fully connected layers; After concatenating the text features with visual attention with the text features, the second early fusion result is obtained through a fusion recognition block composed of a fully connected layer; The first early fusion result and the second early fusion result are fused by assigning weights to obtain fused features; Early fusion includes three fully connected layers: the first fully connected layer Fc-1, the second fully connected layer Fc-2, and the third fully connected layer Fc-3. Dropout layers are added between the fully connected layers: the first Dropout layer Dropout-1 and the second Dropout layer Dropout-2. The source features are concatenated with the generated features and sent to the fully connected layer to obtain the early fusion result: After concatenating the visual features with text attention with the visual features, the first early fusion result is obtained through a fusion recognition module composed of fully connected layers, where EF refers to early fusion: Result-1=EF{att_F V ,gen_F V }; After concatenating the text features with visual attention with the text features, the second early fusion result is obtained through a fusion recognition module composed of a fully connected layer: Result-2=EF{att_F T ,gen_F T }; Late fusion fuses the results from early fusion by assigning weights. The first early fusion result and the second early fusion result are fused by assigning weights, where LF refers to late fusion, to obtain the fused features, i.e., the late fusion result: Result=LF{Result-1,Result-2}; The loss of the module is defined using cross entropy, and Y is used to represent the event label, and Result(i) represents the prediction result of the i-th tweet: By seeking the optimal parameters To minimize the module loss function L F : Based on the fused features, we can get the result of whether each tweet to be detected is a rumor.

2. A rumor detection method based on multimodal interaction as claimed in claim 1, characterized in that: The specific method of generating visual features and text features is: The text features with visual attention are used as the input matrix, and the visual features with text attention are used as the target matrix. The visual features are obtained through Transformer mapping. The visual features with text attention are used as the input matrix, and the text features with visual attention are used as the target matrix, and the text features are obtained through Transformer mapping.

3. The rumor detection method based on multimodal interaction as claimed in claim 1, characterized in that: The specific method for extracting the sequence enhanced text features of each tweet to be detected is: Perform text preprocessing on the text content of each tweet to be detected, and obtain the primary text feature representation of each tweet to be detected based on the preprocessed text content; Based on the primary text feature representation of each tweet to be detected, a text feature with context information of each tweet to be detected is obtained; The self-attention mechanism is used to enhance the text features with contextual information and obtain sequence-enhanced text features.

4. The rumor detection method based on multimodal interaction as claimed in claim 3, characterized in that: The text preprocessing includes: deleting stop words, user handles and punctuation marks in each tweet to be detected.

5. The rumor detection method based on multimodal interaction as claimed in claim 1, characterized in that: The specific method for extracting the sequence enhanced visual features of each tweet to be detected is: Perform image preprocessing on the visual content of each tweet to be detected, and obtain the regional features of each tweet to be detected based on the preprocessed visual content; Based on the regional features of each tweet to be detected, the sequence enhanced visual features are obtained through the sequence attention block.

6. A rumor detection method based on multimodal interaction as claimed in claim 5, characterized in that: The image preprocessing includes: image size adjustment, random horizontal flipping and pixel value standardization.

7. A rumor detection system based on multimodal interaction, characterized in that: include: A data acquisition module is configured to: acquire a number of tweets to be detected that contain text content and visual content; A sequence enhanced text feature extraction module is configured to: extract a sequence enhanced text feature of each tweet to be detected based on the text content of each tweet to be detected; A sequence enhanced visual feature extraction module, configured to: extract a sequence enhanced visual feature of each tweet to be detected based on the visual content of each tweet to be detected; An interactive module, configured to: obtain visual features with text attention and text features with visual attention based on the sequence-enhanced text features and the sequence-enhanced visual features by using an interactive attention mechanism, and generate visual features and text features based on the visual features with text attention and the text features with visual attention; The feature fusion module is configured to: fuse the visual features with text attention, the text features with visual attention, and the visual features and text features to obtain fused features; the specific method of the feature fusion is: After concatenating the visual features with text attention with the visual features, the first early fusion result is obtained through a fusion recognition block composed of fully connected layers; After concatenating the text features with visual attention with the text features, the second early fusion result is obtained through a fusion recognition block composed of a fully connected layer; The first early fusion result and the second early fusion result are fused by assigning weights to obtain fused features; Early fusion includes three fully connected layers: the first fully connected layer Fc-1, the second fully connected layer Fc-2, and the third fully connected layer Fc-3. Dropout layers are added between the fully connected layers: the first Dropout layer Dropout-1 and the second Dropout layer Dropout-2. The source features are concatenated with the generated features and sent to the fully connected layer to obtain the early fusion result: After concatenating the visual features with text attention with the visual features, the first early fusion result is obtained through a fusion recognition module composed of fully connected layers, where EF refers to early fusion: Result-1=EF{att_F V ,gen_F V }; After concatenating the text features with visual attention with the text features, the second early fusion result is obtained through a fusion recognition module composed of a fully connected layer: Result-2=EF{att_F T ,gen_F T }; Late fusion fuses the results from early fusion by assigning weights. The first early fusion result and the second early fusion result are fused by assigning weights, where LF refers to late fusion, to obtain the fused features, i.e., the late fusion result: Result=LF{Result-1,Result-2}; The loss of the module is defined using cross entropy, and Y is used to represent the event label, and Result(i) represents the prediction result of the i-th tweet: By seeking the optimal parameters To minimize the module loss function L F : The detection module is configured to: obtain the result of whether each tweet to be detected is a rumor based on the fused features.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a rumor detection method based on multimodal interaction as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the rumor detection method based on multimodal interaction as described in any one of claims 1-6 are implemented.

Images