A multi-modal rumor detection method and system

Abstract

The application provides a multi-modal rumor detection method and system, belonging to the technical field of big data processing, comprising: acquiring event detection source data; using a pre-trained transformer model to extract text features and visual features of the event detection source data; based on a cross-attention mechanism, performing multi-modal fusion on the text features and the visual features to obtain multi-modal fusion features; performing rumor detection on the multi-modal fusion features to obtain rumor event detection results. By introducing a pre-trained transformer model to extract text features and visual features respectively, and performing detection and classification after multi-modal fusion, compared with existing methods, the application better utilizes the information of multi-modal posts on social media, and has better detection effect and stronger robustness.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a multimodal rumor detection method and system. Background Technology

[0002] With the rapid development of communication technology and smart terminals, the spread of online rumors is also increasing exponentially. From its initial single text format, it has evolved into a multimodal form combining images and text, making it more concealed and harmful. Rumors, especially in the fields of health and medical care, food safety, and public safety, can cause significant harm to society. Therefore, the timely and accurate identification of multimodal rumors combining images and text is of paramount importance.

[0003] Currently, most rumor detection methods based on single-modal content utilize text or image features alone. However, these methods do not fully leverage all the content information in a tweet. Therefore, researchers have begun to focus on rumor detection methods based on multimodal content, typically employing methods that integrate text and image features to identify fake news and improve model effectiveness. Due to the different nature of text and image information, most multimodal rumor detection methods extract text and image features separately and then directly concatenate the vectors to fuse multimodal information. While this direct vector concatenation method is simple to operate, it also has the limitation of being a single-method approach. It can be seen that single-modal rumor detection algorithms suffer from underutilization of text or image information, while general multimodal models have certain limitations in modality fusion methods, resulting in poor model generalization ability. Summary of the Invention

[0004] This invention provides a multimodal rumor detection method and system to address the shortcomings of existing technologies that use single-modal detection for rumor detection, which fails to fully utilize text or images, and general multimodal detection has limitations in modality fusion. The invention achieves multimodal rumor detection based on a pre-trained transformer model and a cross-attention mechanism.

[0005] In a first aspect, the present invention provides a multimodal rumor detection method, comprising:

[0006] Obtain event detection source data;

[0007] A pre-trained transformer model is used to extract textual and visual features from the event detection source data;

[0008] Based on the cross-attention mechanism, the text features and the visual features are fused in a multimodal manner to obtain multimodal fused features;

[0009] The multimodal fusion features are used for rumor detection to obtain rumor event detection results.

[0010] According to a multimodal rumor detection method provided by the present invention, after acquiring the event detection source data, the method further includes:

[0011] The text data in the event detection source data is preprocessed to obtain preprocessed text data. The text preprocessing includes filtering special symbols and word segmentation.

[0012] Image preprocessing is performed on the image data in the event detection source data to obtain preprocessed image data. The image preprocessing includes image format conversion, filtering invalid data, and unifying image size.

[0013] According to a multimodal rumor detection method provided by the present invention, a pre-trained transformer model is used to extract textual features from the event detection source data, including:

[0014] The preprocessed text data is converted into word vectors, wherein the word vectors include any text word and word vector dimensions;

[0015] Based on any given text word and the word vector dimension, a text input matrix is ​​obtained;

[0016] The text input matrix is ​​fed into the Text-CNN text classification convolutional neural network model to obtain text semantic features;

[0017] The text input matrix is ​​input into a pre-trained bidirectional encoder-decoder (BERT) to obtain text word features.

[0018] According to a multimodal rumor detection method provided by the present invention, a pre-trained transformer model is used to extract visual features from the event detection source data, including:

[0019] The preprocessed image data is input into the pre-trained ViT to obtain visual features.

[0020] According to a multimodal rumor detection method provided by the present invention, the method involves fusing text features and visual features in a multimodal manner based on a cross-attention mechanism to obtain multimodal fused features, including:

[0021] Based on the text features, a text query vector and a text key vector are determined. The text query vector and the text key vector are then multiplied by a matrix to obtain a text word attention matrix.

[0022] Based on the visual features, a visual query vector and a visual key vector are determined. The visual attention matrix is ​​obtained by matrix multiplication of the visual query vector and the visual key vector.

[0023] The text modal weights, visual modal weights, and bias values ​​are determined by weighted summation of the text word attention matrix, the text modal weights, the visual attention matrix, the visual modal weights, and the bias values, and then processed by the softmax logistic regression function to obtain the multimodal attention matrix.

[0024] The multimodal attention matrix is ​​multiplied by the text word features and then concatenated with the text semantic features to obtain the multimodal fusion feature.

[0025] According to a multimodal rumor detection method provided by the present invention, the step of performing rumor detection on the multimodal fusion features to obtain a rumor event detection result includes:

[0026] The multimodal fusion features are input into the weight matrix of the fully connected layer, summed with the bias values, and then processed by the softmax logistic regression function to obtain the rumor prediction probability.

[0027] Based on the rumor prediction probability, the rumor event detection result is determined.

[0028] Secondly, the present invention also provides a multimodal rumor detection system, comprising:

[0029] The acquisition module is used to acquire event detection source data;

[0030] The extraction module uses a pre-trained transformer model to extract textual and visual features from the event detection source data.

[0031] The fusion module is used to perform multimodal fusion of the text features and the visual features based on the cross-attention mechanism to obtain multimodal fused features;

[0032] The detection module is used to perform rumor detection on the multimodal fusion features and obtain the rumor event detection results.

[0033] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal rumor detection method as described above.

[0034] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal rumor detection method as described above.

[0035] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal rumor detection method as described above.

[0036] The multimodal rumor detection method and system provided by this invention extracts textual and visual features by introducing a pre-trained transformer model, and then performs multimodal fusion for detection and classification. Compared with existing methods, it makes better use of information from multimodal posts on social media, resulting in better detection performance and stronger robustness. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the multimodal rumor detection method provided by the present invention;

[0039] Figure 2 This is a feature extraction structure diagram of the multimodal rumor detection method provided by the present invention;

[0040] Figure 3 This is a feature fusion structure diagram of the multimodal rumor detection method provided by the present invention;

[0041] Figure 4 This is a diagram illustrating the rumor detection structure of the multimodal rumor detection method provided by this invention.

[0042] Figure 5 This is a schematic diagram of the structure of the multimodal rumor detection system provided by the present invention;

[0043] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] The following is combined with Figures 1-6 This invention describes a multimodal rumor detection method and system.

[0046] Figure 1 This is a flowchart illustrating the multimodal rumor detection method provided by the present invention, as shown below. Figure 1 As shown, it includes:

[0047] Step 100: Obtain event detection source data;

[0048] Step 200: Use a pre-trained transformer model to extract textual and visual features from the event detection source data;

[0049] Step 300: Based on the cross-attention mechanism, the text features and the visual features are fused in a multimodal manner to obtain multimodal fused features;

[0050] Step 400: Perform rumor detection on the multimodal fusion features to obtain rumor event detection results.

[0051] It should be noted that the subject executing the above method can be an electronic device, a component within an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This invention does not impose specific limitations.

[0052] Specifically, the event detection source data for rumors is preprocessed. This can be content from social media such as Weibo or Twitter. Generally, the text data is subjected to symbol filtering and word segmentation, and the image data is converted to a different format to remove invalid data and convert it to a uniform size.

[0053] Then, feature extraction is performed. Text-ConvolutionalNeural Networks (Text-CNN) is used to extract text semantic features from text information. Bidirectional Encoder Representations from Transformers (BERT) is used to extract text word features from text information. Vision in Transformers (Vit) is used to extract visual features from images.

[0054] Furthermore, the extracted text features and visual features are input into the multimodal feature fusion module, and the cross-attention mechanism is used to fuse the text features and visual features to obtain multimodal fused features.

[0055] Finally, the multimodal fusion features are input into the rumor detection module to perform multimodal rumor detection and obtain the rumor event detection results.

[0056] This invention proposes a multimodal rumor detection model based on a pre-trained transformer model and a cross-attention mechanism, aiming to improve the model's ability to learn multimodal features of rumors. Compared to single-modal pure text and pure image methods, the multimodal rumor detection method based on the pre-trained transformer model and cross-attention mechanism shows better performance on various evaluation metrics. This is because single-modal methods can only utilize text or image information in the dataset alone and cannot discover the potential connections between text and images.

[0057] This invention introduces a pre-trained transformer model to extract textual and visual features separately, and then performs multimodal fusion for detection and classification. Compared with existing methods, it makes better use of information from multimodal posts on social media, resulting in better detection performance and stronger robustness.

[0058] Based on the above embodiments, the method further includes the following after step 100:

[0059] The text data in the event detection source data is preprocessed to obtain preprocessed text data. The text preprocessing includes filtering special symbols and word segmentation.

[0060] Image preprocessing is performed on the image data in the event detection source data to obtain preprocessed image data. The image preprocessing includes image format conversion, filtering invalid data, and unifying image size.

[0061] Specifically, in order to better extract features from the input event detection source data, preprocessing is required for both text data and image data.

[0062] The text data is preprocessed by cleaning the short text sequences of rumors in the dataset using special symbols and word segmentation to obtain preprocessed text data.

[0063] Image data preprocessing mainly involves scaling and standardization operations, format conversion to remove invalid data, and conversion to a uniform size.

[0064] This invention preprocesses the event detection source data to give text and image data a unified processing format, making subsequent feature extraction and feature fusion more accurate.

[0065] Based on the above embodiments, step 200, which involves extracting text features from the event detection source data using a pre-trained transformer model, includes:

[0066] The preprocessed text data is converted into word vectors, wherein the word vectors include any text word and word vector dimensions;

[0067] Based on any given text word and the word vector dimension, a text input matrix is ​​obtained;

[0068] The text input matrix is ​​fed into the Text-CNN text classification convolutional neural network model to obtain text semantic features;

[0069] The text input matrix is ​​input into a pre-trained bidirectional encoder-decoder (BERT) to obtain text word features.

[0070] Specifically, such as Figure 2 As shown, the steps for extracting semantic features from text information using Text-CNN are as follows:

[0071] First, the input text is converted into an input vector, where the i-th word of the text is represented by e. i ∈R k Here, k represents the dimension of the word vectors. Therefore, the text input matrix for n words is represented as follows:

[0072] e i:n =[e0,e1,e2,...,e n ]

[0073] Where n represents the number of words in the sentence, e0 is the special symbol [CLS] added before each input example, and the Text-CNN model is denoted as f. Text-CNN , will e i:n Enter f Text-CNN Then, the semantic feature vector of the given sentence is obtained, calculated as follows:

[0074] Tc =f Text-CNN (e i:n )

[0075] The steps for extracting text word features using BERT are as follows: First, convert the input text into an input vector. The i-th word of the text is represented as e. i ∈R k Here, k represents the dimension of the word vectors. Therefore, the text input matrix for n words is represented as follows:

[0076] e i:n =[e0,e1,e2,...,e n ]

[0077] Where n represents the number of words in the sentence, e0 is the special symbol [CLS] added before each input example, and the pre-trained BERT-base model is denoted as f. Bert e i:n Input f Bert Then, the word feature vector matrix of the given sentence is obtained, calculated as follows:

[0078] T o =f Bert (e i:n )

[0079] Based on the above embodiments, step 200, which involves extracting visual features from the event detection source data using a pre-trained transformer model, includes:

[0080] The preprocessed image data is input into the pre-trained ViT to obtain visual features.

[0081] Specifically, visual features are first represented as v∈R q Let the pre-trained vit-base model be denoted as f. Vit The operation of the last layer in the image feature extractor can be represented as:

[0082] v o =f Vit (v)

[0083] This invention presents a multimodal rumor detection method based on pre-trained transformer models and cross-attention mechanisms. Besides using Text-CNN to extract semantic features from text information, it also employs pre-trained BERT and VitTransformer models to extract features from text and images, respectively. Pre-trained models have already been trained on massive amounts of data, better learning the generalized features within the data. Compared to training a model from scratch, using a pre-trained model results in better generalization performance.

[0084] Based on the above embodiments, step 300 includes:

[0085] Based on the text features, a text query vector and a text key vector are determined. The text query vector and the text key vector are then multiplied by a matrix to obtain a text word attention matrix.

[0086] Based on the visual features, a visual query vector and a visual key vector are determined. The visual attention matrix is ​​obtained by matrix multiplication of the visual query vector and the visual key vector.

[0087] The text modal weights, visual modal weights, and bias values ​​are determined by weighted summation of the text word attention matrix, the text modal weights, the visual attention matrix, the visual modal weights, and the bias values, and then processed by the softmax logistic regression function to obtain the multimodal attention matrix.

[0088] The multimodal attention matrix is ​​multiplied by the text word features and then concatenated with the text semantic features to obtain the multimodal fusion feature.

[0089] Specifically, such as Figure 3 As shown, this invention employs a multimodal feature fusion module to fuse text and image features for rumor detection, using a method based on a cross-attention mechanism to fuse text and image features, rather than simply concatenating text and image features.

[0090] After obtaining the text word features T o and visual features V o Subsequently, to enable full interaction between textual and visual information, this invention inputs them into a multimodal attention fusion module, adjusting word weights by combining the word performance across different modalities. The output X of the multimodal attention fusion module is then obtained. att Subsequently, the present invention is in T o and V o Residual joins are used to preserve the original structure of the data. Then X... att The input consists of a fully connected layer and a normalization layer. Finally, this invention yields a representation X of the multimodal features. att This invention uses it as an aggregate representation and inputs it into a fully connected layer to produce the final prediction result. Multimodal attention fusion is the core of the model, which aims to use information from the visual modality to help the text modality adjust the weight of words, thereby making comprehensive use of multimodal information.

[0091] First, this invention evaluates the weight of each word in different modalities. The query vector Q for the text modality... t and bond vector K t Defined as

[0092] Qt =K t =T o

[0093] Where T o It is a scaled text feature.

[0094] Query vector Q of visual modality v and bond vector K v Defined as

[0095] Q v =K v =V o ,

[0096] Where V o These are scaled text features. Then, the text attention matrix α and the visual attention matrix β are defined as follows:

[0097]

[0098]

[0099] To adjust the weight of each word through the interaction between text and visual modalities, this invention sums and weights the text attention matrix α and the visual attention matrix β, and the weighted fused attention matrix γ is calculated as follows:

[0100] γ=w t *α+w v *β+b

[0101] Among them, w t and w v represents the weights of the text and visual modalities, respectively, and b is the bias value.

[0102] Then, the multimodal attention matrix γ is defined. m for:

[0103] γ m =σ(γ)

[0104] σ is the softmax function.

[0105] After obtaining the multimodal attention matrix γ m Then, the present invention will T o With the multimodal attention matrix γ m Multiplying the values ​​yields the output of multimodal attention:

[0106] X att =γ m T o

[0107] Among them, T o This is the output of the text feature extractor.

[0108] This invention uses a cross-attention mechanism to fuse textual and visual features, and dynamically adjusts word weights by combining information from text and image modalities. Compared to other multimodal methods that directly concatenate text and image feature vectors, the cross-attention mechanism multimodal fusion method better preserves the potential connections between multimodal fusion information.

[0109] Based on the above embodiments, step 400 includes:

[0110] The multimodal fusion features are input into the weight matrix of the fully connected layer, summed with the bias values, and then processed by the softmax logistic regression function to obtain the rumor prediction probability.

[0111] Based on the rumor prediction probability, the rumor event detection result is determined.

[0112] Specifically, such as Figure 4 As shown, the multimodal fusion features are sequentially input into the fully connected layer and softmax logistic regression function of the rumor classification module to obtain the rumor prediction category, i.e. whether it is a rumor.

[0113] The rumor classification module uses multimodal fusion features as input and classifies event detection source data into rumors or non-rumors through a fully connected layer and softmax.

[0114]

[0115] Where w is the weight matrix of the fully connected layer, b is the bias value, and s is the softmax function used in the rumor classification module. To predict probabilities.

[0116] Based on the predicted probability and combined with the set probability experience value, determine whether the event detection source data is a rumor.

[0117] The multimodal rumor detection method proposed in this invention, which employs a training and cross-attention mechanism, exhibits strong generalization ability. Compared to traditional models, this method achieves better results on both Chinese and English datasets.

[0118] The multimodal rumor detection system provided by this invention will be described below. The multimodal rumor detection system described below can be referred to in correspondence with the multimodal rumor detection method described above.

[0119] Figure 5 This is a schematic diagram of the structure of the multimodal rumor detection system provided by the present invention, as shown below. Figure 5 As shown, it includes: an acquisition module 51, an extraction module 52, a fusion module 53, and a detection module 54, wherein:

[0120] The acquisition module 51 is used to acquire event detection source data; the extraction module 52 is used to extract the text features and visual features of the event detection source data using a pre-trained transformer model; the fusion module 53 is used to perform multimodal fusion of the text features and the visual features based on a cross-attention mechanism to obtain multimodal fusion features; and the detection module 54 is used to perform rumor detection on the multimodal fusion features to obtain rumor event detection results.

[0121] This invention introduces a pre-trained transformer model to extract textual and visual features separately, and then performs multimodal fusion for detection and classification. Compared with existing methods, it makes better use of information from multimodal posts on social media, resulting in better detection performance and stronger robustness.

[0122] Figure 6 Example: A schematic diagram of the physical structure of an electronic device, such as... Figure 6 The electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a multimodal rumor detection method. This method includes: acquiring event detection source data; extracting textual and visual features from the event detection source data using a pre-trained transformer model; fusing the textual and visual features multimodally based on a cross-attention mechanism to obtain multimodal fused features; and performing rumor detection on the multimodal fused features to obtain a rumor event detection result.

[0123] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal rumor detection method provided by the above methods. The method includes: acquiring event detection source data; extracting text features and visual features of the event detection source data using a pre-trained transformer model; fusing the text features and the visual features in a multimodal manner based on a cross-attention mechanism to obtain multimodal fused features; and performing rumor detection on the multimodal fused features to obtain a rumor event detection result.

[0125] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal rumor detection method provided by the above methods. The method includes: acquiring event detection source data; extracting textual and visual features of the event detection source data using a pre-trained transformer model; performing multimodal fusion of the textual and visual features based on a cross-attention mechanism to obtain multimodal fused features; and performing rumor detection on the multimodal fused features to obtain a rumor event detection result.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal rumor detection method, characterized in that, include: Obtain event detection source data; A pre-trained transformer model is used to extract textual and visual features from the event detection source data. The textual features include textual semantic features and textual word features. The text semantic features are extracted from text information based on the Text-CNN text classification convolutional neural network model, and the text word features are extracted from text information based on the pre-trained bidirectional encoder-representation converter BERT. Based on the text features, text query vectors and text key vectors are determined. A text word attention matrix is ​​obtained by multiplying the text query vectors and text key vectors. Based on the visual features, visual query vectors and visual key vectors are determined. A visual attention matrix is ​​obtained by multiplying the visual query vectors and visual key vectors. Text modality weights, visual modality weights, and bias values ​​are determined. A weighted sum of the text word attention matrix, text modality weights, visual attention matrix, visual modality weights, and bias values ​​is obtained and processed by a softmax logistic regression function to obtain a multimodal attention matrix. The multimodal attention matrix is ​​then multiplied by the text word features and concatenated with the text semantic features to obtain a multimodal fusion feature. The multimodal fusion features are used for rumor detection to obtain rumor event detection results.

2. The multimodal rumor detection method according to claim 1, characterized in that, After acquiring the event detection source data, the process also includes: The text data in the event detection source data is preprocessed to obtain preprocessed text data. The text preprocessing includes filtering special symbols and word segmentation. Image preprocessing is performed on the image data in the event detection source data to obtain preprocessed image data. The image preprocessing includes image format conversion, filtering invalid data, and unifying image size.

3. The multimodal rumor detection method according to claim 2, characterized in that, The text features of the event detection source data are extracted using a pre-trained transformer model, including: The preprocessed text data is converted into word vectors, wherein the word vectors include any text word and word vector dimensions; Based on any given text word and the word vector dimension, a text input matrix is ​​obtained; The text input matrix is ​​fed into the Text-CNN text classification convolutional neural network model to obtain text semantic features; The text input matrix is ​​input into a pre-trained bidirectional encoder-decoder (BERT) to obtain text word features.

4. The multimodal rumor detection method according to claim 2, characterized in that, Visual features of the event detection source data are extracted using a pre-trained transformer model, including: The preprocessed image data is input into the pre-trained visual converter ViT to obtain visual features.

5. The multimodal rumor detection method according to claim 1, characterized in that, The process of performing rumor detection on the multimodal fusion features to obtain rumor event detection results includes: The multimodal fusion features are input into the weight matrix of the fully connected layer, summed with the bias values, and then processed by the softmax logistic regression function to obtain the rumor prediction probability. Based on the rumor prediction probability, the rumor event detection result is determined.

6. A multimodal rumor detection system, characterized in that, include: The acquisition module is used to acquire event detection source data; The extraction module uses a pre-trained transformer model to extract textual and visual features from the event detection source data. The textual features include textual semantic features and textual word features. The text semantic features are extracted from text information based on the Text-CNN text classification convolutional neural network model, and the text word features are extracted from text information based on the pre-trained bidirectional encoder-representation converter BERT. The fusion module is used to determine text query vectors and text key vectors based on the text features, and to obtain a text word attention matrix by performing matrix multiplication of the text query vectors and the text key vectors; to determine visual query vectors and visual key vectors based on the visual features, and to obtain a visual attention matrix by performing matrix multiplication of the visual query vectors and the visual key vectors; to determine text modality weights, visual modality weights, and bias values, and to perform a weighted summation of the text word attention matrix, the text modality weights, the visual attention matrix, the visual modality weights, and the bias values, and to process this summation using a softmax logistic regression function to obtain a multimodal attention matrix; to perform matrix multiplication of the multimodal attention matrix with the text word features, and to perform matrix concatenation with the text semantic features to obtain a multimodal fusion feature; The detection module is used to perform rumor detection on the multimodal fusion features and obtain the rumor event detection results.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal rumor detection method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal rumor detection method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal rumor detection method as described in any one of claims 1 to 5.

Images