Rumor detection method based on multi-modal data conversion network

Abstract

The invention discloses a rumor detection method based on a multi-modal data conversion network, and the method comprises the following steps: obtaining to-be-detected social media multi-modal data, carrying out the preprocessing, obtaining a descriptive text and a visual potential representation through a visual encoder-decoder, obtaining a text potential representation, a text conversion feature and a visual conversion feature by using a BERT module; inputting the text conversion features and the visual conversion features into a common attention module to obtain fusion features; calculating the multi-granularity similarity of the text potential representation and the visual potential representation; and inputting the fusion features into the model to obtain a final result. According to the method, a visual encoder-decoder and a common attention module are introduced, so that cross-modal conversion from an image to a text can be realized to solve the problem of image-text isomerism, bidirectional interaction influence between the text and visual features is fully considered, and associated features of the text and visual information in multi-modal data can be captured more comprehensively; and the rumor detection accuracy and robustness are obviously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a rumor detection method based on multi-modal data conversion network. BACKGROUND

[0002] The rapid development of multimedia has completely changed the way people obtain news, and some social media have promoted various false information, including distorted or even fake multimedia content, to mislead readers. Widely spread rumors can have serious adverse effects. Since a large number of users generate content and spread it quickly on social media every day, an intelligent rumor detection method can minimize harmful effects.

[0003] Traditional rumor detection methods mainly rely on significant features in text data for classification, some use traditional machine learning models such as decision trees, some use neural network-based models such as recurrent neural networks and convolutional neural networks, and in addition, there are knowledge graphs and the like. With the progress of multimedia technology and information dissemination technology, visual content has become an important part of rumors that attract and mislead consumers due to its more credible narrative and faster dissemination. Therefore, rumor detection methods are undergoing a shift from single modality to multi-modal paradigm. However, current multi-modal rumor detection methods tend to abstract data for different modalities through their respective feature extraction models, i.e. converting data into feature vectors using neural networks and other methods, and then performing feature fusion. Although this method can improve the accuracy of the model, it ignores the differences between heterogeneous data and different model feature extraction modes. SUMMARY

[0004] To solve the above technical problems, the present application provides a rumor detection method based on multi-modal data conversion network with simple algorithm and high detection accuracy.

[0005] The technical solution of the present application to solve the above technical problems is: a rumor detection method based on multi-modal data conversion network, comprising the following steps:

[0006] Step S1, text and visual feature extraction: obtaining social media multi-modal data to be detected and preprocessing, obtaining descriptive text and visual latent representation through a visual encoder-decoder, and obtaining text latent representation, text conversion feature and visual conversion feature using a BERT module;

[0007] Step S2, inputting the text conversion feature and the visual conversion feature into a co-attention module to obtain a fusion feature; the specific process is:

[0008] Step S21, linearly transforming the text conversion feature and the visual conversion feature;

[0009] Step S22, the text conversion feature and the visual conversion feature are enhanced by using a common attention module;

[0010] Step S23, the attention features are spliced and fused to obtain preliminary fusion features;

[0011] Step S24, the preliminary fusion features are input into a feedforward neural network FFN to output final fusion features;

[0012] Step S3, a multi-granularity similarity of the text latent representation and the visual latent representation is calculated to guide the model training;

[0013] Step S4, the fusion features are input into the model to obtain a final result.

[0014] The rumor detection method based on the multi-modal data conversion network, in the step S1, the multi-modal data includes text data T, image data I and a true label , wherein =1 indicates that the multi-modal data is a rumor, =0 indicates that the multi-modal data is not a rumor.

[0015] The rumor detection method based on the multi-modal data conversion network, in the step S1, the preprocessing includes: preprocessing the text data T, removing special symbols, emoticons and filling or truncating to a maximum token length of 512; the image data I is preprocessed, scaled to a resolution of 224x224, and the pixel value is normalized to the interval [0, 1].

[0016] The rumor detection method based on the multi-modal data conversion network, in the step S1, the process of obtaining the descriptive text and the visual latent representation by the visual encoder-decoder is as follows:

[0017] The visual encoder-decoder based on the Transformer architecture is used to process the image data I, and the descriptive text and the visual latent representation matching the image content are output, and the conversion relationship is as follows:

[0018] ;

[0019] , wherein is the image matching descriptive text information, Encoder is the visual Transformer encoder, and Decoder is the text Transformer decoder;

[0020] The preprocessed image data I is fed into two parallel branches simultaneously using an Encoder, and the core features are input into a Decoder to obtain descriptive text. The image patch sequence is formed by dividing the preprocessed image data I into non-overlapping image patches of fixed pixel size. The image patches are converted into fixed-dimensional vectors through an embedding layer and then arranged in their original spatial positions to form an image patch sequence. A learnable classification label CLS Token with a dimension of 768 is added before the image patch sequence to aggregate global features. After encoding, the features of the classification label CLS Token are directly taken as the visual latent representation.

[0021] The rumor detection method based on multimodal data transformation networks described above, in step S1, involves obtaining the text latent representation, text transformation features, and visual transformation features using the BERT module as follows:

[0022] The preprocessed text data T is input into the pre-trained BERT model to extract text transformation features with a dimension of 768. and the text latent representation t1 corresponding to the classification label output by the BERT model; Inputting the data into the BERT model yields visual transformation features with a dimension of 768. .

[0023] The rumor detection method based on the multimodal data conversion network described above, specifically step S2, is as follows:

[0024] Step S21, Linear Transformation: Transform the text features Perform a linear transformation to generate a text query vector. Text key vector Text value vector The formula is as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] in, , , These are learnable weight matrices for the query, key, and value, respectively. , , These are the learnable biases for the query, key, and value, respectively, with the superscript 'i' indicating the attention head index.

[0029] Visual transformation features Perform the same linear transformation to generate a visual query vector. Visual key vector Visual value vector The formula is as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] in, , , These are the learnable weight matrices for queries, keys, and values, respectively. , , These are the learnable biases for queries, keys, and values, respectively.

[0034] Step S22: Use a shared attention module to mutually enhance the text transformation features and visual transformation features:

[0035] The formula for attention output from text to vision is as follows:

[0036] ;

[0037] The formula for visual-to-text attention output is as follows:

[0038] ;

[0039] in, For text-to-visual attention, For visual-to-text attention, the superscript 'i' indicates the attention head index. For Transpose for transpose, It is the scaling factor, and softmax is the normalization function;

[0040] Step S23, Attention Feature Concatenation and Fusion: Concatenate the text-to-visual attention with the visual-to-text attention to obtain preliminary fused features. ;

[0041] ;

[0042] Here, concat is the feature concatenation operation. The weight matrix is ​​a learnable matrix;

[0043] Step S24, Feature Optimization: Initially fuse features Input to a feedforward neural network (FFN), output the final fused features. The formula is as follows:

[0044] ;

[0045] FFN stands for Feedforward Neural Network.

[0046] The rumor detection method based on multimodal data conversion networks described above, specifically step S3, is as follows:

[0047] Step S31, High-dimensional projection: Latent representation of text Projection The formula is as follows:

[0048] ;

[0049] in, It is the ReLU activation function. , For learnable parameters, yes High-dimensional features obtained through projection;

[0050] Projecting the visual latent representation yields , It is a high-dimensional feature obtained by projecting the visual latent representation;

[0051] Step S32, perform multi-granularity similarity measurement: multi-granularity similarity measurement is divided into global similarity and local similarity; global similarity uses cosine similarity to calculate the similarity between the overall text latent representation and visual latent representation; local similarity divides the text and image into n parts, calculates them separately and takes the average value;

[0052] Step S33: Weighted fusion of global similarity and local similarity to obtain cosine similarity S;

[0053] Step S34: Calculate the binary cross-entropy loss between S and the true label, as shown in the following formula:

[0054] ;

[0055] in, Let S be the binary cross-entropy loss between the true label and the S.

[0056] In the aforementioned rumor detection method based on a multimodal data conversion network, step S32 involves global similarity. The calculation formula is as follows:

[0057] ;

[0058] Local similarity :Will , Divide the data into n segments and calculate the average value using the following formula:

[0059] ;

[0060] in, for The m-th local feature vector after segmentation for The m-th local feature vector after segmentation It is the Euclidean norm.

[0061] In the aforementioned rumor detection method based on a multimodal data conversion network, the formula for calculating the cosine similarity S in step S33 is as follows:

[0062] ;

[0063] in, It is the weight of local similarity.

[0064] The rumor detection method based on the multimodal data conversion network described above, specifically step S4, is as follows:

[0065] Step S41: Merge the final features The input is fed into a fully connected layer, which transforms the high-dimensional fusion features into probability values ​​between [0,1] to predict whether it is a rumor. The formula is as follows:

[0066] ;

[0067] Where FC() is a fully connected layer, and Sigmoid is... function, For predicting labels;

[0068] Step S42: Calculate the binary cross-entropy loss between the predicted label and the true label. Quantitative prediction results and true labels The difference between them is expressed by the following formula:

[0069] ;

[0070] Step S43: The final optimization objective is the weighted sum of the losses, as shown in the following formula:

[0071] ;

[0072] in, It is the similarity loss weight. It is a joint loss function, which guides the updating of model parameters through joint optimization.

[0073] The beneficial effects of this invention are as follows:

[0074] 1. This invention introduces a visual encoder-decoder and a co-attention module, which not only enables cross-modal conversion from image to text to solve the problem of image-text heterogeneity, but also fully considers the bidirectional interaction between text and visual features. Compared with traditional single-modal detection or simple modal concatenation methods, this invention can more comprehensively capture the correlation features between text and visual information in multimodal data, significantly improving the accuracy and robustness of rumor detection.

[0075] 2. This invention, through the co-attention fusion of text conversion features and visual conversion features and multi-granularity similarity aggregation, can establish richer cross-modal semantic associations between text and images, and is particularly suitable for rumor detection tasks in multimodal data scenarios such as social media, news dissemination, and e-commerce reviews.

[0076] 3. By jointly optimizing classification loss and multi-granularity similarity loss, this invention can ensure the objectivity and accuracy of detection results. The joint loss calculation not only considers the classification tendency of text or visual single modality, but also integrates the matching degree distribution between image and text modality, making rumor assessment more robust, especially in noisy scenarios where image and text do not match, the detection results are more reliable. Attached Figure Description

[0077] Figure 1 This is the overall flowchart of the present invention.

[0078] Figure 2 This is a comparison chart showing the accuracy of different values ​​for multi-granularity similarity in this invention. Detailed Implementation

[0079] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0080] like Figure 1 As shown, a rumor detection method based on a multimodal data conversion network includes the following steps:

[0081] Step S1, Text and Visual Feature Extraction: Obtain the multimodal data of the social media to be detected and preprocess it. Obtain descriptive text and visual latent representation through visual encoder-decoder. Use the BERT module to obtain text latent representation, text transformation features and visual transformation features.

[0082] Multimodal data includes text data T, image data I, and real labels. ,in =1 indicates that the multimodal data is a rumor. =0 indicates that the multimodal data is not a rumor.

[0083] Preprocessing includes: preprocessing text data T by removing special symbols and emojis and padding or truncating to the maximum token length of 512; and preprocessing image data I by scaling to a 224×224 resolution and normalizing pixel values ​​to the [0,1] range.

[0084] The process of obtaining descriptive text and visual latent representations through a visual encoder-decoder is as follows:

[0085] A visual encoder-decoder based on the Transformer architecture is used to process image data I, outputting descriptive text and visual latent representations that match the image content. The transformation relationship is as follows:

[0086] ;

[0087] in, To convert image matching into descriptive text information, the Encoder is a visual Transformer encoder used to extract pixel-level features from the image; the Decoder is a text Transformer decoder used to convert image features into natural language descriptions.

[0088] The preprocessed image data I is fed into two parallel branches simultaneously using an Encoder, and the core features are input into a Decoder to obtain descriptive text. The image patch sequence is formed by dividing the preprocessed image data I into non-overlapping image patches of fixed pixel size. The image patches are converted into fixed-dimensional vectors through an embedding layer and then arranged in their original spatial positions to form an image patch sequence. A learnable classification label CLS Token with a dimension of 768 is added before the image patch sequence to aggregate global features. After encoding, the features of the classification label CLS Token are directly taken as the visual latent representation.

[0089] The process of obtaining text latent representation, text transformation features, and visual transformation features using the BERT module is as follows:

[0090] The preprocessed text data T is input into the pre-trained BERT model to extract text transformation features with a dimension of 768. and the text latent representation t1 corresponding to the classification label output by the BERT model; Inputting the data into the BERT model yields visual transformation features with a dimension of 768. .

[0091] Step S2: Input the text conversion features and visual conversion features into the co-attention module to obtain the fused features.

[0092] The specific process of step S2 is as follows:

[0093] Step S21, Linear Transformation: Transform the text features Perform a linear transformation to generate a text query vector. Text key vector Text value vector The formula is as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] in, , , These are learnable weight matrices for the query, key, and value, respectively. , , These are the learnable biases for the query, key, and value, respectively, with the superscript 'i' indicating the attention head index.

[0098] Visual transformation features Perform the same linear transformation to generate a visual query vector. Visual key vector Visual value vector The formula is as follows:

[0099] ;

[0100] ;

[0101] ;

[0102] in, , , These are the learnable weight matrices for queries, keys, and values, respectively. , , These are the learnable biases for queries, keys, and values, respectively.

[0103] Step S22: Use a shared attention module to mutually enhance the text transformation features and visual transformation features:

[0104] The formula for attention output from text to vision is as follows:

[0105] ;

[0106] The formula for visual-to-text attention output is as follows:

[0107] ;

[0108] in, For text-to-visual attention, For visual-to-text attention, the superscript 'i' indicates the attention head index. For Transpose for transpose, It is a scaling factor, and softmax is a normalization function that makes the sum of the attention weights of each token equal to 1, highlighting the contribution of highly correlated features;

[0109] Step S23, Attention Feature Concatenation and Fusion: Concatenate the text-to-visual attention with the visual-to-text attention to obtain preliminary fused features. ;

[0110] ;

[0111] Here, concat is the feature concatenation operation. The weight matrix is ​​learnable; by integrating bidirectional interactive information between text and vision and vision and text, fusion features are obtained, providing cross-modal association evidence for subsequent rumor classification.

[0112] Step S24, Feature Optimization: Initially fuse features Input to a feedforward neural network (FFN), output the final fused features. The formula is as follows:

[0113] ;

[0114] FFN stands for Feedforward Neural Network.

[0115] Step S3: Calculate the multi-granularity similarity between the text latent representation and the visual latent representation to guide model training and reduce the impact of image-text mismatch.

[0116] Noise in the data can cause fused features to point in the wrong direction, leading to negative optimization of the model; this loss can be obtained by using a multi-granularity similarity metric and optimizing the model parameters.

[0117] The specific process of step S3 is as follows:

[0118] Step S31, High-dimensional projection: Latent representation of text Projection The formula is as follows:

[0119] ;

[0120] in, It is the ReLU activation function. , For learnable parameters, yes High-dimensional features obtained through projection;

[0121] Projecting the visual latent representation yields , It is a high-dimensional feature obtained by projecting the visual latent representation;

[0122] Step S32, perform multi-granularity similarity measurement: multi-granularity similarity measurement is divided into global similarity and local similarity; global similarity uses cosine similarity to calculate the similarity between the overall text latent representation and visual latent representation; local similarity divides the text and image into n parts, calculates them separately and takes the average value;

[0123] Global similarity The calculation formula is as follows:

[0124] ;

[0125] Local similarity :Will , Divide the data into n segments and calculate the average value using the following formula:

[0126] ;

[0127] in, for The m-th local feature vector after segmentation for The m-th local feature vector after segmentation Using the Euclidean norm, the feature vectors are standardized to eliminate the magnitude difference in high-dimensional features between images and text.

[0128] Step S33: Weighted fusion of global similarity and local similarity to obtain cosine similarity S;

[0129] The formula for calculating the cosine similarity S is:

[0130] ;

[0131] in, It is the weight of local similarity; in order to let similarity guide model training, the cosine similarity S is mapped to between 0 and 1 through the sigmoid function;

[0132] Step S34: Calculate the binary cross-entropy loss between S and the true label, as shown in the following formula:

[0133] ;

[0134] in, Let S be the binary cross-entropy loss between the true label and the S.

[0135] Step S4: Input the fused features into the model to obtain the final result.

[0136] The specific process of step S4 is as follows:

[0137] Step S41: Merge the final features The input is fed into a fully connected layer, which transforms the high-dimensional fusion features into probability values ​​between [0,1] to predict whether it is a rumor. The formula is as follows:

[0138] ;

[0139] In this context, FC() is a fully connected layer that performs dimensionality mapping, ultimately fusing high-dimensional features. Compressed into low-dimensional original predictions; Sigmoid is The function compresses the original predicted values ​​output by the fully connected layer to the [0,1] interval to obtain the final prediction result; For predicting labels;

[0140] Step S42: Calculate the binary cross-entropy loss between the predicted label and the true label. Quantitative prediction results and true labels The difference between them is expressed by the following formula:

[0141] ;

[0142] That is, the training results can Minimize the size to ensure the predicted results closely approximate the true labels;

[0143] Step S43: The final optimization objective is the weighted sum of the losses, as shown in the following formula:

[0144] ;

[0145] in, It is the similarity loss weight. It is a joint loss function that guides model parameter updates through joint optimization, thereby improving the accuracy, robustness, and noise resistance of rumor detection.

[0146] The following is the experimental setup and results analysis of this invention:

[0147] To effectively evaluate model performance, this invention selected three public rumor datasets: PHEME, Weibo, and Weibo21. The data for PHEME comes from the Twitter platform, while the data for Weibo and Weibo21 both come from the Weibo platform. Table 1 shows the dataset information.

[0148]

[0149] Among them, Non-rumors represents the number of non-rumor samples, Rumors represents the number of rumor samples, Images represents the number of images, Users represents the number of users, and Comments represents the number of comments. The Weibo Rumor Dataset contains 4,534 comments, including not only the Weibo posts containing rumors and non-rumor content, but also information such as reposts, likes, and timestamps related to these posts. The English Rumor Dataset PHEME contains 5 news events and 7,388 comments. In addition to the Weibo posts containing rumors and non-rumor content, it also includes information such as reposts, likes, and timestamps related to these posts. The Weibo21 Rumor Dataset collects rumor data based on nine different fields, including science, military, education, and disaster, and contains 9,128 comments.

[0150] In this experiment, the three datasets were divided into training, validation, and test sets in a 7:1:2 ratio. The evaluation metrics used were the four common metrics: Accuracy, Precision, Recall, and F1 Score. The Modality Data Transformation Network (MDCN) rumor detection method is abbreviated as MDCN.

[0151] The comparative experimental model is as follows:

[0152] MVAE: Multimodal variational autoencoder (MVAE) is an extension of variational autoencoder in multimodal scenarios. It maps different modal features such as text and images to a unified latent feature space through probabilistic modeling, while taking into account the independence of single-modal features and the correlation of cross-modal features, so as to achieve effective extraction and fusion of multimodal features.

[0153] EANN: The multimodal fake news detection model based on generative adversarial networks (EANN) relies on the adversarial training mechanism of GAN. It integrates heterogeneous information through a multimodal feature extractor, mines the consistency of event correlations between modalities through an event discriminator, and completes the final truth and falsehood judgment through a fake news detector. The three work together to achieve accurate identification of multimodal fake news.

[0154] BERT: The attention-based pre-trained language model (BERT) uses a bidirectional Transformer as its basic architecture. Through the pre-training and fine-tuning paradigm, it completes masked language modeling and next sentence prediction tasks in massive texts. It can capture the bidirectional contextual semantics of words, and the extracted word vectors have stronger contextual adaptability and general semantic expression capabilities.

[0155] MFAN: Multimodal Feature Alignment Fusion Network (MFAN) focuses on the perspective differences of multimodal source posts. It uses self-supervised loss to constrain the alignment of modal representations under different perspectives, reduce the misalignment and redundancy of cross-modal features, and make the fused multimodal features more in line with the real semantics, thereby improving the feature utilization efficiency of downstream multimodal tasks.

[0156] MRML: Metric and Contrastive Learning Model for Multimodal Rumor Detection (MRML) integrates the core ideas of metric learning and contrastive learning. Triple learning strengthens the rumor discrimination features within a single modality by constructing rumor and non-rumor samples, while cross-modal contrastive learning explores the rumor association patterns between different modalities, achieving dual optimization of features within and between modalities.

[0157] ELKP: The Knowledge-Enhanced Exploratory Enhancement Framework (ELKP) integrates domain expertise into the model training process. Through a knowledge-enhanced enhancement learning mechanism, it explicitly models key entities and relationships in text, compensating for the insufficient understanding of domain knowledge in traditional models. This results in extracted semantic representations that are more relevant to the domain context and contain richer entity and semantic information. These models represent the latest advancements in rumor detection methods. The results of comparative experiments are shown in Table 2.

[0158]

[0159] The experiments shown in the table demonstrate that the present invention (MDCN) outperforms other baseline models in terms of accuracy, recall, and F1 score across the three datasets. The significant performance gap between EANN and MVAE on the Chinese dataset indicates that variational autoencoders perform differently across different languages. BERT significantly outperforms MVAE, demonstrating the excellent generalization ability of pre-trained language models and highlighting the impact of data volume on rumor detection. Furthermore, MFAN outperforms previous models, indicating that considering propagation structure and modality alignment is beneficial for rumor detection. ELKP outperforms MFAN, suggesting that domain knowledge and modular structure contribute to improved generalization performance. MDCN outperforms ELKP; analysis of the training content reveals that the model used in the visual transformation module underwent text-image pair generation training on other datasets, which enhances the effectiveness of feature fusion and improves the model's detection capability.

[0160] To verify the effectiveness of different modules in MDCN, the ablation experiment of this method is as follows:

[0161] w / o VC: Removes visual transformation features from MDCN;

[0162] w / o CA: Based on MDCN, the co-attention module is removed and replaced by a simple concatenation of text features and visual features;

[0163] w / o Sim: Based on MDCN, remove the multi-granularity similarity measure, that is, directly make predictions after fusing textual and visual features.

[0164] The results of the ablation experiment are shown in Table 3:

[0165]

[0166] The data in the table shows that the descriptive text obtained through the visual encoding and decoding model provides additional and valuable textual data for the model, which helps in its optimization. Compared to simply evaluating textual and visual features together, the co-attention module utilizes an attention mechanism to enable textual and visual features to interact during the optimization process. By capturing the potential connections between sentences, it improves the fusion effect of multimodal features. The multi-granularity similarity module demonstrates the importance of global and local correlations between different modal features.

[0167] Figure 2 This demonstrates the impact of the chosen granularity on prediction accuracy when using multimodal features for similarity measurement. `n` represents the number of granularity parts for feature partitioning, used for calculating local similarity. Accuracy reaches its peak and stabilizes after approximately 30 training iterations, with the curve for `n=3` showing significantly higher accuracy than other curves after stabilization. Considering the length and structure of the original text and the visually transformed text, such as "a dog sitting on top of a surfboard in the ocean," this type of text is relatively short and often contains around three objects. Therefore, dividing the features into three parts yields better results.

[0168] This invention, by introducing a visual encoder-decoder and a co-attention module, not only enables cross-modal conversion from image to text to solve the problem of image-text heterogeneity, but also fully considers the bidirectional interaction between text and visual features and the association weights of different tokens. It can comprehensively capture the complex association features between text and visual information in multimodal data, significantly improving the accuracy of rumor detection.

Claims

1. A rumor detection method based on a multimodal data conversion network, characterized in that, Includes the following steps: Step S1, Text and Visual Feature Extraction: Obtain the multimodal data of the social media to be detected and preprocess it. Obtain descriptive text and visual latent representation through visual encoder-decoder. Use the BERT module to obtain text latent representation, text transformation features and visual transformation features. Step S2: Input the text conversion features and visual conversion features into the co-attention module to obtain the fused features; The specific process is as follows: Step S21: Perform linear transformation on the text conversion features and visual conversion features; Step S22: Use a common attention module to mutually enhance the text transformation features and visual transformation features; Step S23: Attention features are spliced ​​and fused to obtain preliminary fused features; Step S24: Input the preliminary fused features into the feedforward neural network FFN and output the final fused features; Step S3: Calculate the multi-granularity similarity between the text latent representation and the visual latent representation to guide model training; Step S4: Input the fused features into the model to obtain the final result.

2. The rumor detection method based on a multimodal data conversion network according to claim 1, characterized in that, In step S1, the multimodal data includes text data T, image data I, and real labels. ,in =1 indicates that the multimodal data is a rumor. =0 indicates that the multimodal data is not a rumor.

3. The rumor detection method based on a multimodal data conversion network according to claim 2, characterized in that, In step S1, the preprocessing includes: preprocessing the text data T by removing special symbols and emoticons and filling or truncating it to the maximum token length of 512; and preprocessing the image data I by scaling it to a resolution of 224×224 and normalizing the pixel values ​​to the [0,1] range.

4. The rumor detection method based on a multimodal data conversion network according to claim 3, characterized in that, In step S1, the process of obtaining descriptive text and visual latent representation through visual encoder-decoder is as follows: A visual encoder-decoder based on the Transformer architecture is used to process image data I, outputting descriptive text and visual latent representations that match the image content. The transformation relationship is as follows: ； in, To convert image matching into descriptive text information, the Encoder is a visual Transformer encoder, and the Decoder is a text Transformer decoder. The preprocessed image data I is fed into two parallel branches simultaneously using an Encoder, and the core features are input into a Decoder to obtain descriptive text. The image patch sequence is formed by dividing the preprocessed image data I into non-overlapping image patches of fixed pixel size. The image patches are converted into fixed-dimensional vectors through an embedding layer and then arranged in their original spatial positions to form an image patch sequence. A learnable classification label CLS Token with a dimension of 768 is added before the image patch sequence to aggregate global features. After encoding, the features of the classification label CLS Token are directly taken as the visual latent representation.

5. The rumor detection method based on a multimodal data conversion network according to claim 4, characterized in that, In step S1, the process of obtaining the text latent representation, text transformation features, and visual transformation features using the BERT module is as follows: The preprocessed text data T is input into the pre-trained BERT model to extract text transformation features with a dimension of 768. and the text latent representation t1 corresponding to the classification label output by the BERT model; Inputting the data into the BERT model yields visual transformation features with a dimension of 768. .

6. The rumor detection method based on a multimodal data conversion network according to claim 5, characterized in that, The specific process of step S2 is as follows: Step S21, Linear Transformation: Transform the text features Perform a linear transformation to generate a text query vector. Text key vector Text value vector The formula is as follows: ； ； ； in, , , These are learnable weight matrices for the query, key, and value, respectively. , , These are the learnable biases for the query, key, and value, respectively, with the superscript 'i' indicating the attention head index. Visual transformation features Perform the same linear transformation to generate a visual query vector. Visual key vector Visual value vector The formula is as follows: ； ； ； in, , , These are the learnable weight matrices for queries, keys, and values, respectively. , , These are the learnable biases for queries, keys, and values, respectively. Step S22: Use a shared attention module to mutually enhance the text transformation features and visual transformation features: The formula for attention output from text to vision is as follows: ； The formula for visual-to-text attention output is as follows: ； in, For text-to-visual attention, For visual-to-text attention, the superscript 'i' indicates the attention head index. For Transpose for transpose, It is the scaling factor, and softmax is the normalization function; Step S23, Attention Feature Concatenation and Fusion: Concatenate the text-to-visual attention with the visual-to-text attention to obtain preliminary fused features. ; ； Here, concat is the feature concatenation operation. The weight matrix is ​​a learnable matrix; Step S24, Feature Optimization: Initially fuse features Input to a feedforward neural network (FFN), output the final fused features. The formula is as follows: ； FFN stands for Feedforward Neural Network.

7. The rumor detection method based on a multimodal data conversion network according to claim 6, characterized in that, The specific process of step S3 is as follows: Step S31, High-dimensional projection: Latent representation of text Projection The formula is as follows: ； in, It is the ReLU activation function. , For learnable parameters, yes High-dimensional features obtained through projection; Projecting the visual latent representation yields , It is a high-dimensional feature obtained by projecting the visual latent representation; Step S32, perform multi-granularity similarity measurement: multi-granularity similarity measurement is divided into global similarity and local similarity; global similarity uses cosine similarity to calculate the similarity between the overall text latent representation and visual latent representation; local similarity divides the text and image into n parts, calculates them separately and takes the average value; Step S33: Weighted fusion of global similarity and local similarity to obtain cosine similarity S; Step S34: Calculate the binary cross-entropy loss between S and the true label, as shown in the following formula: ； in, Let S be the binary cross-entropy loss between the true label and the S.

8. The rumor detection method based on a multimodal data conversion network according to claim 7, characterized in that, In step S32, global similarity The calculation formula is as follows: ； Local similarity :Will , Divide the data into n segments and calculate the average value using the following formula: ； in, for The m-th local feature vector after segmentation for The m-th local feature vector after segmentation It is the Euclidean norm.

9. The rumor detection method based on a multimodal data conversion network according to claim 8, characterized in that, In step S33, the formula for calculating the cosine similarity S is: ； in, It is the weight of local similarity.

10. The rumor detection method based on a multimodal data conversion network according to claim 7, characterized in that, The specific process of step S4 is as follows: Step S41: Merge the final features The input is fed into a fully connected layer, which transforms the high-dimensional fusion features into probability values ​​between [0,1] to predict whether it is a rumor. The formula is as follows: ； Where FC() is a fully connected layer, and Sigmoid is... function, For predicting labels; Step S42: Calculate the binary cross-entropy loss between the predicted label and the true label. Quantitative prediction results and true labels The difference between them is expressed by the following formula: ； Step S43: The final optimization objective is the weighted sum of the losses, as shown in the following formula: ； in, It is the similarity loss weight. It is a joint loss function, which guides the updating of model parameters through joint optimization.

Images