Method for detecting fake news video based on multi-modal large model self-driving question and answer framework

Abstract

This invention proposes a fake news video detection method based on a multimodal large-scale model self-driven question-answering framework, belonging to the field of fake news video detection technology. The method includes the following steps: multimodal feature extraction and data structure initialization of the input video; configuring the roles and prompt templates of the questioner LMM and responder LMM; performing multi-turn dialogues through the questioner LMM and responder LMM to generate high-quality question-answer pairs; calculating cross-modal comprehensive relevance scores; filtering global question-answering knowledge representations based on the comprehensive relevance scores; deep feature extraction of each modality of the video; cross-modal attention fusion for question-answering perception; feature dimension unification and connection; knowledge-enhanced prediction and loss calculation, with final binary classification prediction through a predictor network; model training and parameter updating to obtain a trained model; and using the trained model to detect the input video. This invention can effectively identify complex fake news video content, improving the accuracy and robustness of detection.

Description

Technical Field

[0001] This invention belongs to the field of fake news video detection technology, and in particular relates to a fake news video detection method based on a multimodal large model self-driven question answering framework. Background Technology

[0002] With the booming development of online video sharing platforms such as YouTube and TikTok, the spread of fake news through video is becoming increasingly rampant, seriously endangering political stability and public health. Therefore, developing effective fake news video detection methods is crucial to curbing the spread of misinformation.

[0003] Early methods for detecting fake news videos primarily employed single-modal detection techniques, such as BERT for text analysis, ViT for visual feature extraction, and AST for audio analysis. These methods focus only on a single modality of information, easily overlooking false content in other modalities and exhibiting limited detection accuracy.

[0004] With technological advancements, current methods for detecting fake news videos primarily focus on analyzing the authenticity of multimodal content (text, audio, and visual) and modeling cross-modal correlations. For example, SV-FEND combines video multimodal features with user comments to detect fake news content; FakingRecipe effectively detects manipulation by modeling the content creation process and analyzing material selection and editing patterns across sentiment, semantics, spatial, and temporal dimensions. While these multimodal methods have made significant progress compared to unimodal methods, they still face the following key challenges:

[0005] Challenge 1: Difficulty in detecting subtly manipulated fake news videos. The rapid advancements in video editing technology have significantly lowered the barrier to generating visually realistic but misleading content. Malicious actors can now create fake news videos by constructing deceptive multimodal narratives or subtly manipulating real video clips—for example, editing specific frames or altering contextual audiovisual cues—without disrupting surface-level coherence. These sophisticated forgeries often maintain a high degree of consistency in appearance and modal alignment while embedding deceptive semantics that are difficult to detect. Therefore, detection methods that rely solely on modeling the original multimodal content of the video itself often perform poorly, failing to reveal such deeply embedded inconsistencies or semantic manipulation.

[0006] Challenge 2: Scarcity of Annotated Data for Fake News Video Detection. A key bottleneck in advancing fake news video detection lies in the limited availability of labeled training data. Unlike image or text domains, collecting and annotating video data requires substantial resources, necessitates manual inspection of multimodal content across time spans, and is also subject to legal and copyright constraints. Consequently, current benchmark datasets for fake news video detection typically contain only a few thousand labeled samples (e.g., 1000 videos), insufficient for training robust deep models. This data scarcity severely limits the model's ability to generalize to unseen or emerging fake news patterns, particularly those involving sophisticated cross-modal manipulation or domain transfer.

[0007] Existing methods based on large multimodal models, such as GPT-4o, LLaVA, and Qwen2-VL, while possessing strong zero-shot capabilities, lack task-specific adaptability and often produce inconsistent or unreliable predictions when faced with complex and deceptive real-world content. Furthermore, directly fine-tuning these large models is computationally infeasible and memory-intensive, posing a challenge for practical deployment.

[0008] In summary, current fake news video detection methods suffer from insufficient performance when dealing with complex semantic manipulation due to over-reliance on inherent video features, lack of external knowledge support, and inadequate cross-modal information fusion. Furthermore, they exhibit poor generalization ability in cross-platform scenarios. Summary of the Invention

[0009] The purpose of this invention is to provide a fake news video detection method based on a multimodal large-model self-driven question answering framework. By introducing self-driven divergent reasoning, cross-modal relevance-guided integration, and QA-aware cross-modal fusion mechanisms, it effectively identifies complex fake news video content and improves detection accuracy and robustness. This addresses the technical problems of existing fake news video detection methods, such as insufficient detection capability for complex semantic manipulation, lack of external knowledge enhancement, limited cross-modal information fusion effects, and poor cross-platform generalization ability.

[0010] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0011] A method for detecting fake news videos based on a multimodal large model self-driven question answering framework, the method comprising the following steps:

[0012] Step S1: Preprocess and initialize the input video, i.e., extract multimodal features and initialize the data structure of the input video;

[0013] Step S2: Initialize the self-driven divergent reasoning module, which includes the questioner LMM and the responder LMM. Configure the roles and prompt templates of the questioner LMM and the responder LMM; initialize the dialogue history.

[0014] Step S3: Perform multi-turn dialogue through the questioner LMM and responder LMM of the self-driven divergent reasoning module, and generate high-quality question-answer pairs through the iterative dialogue process;

[0015] Step S4: Calculate the cross-modal comprehensive relevance score to quantify the semantic alignment of each question-answer pair with the video content;

[0016] Step S5: Semantic encoding and weighted aggregation of question-answer pairs, and filtering and fusion of global question-answer knowledge representation based on comprehensive relevance scores;

[0017] Step S6: Multimodal feature encoding, performing independent deep feature extraction for each modality of the video;

[0018] Step S7: Question-answering awareness cross-modal attention fusion to achieve fine-grained semantic alignment between global question-answering knowledge representation and features of each modality;

[0019] Step S8: Feature dimension unification and connection, ensuring that all features are fused in the same dimensional space;

[0020] Step S9: Knowledge-enhanced prediction and loss calculation, using the predictor network to perform the final binary classification prediction;

[0021] Step S10: Model training and parameter update, perform end-to-end supervised learning optimization to obtain the trained model;

[0022] Step S11: Use the trained model to detect the input video.

[0023] Further, step S1 includes the following steps:

[0024] Step S11: For the visual modality, extract keyframes from the input video using a uniform sampling strategy to obtain a sampled video frame sequence;

[0025] Step S12: For audio modalities, extract the complete audio track from the video;

[0026] Step S13: For the text modality, collect the title information, the screen-displayed text, and the speech-transcribed text, and concatenate the collected title information, screen-displayed text, and speech-transcribed text into a complete text input;

[0027] Step S14: Combine the video frame sequence representation, audio track, and text input to initialize the multimodal input and the question-answer pair set.

[0028] Furthermore, step S3 executes a multi-turn dialogue loop, each turn of which includes the following sub-steps:

[0029] Step S31: Question Generation: The questioner LMM generates the question for round t+1 based on the multimodal input and dialogue history;

[0030] Step S32: Termination condition check: If the question generated by the LMM is "No question" or the number of dialogue rounds reaches the maximum number of dialogue rounds M. max If the condition is met, the dialogue process will terminate; otherwise, it will continue.

[0031] Step S33: Answer Generation: The respondent LMM generates the corresponding answer based on the multimodal input and the current question;

[0032] Step S34: Store question-answer pairs: Add the newly generated question-answer pairs to the question-answer pair set;

[0033] Step S35: Update dialogue history: Update the dialogue history and increment the round counter by 1.

[0034] Further, step S4 includes the following steps:

[0035] Step S41: Question-answer pair textification: Convert the question and corresponding answer of each question-answer pair into text through a concatenation operation;

[0036] Step S42: Feature Encoding: Use the pre-trained CLIP model's text encoder to encode the question-and-answer text and the text input to obtain the question-and-answer pair feature encoding and the text feature encoding; use the pre-trained CLIP model's visual encoder to encode the video frame sequence to obtain the video feature encoding;

[0037] Step S43: Calculate the comprehensive relevance score: Calculate the semantic relevance score between the question-answer pair feature code and the video feature code and the semantic relevance score between the question-answer pair feature code and the text feature code using cosine similarity. The two semantic relevance scores are added together to obtain the comprehensive relevance score.

[0038] Further, step S5 includes the following steps:

[0039] Step S51: Question-answer pair semantic encoding: Use a pre-trained BERT encoder to perform deep semantic encoding on each question-answer pair to obtain the question-answer pair semantic encoding;

[0040] Step S52: Correlation weight normalization: Use the softmax function to normalize the overall correlation score and obtain the weighting coefficients;

[0041] Step S53: Weighted aggregation: Based on the weighting coefficients, perform weighted aggregation on the semantic encoding of all question-answer pairs to obtain a global question-answer knowledge representation.

[0042] Further, step S6 includes the following steps:

[0043] Step S61: Text modality encoding: Encode the text input using a pre-trained BERT encoder to obtain the text semantic encoding;

[0044] Step S62: Visual modality coding: Encode the video frame sequence using a pre-trained Vision Transformer (ViT) encoder to obtain the video semantic code;

[0045] Step S63: Audio Modality Coding: First, convert the audio track into MFCC features, and then use an audio encoder to extract features to obtain the audio semantic code.

[0046] Further, step S7 includes the following steps:

[0047] Step S71: Attention weight calculation: Define a scaled dot product attention mechanism, using text semantic encoding, video semantic encoding, or audio semantic encoding as the query, and global question-answering knowledge representation as the key and value, and calculate the attention output;

[0048] Step S72: Modality-specific attention fusion: Apply scaling dot product attention mechanism to text semantic coding, video semantic coding and audio semantic coding respectively to calculate the three modality attention outputs;

[0049] Step S73: Question-answering augmented representation fusion: Calculate the average of the attention outputs of the three modalities to obtain the fused representation of question-answering augmentation.

[0050] Further, step S8 includes the following steps:

[0051] Step S81: Feature Dimension Alignment: Project the video semantic code and audio semantic code onto the same dimensional space as the text semantic code to obtain the aligned video semantic code and the aligned audio semantic code;

[0052] Step S82: Multimodal feature connection: Connect the question-answering enhanced fusion representation with the text semantic code, the aligned video semantic code, and the aligned audio semantic code to obtain the final representation based on question-answering enhancement.

[0053] Further, step S9 includes the following steps:

[0054] Step S91: Predictor Network: Use a two-layer multilayer perceptron as the classifier;

[0055] Step S92: Loss function calculation: Optimize using the binary cross-entropy loss function.

[0056] Compared with the prior art, the present invention has the following beneficial technical effects:

[0057] 1) By introducing external knowledge enhancement and semantic reasoning mechanisms, combined with self-driven divergent reasoning, cross-modal relevance-guided integration, and question-answering perception cross-modal fusion mechanisms, this invention can effectively identify complex fake news video content and improve the accuracy and robustness of detection with the support of a multimodal large model.

[0058] 2) Improved detection accuracy: This invention enhances the model's ability to perceive and distinguish complex semantic manipulation and hidden rumor features through external knowledge enhancement and multi-level semantic reasoning.

[0059] 3) Enhanced robustness: This invention reduces the negative impact of single-modal noise and interference by deep fusion and inference of cross-modal information, making the detection process more stable and reliable.

[0060] 4) Optimization of cross-modal fusion effect: This invention guides the collaborative understanding and reasoning of multimodal information such as speech, text, and images through a QA perception mechanism, so as to achieve efficient alignment and effective utilization of cross-modal features.

[0061] 5) Enhanced generalization ability: This invention enhances the model's adaptability to different platforms and different types of fake news video scenarios through cross-modal correlation guidance and self-driven divergent reasoning.

[0062] 6) Technical problem solving: The present invention effectively overcomes the problems existing in the prior art, such as insufficient detection capability for complex semantic manipulation, lack of external knowledge enhancement, limited cross-modal information fusion effect, and poor cross-platform generalization capability. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a conceptual diagram of the fake news video detection method based on a multimodal large model self-driven question answering framework of the present invention.

[0065] Figure 2 This is a flowchart illustrating the fake news video detection method based on a multimodal large model self-driven question answering framework of the present invention.

[0066] Figure 3 This is a schematic diagram showing the cross-platform comparison of SAFE and REAL datasets in the FakeTT and FVC dataset transfer experiments of this invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] This invention proposes a method for detecting fake news videos based on a multimodal large model self-driven question answering framework, such as... Figure 1-2 As shown, the method includes the following steps:

[0069] Step S1: Preprocess and initialize the input video, that is, extract multimodal features and initialize the data structure of the input video.

[0070] Step S11: For the visual modality, extract keyframes from the input video using a uniform sampling strategy to obtain a sampled video frame sequence.

[0071] Extract L keyframes from the input video V using a uniform sampling strategy, where L is a preset number of frames (usually 16 frames). The sampled video frame sequence is represented as follows: in This represents the i-th video frame. J and W represent the height and width of the video frame, respectively, and 3 represents the three RGB color channels.

[0072] Step S12: For the audio modality, extract the complete audio track from the video.

[0073] Specifically, extracting the complete audio track from the video involves calling multimedia processing tools (such as FFmpeg) to deencode the video file and directly extract the original audio stream without transcoding. This ensures that the audio sampling rate, number of channels, and bit depth remain unchanged, thereby obtaining high-fidelity audio data that is completely consistent with the video.

[0074] Audio tracks are represented as Where T represents the duration of the audio and F represents the dimension of the audio features.

[0075] Step S13: For the text modality, collect the title information, screen display text, and speech-transcribed text, and concatenate the collected title information, screen display text, and speech-transcribed text into a complete text input.

[0076] Step S131: Obtain title information from input video metadata

[0077] Step S132: Extract the screen display text from the sampled video frame sequence

[0078] Specifically, the PaddleOCR tool was used to extract the screen display text from the sampled video frame sequence.

[0079] Step S133: Derive speech-transcribed text from the audio track

[0080] Specifically, the Whisper speech recognition model is used to derive the speech-transcribed text from the audio track.

[0081] Step S134: Combine the title information, screen-displayed text, and speech-transcribed text into a complete text input through a concatenation operation.

[0082] Text input is represented as Where ⊕ represents the string concatenation operation.

[0083] Step S14: Combine the video frame sequence representation, audio track, and text input to initialize the multimodal input and the question-answer pair set.

[0084] Multimodal input is represented as And set up a question-answer pair set. Dialogue turn counter t = 0, maximum dialogue turn number M max =5.

[0085] Step S2: Initialize the self-driven divergent reasoning module, which includes the questioner LMM (Large Language Model) and the responder LMM. Configure the roles and prompt templates of the questioner LMM and the responder LMM; initialize the dialogue history.

[0086] To support a self-driven divergent question-and-answer process, a specific prompt template was designed. For the questioner (LMM), the prompt requires them to act as an expert in verifying the authenticity of news videos, focusing on basic facts, visible elements, and obvious inconsistencies, asking only one short, direct, and easy-to-answer question at a time. For the respondent (LMM), the prompt requires them to answer each question precisely and concisely based on the video content, highlighting potential signs of misinformation.

[0087] Configure the Inquirer LMM (LMM) Inquirer ) prompt template They are required to act as experts in verifying the authenticity of news videos, focusing on identifying basic facts, visible elements, and obvious inconsistencies in the video content, and generating a clear, focused, and easy-to-answer question each time.

[0088] Configure Responder LMM (LMM) Responder ) prompt template They are required to answer the questions posed by the inquirer accurately and concisely based on the video content, highlighting potential signs of misinformation and conveying impactful insights using clear and precise language.

[0089] Initialize conversation history Used to store all generated question-and-answer pairs.

[0090] Step S3: Perform multi-turn dialogues through the questioner LMM and responder LMM of the self-driven divergent reasoning module, and generate high-quality question-answer pairs through the iterative dialogue process.

[0091] Perform a multi-turn dialogue loop, each turn of which includes the following sub-steps:

[0092] Step S31: Question Generation: Interrogator LMM based on multimodal input and conversation history Generate the problem for round t+1:

[0093]

[0094] in, LMM represents the question text generated in round t+1. Inquirer This indicates the inquirer, LMM.

[0095] Step S32: Termination condition check: If the question generated by the LMM is "No question" or the number of dialogue rounds reaches the maximum number of dialogue rounds M. max If the dialogue ends, the process will terminate; otherwise, it will continue.

[0096] Step S33: Answer Generation: Responder LMM Based on Multimodal Input and current issues Generate the corresponding answer:

[0097]

[0098] in, LMM represents the answer text generated in round t+1. Responder This indicates the respondent LMM.

[0099] Step S34: Store question-answer pairs: Add the newly generated question-answer pairs to the question-answer pair set:

[0100] Step S35: Update conversation history: Update conversation history Round counter t = t + 1.

[0101] Finally, a set of question-answer pairs containing M question-answer pairs is obtained. Where M represents the number of question-and-answer rounds, M≤M max ; This represents the question text generated in the i-th round. This represents the answer text generated in the i-th round.

[0102] Step S4: Calculate cross-modal relevance scores to quantify the semantic alignment of each question-answer pair with the video content.

[0103] The relevance score calculation considers the dual alignment of question-answer pairs with both video visual content and textual content. For visual content, a uniform representation is obtained by averaging features across all video frames. This design ensures that only question-answer knowledge highly relevant to the video content is retained, effectively filtering out illusory or noisy information that LMM might produce.

[0104] Step S41: Question-answer pair textification: Transform each question-answer pair into text. Convert the response pair into text using a join operation:

[0105]

[0106] in, This represents the text of the question-and-answer pair in the i-th round.

[0107] Step S42: Feature Encoding: Text Encoder Ψ using a pre-trained CLIP model t The question-and-answer text and the text input are encoded to obtain question-and-answer pair feature codes and text feature codes; the visual encoder Ψ of the pre-trained CLIP model is used. v Encode the video frame sequence to obtain the video feature code:

[0108]

[0109] Among them, e qa,i Ψ represents the feature encoding of the i-th round of question-and-answer pairs. t Indicates a text encoder, d clip Indicates the encoder's output dimension (typically 768 or 1024), e text Represents text feature encoding, e visual It is video feature encoding.

[0110] Step S43: Calculate the comprehensive relevance score: Calculate the semantic relevance score between the question-answer pair feature code and the video feature code, and the semantic relevance score between the question-answer pair feature code and the text feature code using cosine similarity. The sum of these two semantic relevance scores is the comprehensive relevance score.

[0111]

[0112] in, This represents the semantic relevance score between the feature encoding and the video feature encoding in the i-th round of question answering. Let ||·||2 represent the semantic relevance score between the feature encoding and the text feature encoding in the i-th round of question answering, and let ||·||2 represent the L2 norm.

[0113] The overall relevance score is calculated as follows:

[0114]

[0115] Where, α i This represents the overall relevance score of the i-th round of question and answer.

[0116] Step S5: Semantic encoding and weighted aggregation of question-answer pairs, and filtering and fusion of global question-answer knowledge representation based on comprehensive relevance scores.

[0117] Step S51: Question-answer pair semantic encoding: using a pre-trained BERT encoder ε t Deep semantic encoding is performed on each question-answer pair to obtain the question-answer pair semantic encoding:

[0118]

[0119] Among them, h qa,i Let ε represent the semantic encoding of the i-th round of question-and-answer pairs. t d represents the BERT encoder. bert This represents the hidden layer dimension of the BERT encoder (usually 768).

[0120] Step S52: Relevance Weight Normalization: Use the softmax function to normalize the overall relevance score and obtain the weighting coefficients.

[0121]

[0122] Among them, w i Let α represent the weighting coefficient for the i-th round of question and answer. j This represents the overall relevance score of the j-th round of question and answer.

[0123] Step S53: Weighted Aggregation: Based on the weighting coefficients, perform weighted aggregation on the semantic encoding of all question-answer pairs to obtain the global question-answer knowledge representation:

[0124]

[0125] Among them, h p This represents the global question-and-answer knowledge representation.

[0126] Step S6: Multimodal feature encoding, extracting independent deep features for each modality of the video.

[0127] Step S61: Text Modality Encoding: Encode the text input using a pre-trained BERT encoder to obtain the text semantic encoding.

[0128]

[0129] Where, x t This represents the semantic encoding of the text.

[0130] Step S62: Visual Modality Coding: Encode the video frame sequence using a pre-trained Vision Transformer (ViT) encoder to obtain the video semantic code.

[0131]

[0132] Where, x v Represents video semantic coding, ε v Indicates ViT encoder, d vit This represents the output feature dimension of ViT.

[0133] Step S63: Audio Modality Coding: First, convert the audio track into MFCC features, then use an audio encoder to extract features and obtain the audio semantic code.

[0134]

[0135] Where x a ε represents audio semantic coding, MFCC represents the Mel-frequency cepstral coefficient function extracted from the audio track, and ε represents the audio semantic coding. a Represents an audio feature encoder, d audio This indicates the dimension of audio features.

[0136] Step S7: Question-answering awareness cross-modal attention fusion to achieve fine-grained semantic alignment between global question-answering knowledge representation and features of each modality.

[0137] Cross-attention mechanisms enable each video modality to selectively focus on the most relevant parts of the global question-answering knowledge representation, rather than relying on simple connections or unified fusion. This fine-grained interaction allows the model to capture the semantic relationships between the global question-answering knowledge representation and different modalities, thereby improving detection performance.

[0138] Step S71: Attention Weight Calculation: Define a scaled dot product attention mechanism, using text semantic encoding, video semantic encoding, or audio semantic encoding as the query, and the global question-answering knowledge representation as the key and value.

[0139]

[0140] in, d represents the query vector.q Indicates the query vector dimension. Let d represent the key vector and value vector, respectively. k It is the dimension of the key vector. Here, T represents the scaling factor, T represents the transpose, and softmax represents the activation function. Attn(h) Q ,h K ,h V ) represents the attention weight calculation function.

[0141] Step S72: Modality-Specific Attention Fusion: Apply the scaled dot product attention mechanism to the text semantic code, video semantic code, and audio semantic code respectively to calculate the three modality attention outputs:

[0142]

[0143] Among them, h t h represents the text modal attention output. v h represents the video modal attention output. a This represents the audio modality attention output. and It is a learnable linear transformation matrix, W v W is used to unify the video feature dimensions to the same dimensionality space as the BERT encoder. a Unify the audio feature dimensions to the same dimensional space as the BERT encoder.

[0144] Step S73: Question-Answer Enhancement Representation Fusion: Calculate the average of the attention outputs from the three modalities to obtain the fused representation of question-answer enhancement.

[0145]

[0146] Where, x p This represents a question-and-answer enhanced fusion representation.

[0147] Step S8: Feature dimension unification and connection, ensuring that all features are fused in the same dimensional space.

[0148] Step S81: Feature Dimension Alignment: Project the video semantic code and audio semantic code onto the same dimensional space as the text semantic code to obtain the aligned video semantic code and the aligned audio semantic code.

[0149]

[0150] Where, x′ v This represents the aligned video semantic code, x′ a W′ represents the aligned audio semantic encoding. v ,W′ aThese are the learnable video weight matrix and audio weight matrix, respectively, b v ,b a These are the video bias vector and the audio bias vector, respectively.

[0151] Step S82: Multimodal Feature Concatenation: Concatenate the question-answering enhanced fusion representation with the text semantic code, the aligned video semantic code, and the aligned audio semantic code to obtain the final representation based on question-answering enhancement.

[0152]

[0153] Where, x final ⊕ represents the final representation based on question-answering enhancement, and ⊕ represents the vector concatenation operation.

[0154] Step S9: Knowledge-enhanced prediction and loss calculation, using the predictor network to perform the final binary classification prediction.

[0155] Step S91: Predictor Network: Using a two-layer Multilayer Perceptron (MLP) as the classifier:

[0156]

[0157] Where z represents the intermediate layer representation of the classifier. Indicates the prediction result. It is a learnable weight matrix, b1, b2 are bias vectors, and d hidden It is the hidden layer dimension, ReLU is the activation function, sigmoid is the activation function, and it ensures that the output probability is in the range [0,1].

[0158] Step S92: Loss function calculation: Optimization is performed using the binary cross-entropy loss function.

[0159]

[0160] in, Let y ∈ {0,1} represent the loss function, where y ∈ {0,1} is the true label (0 represents a real video, and 1 represents a fake news video). It is the probability output predicted by the model.

[0161] Step S10: Model training and parameter update, perform end-to-end supervised learning optimization to obtain the trained model.

[0162] Freeze the parameters of all pre-trained encoders (BERT encoder, ViT encoder, CLIP encoder, audio encoder) and optimize only task-specific parameters, including the dimensionality transformation matrix W. v W a ,W′ v ,W′ aAnd predictor parameters W1, W2 and their corresponding bias terms.

[0163] Step S11: Use the trained model to detect the input video.

[0164] To verify the effectiveness of the Fake News Video Detection Method (SAFE) based on a multimodal large model self-driven question answering framework on various types of datasets, this invention uses three publicly available and widely used video benchmark datasets for comprehensive evaluation:

[0165] FakeSV: A dataset specifically designed to detect fake news spread through Chinese short videos. The data was collected from two well-known Chinese short video platforms, Douyin and Kuaishou, and contains 3,624 videos (1,810 fake videos and 1,814 real videos), with an average duration of 39.88 seconds.

[0166] FakeTT: A dataset for detecting fake news in short English videos. The data comes from the TikTok platform and contains 1,991 videos (1,172 fake videos and 819 real videos), with an average duration of 47.69 seconds.

[0167] FVC: Constructs a dataset for analyzing the comparison between fake videos and real user-generated videos. The data comes from multiple social media platforms such as YouTube, Facebook, and Twitter, and contains 2,764 videos (1,633 fake videos and 1,131 real videos), with an average duration of 87.83 seconds.

[0168] To verify the effectiveness of the fake news video detection method based on the multimodal large model self-driven question answering framework of this invention, it was compared with 11 competing benchmark methods, divided into three groups:

[0169] Single-modal detection methods: BERT, ViT, AST, Qwen3. These methods mainly rely on single-modal features for anomaly detection.

[0170] Multimodal detection methods include FANVM, SV-FEND, NEED, FakingRecipe, and REAL. These methods combine information from multiple modalities for detection, but lack external knowledge enhancement.

[0171] LMM-based methods include GPT-4o-mini, LLaVA-OV, Qwen2-VL, and ExMRD. These methods utilize the capabilities of multimodal large models for detection.

[0172] Following previous research, this invention uses accuracy (ACC), macro average F1 score (M-F1), macro average precision (MP), and macro average recall (MR) as evaluation metrics.

[0173] This invention employs the advanced open-source multimodal large-scale model Qwen2.5-VL-72B-Instruct as the backbone network for both the questioner and responder. For the English datasets (FakeTT and FVC), openai / clip-vit-large-patch14 is used for encoding; for the Chinese dataset FakeSV, OFA-Sys / chinese-clip-vit-large-patch14 is used. Videos are uniformly sampled at 16 frames per second, the maximum text token length is set to 512, and the maximum number of rounds in a multi-turn dialogue is set to 5. During training, all pre-trained encoders are frozen, and only the detection-specific module is optimized using the AdamW optimizer with a learning rate of 5×10⁻⁶. -4 The batch size is 128.

[0174] Table 1 shows the performance comparison results of the Fake News Video Detection Method (SAFE) based on the multimodal large model self-driven question answering framework of this invention on three datasets with competing benchmark methods:

[0175] Table 1 shows the performance comparison results of SAFE and competing benchmark methods on three datasets.

[0176]

[0177] Table 1 shows the performance comparison results of SAFE and other comparative methods on the FakeSV, FakeTT, and FVC datasets. The best results are marked in bold, and the second best results are marked with an underline. Higher values ​​for ACC, M-F1, MP, and MR indicate better performance. Table 1 shows that: 1) Single-modal benchmark methods (BERT, ViT, AST, Qwen3) consistently perform poorly across all datasets, failing to capture cross-modal interactions and complementary cues in fake news videos. 2) Multimodal benchmark methods (FANVM, SV-FEND, NEED, FakingRecipe, REAL) show improvement over single-modal methods, but are still limited to utilizing the inherent signals in the video, lacking external knowledge or advanced reasoning capabilities. 3) LMM-based methods (GPT-4o-m, LLaVA-OV, Qwen2-VL, ExMRD), while possessing strong zero-shot capability, lack task-specific adaptability and exhibit inconsistent predictions when faced with complex deceptive content. 4) SAFE achieved state-of-the-art performance on all three benchmark datasets, with an average accuracy improvement of 2.74%, validating the effectiveness of external semantic knowledge and self-driven divergent reasoning.

[0178] To understand the contributions of each core component in SAFE, this invention conducted comprehensive ablation experiments. The ablation experiment results for the key components in SAFE are shown in Table 2.

[0179] Table 2: Ablation Experiment Table of Key Components in SAFE

[0180]

[0181] As shown in Table 2, the ablation experiments revealed that removing the multi-turn dialogue mechanism (without Dialogue) led to a performance degradation, validating the importance of self-driven divergent reasoning. Completely removing QA generation (without Reasoning) resulted in the most significant performance degradation, demonstrating the crucial role of external LMM knowledge. Removing relevance-guided weights (without Weight) also caused a performance degradation, highlighting the importance of selective integration. Removing the cross-attention mechanism (without Attention) impacted performance, validating the necessity of fine-grained semantic alignment. Using only QA knowledge (without Video) resulted in the worst performance, confirming the synergistic effect between intrinsic video features and external reasoning.

[0182] To evaluate SAFE's cross-platform generalization ability, this invention conducted transfer experiments between the FakeTT and FVC datasets. Figure 3 As shown, the results demonstrate that SAFE outperforms the strongest benchmark method, REAL, in both transition directions (FakeTT→FVC and FVC→FakeTT), with an average generalization performance improvement of 27.4%. This superior performance is attributed to the fact that the QA knowledge generated by LMM captures high-level, transferable semantic reasoning and is less sensitive to platform-specific surface patterns.

[0183] In summary, the experimental results fully verify the effectiveness of the SAFE framework of this invention and demonstrate the significant advantages of the self-driven question-answering mechanism in fake news video detection.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting fake news videos based on a multimodal large model self-driven question answering framework, characterized in that, The method includes the following steps: Step S1: Preprocess and initialize the input video, i.e., extract multimodal features and initialize the data structure of the input video; Step S2: Initialize the self-driven divergent reasoning module, which includes the questioner LMM and the responder LMM. Configure the roles and prompt templates of the questioner LMM and the responder LMM; initialize the dialogue history. To support the self-driven divergent question-and-answer process, a special prompt template was designed. For the questioner LMM, the prompt requires them to act as an expert in verifying the authenticity of news videos, focusing on basic facts, visible elements, and obvious inconsistencies, and asking only one short, direct, and easy-to-answer question each time. For the respondent LMM, the prompt requires them to answer each question accurately and concisely based on the video content, highlighting potential signs of misinformation. Step S3: Perform multi-turn dialogue through the questioner LMM and responder LMM of the self-driven divergent reasoning module, and generate high-quality question-answer pairs through the iterative dialogue process; Step S4: Calculate the cross-modal comprehensive relevance score to quantify the semantic alignment of each question-answer pair with the video content; Step S5: Semantic encoding and weighted aggregation of question-answer pairs, and filtering and fusion of global question-answer knowledge representation based on comprehensive relevance scores; Step S6: Multimodal feature encoding, performing independent deep feature extraction for each modality of the video; Step S7: Question-answering awareness cross-modal attention fusion to achieve fine-grained semantic alignment between global question-answering knowledge representation and features of each modality; Step S8: Feature dimension unification and connection, ensuring that all features are fused in the same dimensional space; Step S9: Knowledge-enhanced prediction and loss calculation, using the predictor network to perform the final binary classification prediction; Step S10: Model training and parameter update, perform end-to-end supervised learning optimization to obtain the trained model; Step S11: Use the trained model to detect the input video; Step S3 executes a multi-turn dialogue loop, each turn of dialogue containing the following sub-steps: Step S31: Question Generation: The questioner LMM generates the first question based on multimodal input and dialogue history. Wheel problem; Step S32: Termination condition check: If the question generated by the LMM is "No question" or the maximum number of dialogue rounds has been reached. If the condition is met, the dialogue process will terminate; otherwise, it will continue. Step S33: Answer Generation: The respondent LMM generates the corresponding answer based on the multimodal input and the current question; Step S34: Store question-answer pairs: Add the newly generated question-answer pairs to the question-answer pair set; Step S35: Update dialogue history: Update the dialogue history and increment the round counter by 1.

2. The fake news video detection method based on a multimodal large model self-driven question answering framework according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: For the visual modality, extract keyframes from the input video using a uniform sampling strategy to obtain a sampled video frame sequence; Step S12: For audio modalities, extract the complete audio track from the video; Step S13: For the text modality, collect the title information, the screen-displayed text, and the speech-transcribed text, and concatenate the collected title information, screen-displayed text, and speech-transcribed text into a complete text input; Step S14: Combine the video frame sequence representation, audio track, and text input to initialize the multimodal input and the question-answer pair set.

3. The fake news video detection method based on a multimodal large model self-driven question answering framework according to claim 2, characterized in that, Step S4 includes the following steps: Step S41: Question-answer pair textification: Convert the question and corresponding answer of each question-answer pair into text through a concatenation operation; Step S42: Feature Encoding: Use the pre-trained CLIP model's text encoder to encode the question-and-answer text and the text input to obtain the question-and-answer pair feature encoding and the text feature encoding; use the pre-trained CLIP model's visual encoder to encode the video frame sequence to obtain the video feature encoding; Step S43: Calculate the comprehensive relevance score: Calculate the semantic relevance score between the question-answer pair feature code and the video feature code and the semantic relevance score between the question-answer pair feature code and the text feature code using cosine similarity. The two semantic relevance scores are added together to obtain the comprehensive relevance score.

4. The fake news video detection method based on a multimodal large model self-driven question answering framework according to claim 3, characterized in that, Step S5 includes the following steps: Step S51: Question-answer pair semantic encoding: Use a pre-trained BERT encoder to perform deep semantic encoding on each question-answer pair to obtain the question-answer pair semantic encoding; Step S52: Correlation weight normalization: Use the softmax function to normalize the overall correlation score and obtain the weighting coefficients; Step S53: Weighted aggregation: Based on the weighting coefficients, perform weighted aggregation on the semantic encoding of all question-answer pairs to obtain a global question-answer knowledge representation.

5. The fake news video detection method based on a multimodal large model self-driven question answering framework according to claim 4, characterized in that, Step S6 includes the following steps: Step S61: Text modality encoding: Encode the text input using a pre-trained BERT encoder to obtain the text semantic encoding; Step S62: Visual modality coding: Encode the video frame sequence using a pre-trained Vision Transformer encoder to obtain the video semantic code; Step S63: Audio Modality Coding: First, convert the audio track into MFCC features, and then use an audio encoder to extract features to obtain the audio semantic code.

6. The fake news video detection method based on a multimodal large model self-driven question answering framework according to claim 5, characterized in that, Step S7 includes the following steps: Step S71: Attention weight calculation: Define a scaled dot product attention mechanism, using text semantic encoding, video semantic encoding, or audio semantic encoding as the query, and global question-answering knowledge representation as the key and value, and calculate the attention output; Step S72: Modality-specific attention fusion: Apply scaling dot product attention mechanism to text semantic coding, video semantic coding and audio semantic coding respectively to calculate the three modality attention outputs; Step S73: Question-answering augmented representation fusion: Calculate the average of the attention outputs of the three modalities to obtain the fused representation of question-answering augmentation.

7. The fake news video detection method based on a multimodal large model self-driven question answering framework according to claim 6, characterized in that, Step S8 includes the following steps: Step S81: Feature Dimension Alignment: Project the video semantic code and audio semantic code onto the same dimensional space as the text semantic code to obtain the aligned video semantic code and the aligned audio semantic code; Step S82: Multimodal feature connection: Connect the question-answering enhanced fusion representation with the text semantic code, the aligned video semantic code, and the aligned audio semantic code to obtain the final representation based on question-answering enhancement.

8. The fake news video detection method based on a multimodal large model self-driven question answering framework according to claim 6, characterized in that, Step S9 includes the following steps: Step S91: Predictor Network: Use a two-layer multilayer perceptron as the classifier; Step S92: Loss function calculation: Optimize using the binary cross-entropy loss function.

Images