False news video detection method based on retrieval enhancement and prototype alignment technology

Abstract

The invention discloses a false news video detection method based on a retrieval enhancement and prototype alignment technology, which comprises the following steps of: processing a target video, constructing multi-modal information of the target video, and integrating the multi-modal information under a large language model to generate a unified text center query; performing video retrieval to obtain real and false video samples related to target video semantics; then respectively constructing prototype representations of real and false categories by using a graph attention network through a double prototype alignment mechanism, and generating a final operation perception representation through prototype alignment learning; and integrating the final operation perception representation with the existing model to complete the detection of the target video content. The subtle difference between the real news video and the false news video can be effectively identified, the detection of the video content is enhanced, the representation learning is realized, and the accuracy and robustness of the detection of the false news video are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of false news video detection and relates to a false news video detection method based on retrieval enhancement and prototype alignment technology. Background Art

[0002] Fake news video detection has become an increasingly important research area in recent years with the rapid growth of short video platforms. Video content has become a primary medium for information dissemination, with platforms such as TikTok and YouTube Shorts attracting billions of active users each month. However, the widespread consumption of news on video platforms has also accelerated the spread of malicious content (such as fake news) in videos, posing a real threat to politics, finance, and public health. Therefore, developing effective fake news video detection methods has become an urgent problem.

[0003] Existing methods for detecting fake news videos primarily focus on modeling multimodal content and capturing cross-modal correlations. Early detection methods, such as BERT, ViT, and AST, primarily relied on single-modal information to assess video authenticity. However, due to the multimodal nature of video, where text, visual, and audio modalities provide complementary information to describe content, single-modal models cannot meet the requirements for accurate video detection. To address this issue, multimodal learning has attracted widespread attention in the field of fake news video detection, with works such as FANVM, SV-FEND, NEED, and FakingRec.

[0004] However, existing work still faces challenges in effectively identifying the subtle differences between real news videos and their subtly manipulated fake versions, resulting in limited detection performance. Research shows that most fake news videos are not newly created, but are instead manipulated from real news videos. Fake news video creators often make subtle but critical changes to the original content, such as distorting the narrative or editing visual, text, and audio elements to spread false information. For example, a manipulated fake news video may only make minor changes to the on-screen text and audio, while maintaining almost identical visual content to the original real version. Due to this high similarity, identifying reliable and discriminative features becomes particularly challenging, causing existing detection methods to incorrectly classify them as the same category.

[0005] In summary, current methods for detecting fake news videos suffer from over-reliance on the inherent information of a single video, neglect of the reference value between related videos, and insufficient discriminative detection representation capabilities. As a result, these models struggle to effectively distinguish even subtly altered versions of real news videos, resulting in limited detection accuracy. Therefore, a fake news video detection method based on retrieval enhancement and prototype alignment is urgently needed to improve detection performance. Summary of the Invention

[0006] To address the above problems, the present invention proposes a fake news video detection method based on retrieval enhancement and prototype alignment technology, which effectively solves the problems in the existing technology.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for detecting fake news videos based on retrieval enhancement and prototype alignment technology, comprising the following steps:

[0008] S1. Processing the multimodal content of the target video and constructing the multimodal information of the target video;

[0009] S2, integrating multimodal information through large language models to generate unified text-centric queries;

[0010] S3. Based on the generated text-centric query, perform video-to-video retrieval to obtain real video samples and fake video samples that are semantically related to the target video;

[0011] S4. Using the real video samples and fake video samples obtained in step S3, construct a real video category prototype representation and a fake video category prototype representation based on the context information graph;

[0012] S5. Calculate the attention weights between nodes through the graph attention network and perform information aggregation to form aggregated fake video prototypes and real video prototypes;

[0013] S6, perform prototype alignment learning on the target video to generate the final action perception representation;

[0014] S7. The final operation perception representation is used to output the predicted category of the target video by the existing model in a residual connection manner.

[0015] Furthermore, the method further includes step S8: optimizing existing model parameters by combining binary cross entropy loss and prototype loss.

[0016] Furthermore, the process in step S3 is expressed as follows:

[0017]

[0018] where Ψ(·) represents the pre-trained text encoder, It is a repository for storing diverse videos. represents the first K videos retrieved;

[0019] The top K retrieved f Fake video samples, represented as fake video samples

[0020] The top K retrieved rreal video samples, represented as real video samples

[0021] Furthermore, the process in step S4 is expressed as follows:

[0022] For the retrieved fake video samples Build contextual infographics Each node Represents a fake video sample Modality-specific features of Where m∈{t, v, a} is the modal type;

[0023] For the retrieved real video samples Build contextual infographics Each node Represents a real video sample Modality-specific features of

[0024] Furthermore, the fake video prototype process aggregated in step S5 is expressed as follows:

[0025]

[0026] Among them, e ij ∈ε f Reflects the pairwise relationship between nodes i and j based on feature semantic similarity; where W and a are trainable parameters, Represents node v i The set of neighbors in the graph;

[0027] Using attention weight α ij , each node v i Update representation Compute the weighted aggregate of its neighbors:

[0028]

[0029] Where σ(·) is a nonlinear activation function;

[0030] After graph aggregation, The embeddings of all nodes in are pooled to form a false prototype P f,m :

[0031]

[0032] The aggregated real video prototype is obtained through the same process as above and is represented as P r,m .

[0033] Furthermore, the process in step 6 is described as follows:

[0034] First, the modal feature E of the target video m Input the feedforward network to obtain the original manipulation perception representation:

[0035] M m =FFN(E m )=W2·σ(W1E m +b1)+b2;

[0036] Among them, M m ,m∈{t, v, a} is the manipulation perception representation; FFN is the feedforward network; W2, W1 are weight matrices; b1, b2 are bias terms;

[0037] Then define the dual prototype alignment loss, and transform a batch of N target videos of Aligned with its true category prototype and away from the opposite category prototype, expressed by the following formula:

[0038]

[0039] in express The true category prototype, yes The opposite category prototype.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] In this invention, by introducing a large-model-based video retrieval mechanism and a dual prototype alignment learning framework, it can effectively identify the subtle differences between real news videos and fake news videos, realize enhanced representation learning for video content detection, and improve the accuracy and robustness of fake news video detection. It is suitable for social media platforms, news organizations, content review systems and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] like Figure 1As shown, the present invention proposes a fake news video detection method based on retrieval enhancement and prototype alignment technology, comprising the following steps:

[0045] S1. Process the multimodal content of the target video and construct the multimodal information of the target video; specifically, given a target video The video consists of text, visual and audio modalities, represented as First, M frames are uniformly sampled from the target video, and then a pre-trained BLIP model is used to generate a text description for each frame, converting the visual information into text form, which is expressed as At the same time, the Whisper model is used to convert the target video The audio is converted into text form, represented as

[0046] S2, integrate multimodal information through a large language model to generate a unified text-centric query; specifically, the text description in step S1 Original text and audio text Input a large language model to generate unified text-centric queries It is expressed as the following formula:

[0047]

[0048] in Represents a large language model, is the prompt input to a large language model. The concise version is: "Assume that you are an expert in multimodal information organization. Your task is to summarize information from the visual, textual, and audio content of a given target video: and Provide a concise and accurate description to effectively represent the content of the video. The large language model can adopt the Doubao large model, Deepseek large model or ChatGPT large model.

[0049] S3. Based on the generated text-centric query, perform video-to-video retrieval to obtain real video samples and fake video samples that are semantically related to the target video. Specifically, it is expressed as follows:

[0050]

[0051] where Ψ(·) represents the pre-trained text encoder, It is a repository for storing diverse videos. represents the first K videos retrieved;

[0052] The top K retrieved f Fake video samples, represented as fake video samples

[0053] The first Kr real video samples retrieved are denoted as real video samples

[0054] For the retrieved fake and real videos as context-related reference sets, the target video is guided Manipulation-aware representation learning.

[0055] S4. Using the real video samples and fake video samples obtained in step S3, construct a real video category prototype representation and a fake video category prototype representation based on the context information graph; specifically, they are expressed as follows:

[0056] For the retrieved fake video samples Build contextual infographics Each node Represents a fake video sample Modality-specific features of Where m∈{t, v, a} is the modal type;

[0057] For the retrieved real video samples Build contextual infographics Each node Represents a real video sample Modality-specific features of

[0058] S5. Calculate the attention weights between nodes through the graph attention network and perform information aggregation to form aggregated fake video prototypes and real video prototypes. Specifically, the aggregation process of fake video prototypes is expressed as follows:

[0059]

[0060] Among them, e ij ∈ε f Reflects the pairwise relationship between nodes i and j based on feature semantic similarity; where W and a are trainable parameters, Represents node v i The set of neighbors in the graph;

[0061] Using attention weight α ij , each node v i Update representation Compute the weighted aggregate of its neighbors:

[0062]

[0063] Where σ(·) is a nonlinear activation function;

[0064] After graph aggregation, The embeddings of all nodes in are pooled to form a false prototype P f,m :

[0065]

[0066] The aggregated real video prototype is obtained through the same process as above and is represented as P r,m .

[0067] S6. Perform prototype alignment learning on the target video to generate the final operation perception representation. Specifically, the process is described as follows: First, the modal feature E m Input the feedforward network to obtain the original manipulation perception representation:

[0068] M m =FFN(E m )=W2·σ(W1E m +b1)+b2;

[0069] Among them, M m ,m∈{t, v, a} is the manipulation perception representation; FFN is the feedforward network; W2, W1 are weight matrices; b1, b2 are bias terms;

[0070] Then define the dual prototype alignment loss, and transform a batch of N target videos of Aligned with its true category prototype and away from the opposite category prototype, expressed by the following formula:

[0071]

[0072] in express The true category prototype, yes The opposite category prototype.

[0073] S7. The final operation perception representation is used for the prediction category of the target video output by the existing model in a residual connection manner. Specifically, the process is expressed as follows:

[0074]

[0075] in The target video The predicted category, and Predictor(·) represent the multimodal fusion network and prediction network in various fake news video detection methods, respectively;

[0076] The present invention further includes step S8: optimizing the existing model parameters by combining binary cross entropy loss and prototype loss. Specifically, the process is expressed as follows:

[0077]

[0078] where y i yes The proposed method combines the two losses into a single loss function, where N represents the batch size, α and β are parameters that balance the two losses. By optimizing the joint loss, the performance of existing methods in fake news video detection is improved.

[0079] Finally, the model trained by this invention is applied to test data to effectively identify fake news videos by judging the authenticity of the videos. Specifically, the experimental verification results are shown as follows:

[0080] 1. Experimental Setup

[0081] Dataset

[0082] To validate our invention, we conducted experiments on two real-world video datasets: FakeSV and FakeTT. These datasets contain a large number of real and fake news videos, covering a variety of languages ​​and video types. Detailed statistics for the datasets are shown in the following table:

[0083]

[0084] Metrics

[0085] Two metrics are used to evaluate performance: accuracy (ACC) and macro F1 score (M-F1). ACC measures the accuracy of the model's overall predictions, representing the proportion of correctly classified samples among all predictions. Its value range is [0, 1], with higher values ​​indicating more accurate predictions. The M-F1 score, calculated by averaging the F1 values ​​for each class, more objectively reflects the model's comprehensive ability to discriminate across classes when the sample classes are imbalanced. Its value range is also [0, 1], with higher values ​​indicating a better balance between precision and recall.

[0086] Baseline Methods

[0087] To evaluate the performance of the proposed method (REAL), we selected five baseline detectors and inserted the proposed method (REAL) into these baseline detectors using a plug-and-play approach (i.e., residual connections). The performance improvement of the proposed method was used to determine the effectiveness of the proposed model. These baseline detection methods were divided into two groups:

[0088] (1) Single-modal detection methods: BERT, ViT, and AST. These methods only use information from one modality in the video to detect whether it contains false information.

[0089] (2) Multimodal detection methods: FANVM and SV-FEND. These methods use multimodal interaction and fusion to detect false information content using multiple modal information in the video.

[0090] 2. Overall performance

[0091] In the experiment, the performance of each baseline model was evaluated without and with the method of the present invention. The following table shows the performance comparison of the baseline model before and after integrating the method of the present invention on two datasets:

[0092]

[0093] Table 1: Comparison of ACC and M-F1 values ​​of different baseline models and after adding the REAL model of the present invention

[0094] Based on the comparison of the above experimental results, the following observations are drawn:

[0095] (1) The results show that all five baseline models show significant performance improvements after integrating our method. These results highlight the effectiveness and versatility of our method, which enhances the discriminative ability of baseline models in fake news video detection by generating more discriminative and expressive representations.

[0096] (2) The present invention produces more significant improvements for poorly performing models. We hypothesize that this phenomenon stems from the tendency of these models to misclassify subtly altered versions of real news videos. By leveraging manipulated perceptual representations, the present invention mitigates this limitation and establishes a stronger lower bound on the detection performance of these poorly performing methods.

[0097] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fake news video detection method based on retrieval enhancement and prototype alignment technology, characterized by: The following steps are involved: S1. Processing the multimodal content of the target video and constructing the multimodal information of the target video; S2, integrating multimodal information through large language models to generate unified text-centric queries; S3. Based on the generated text-centric query, perform video-to-video retrieval to obtain real video samples and fake video samples that are semantically related to the target video; S4. Using the real video samples and fake video samples obtained in step S3, construct a real video category prototype representation and a fake video category prototype representation based on the context information graph; S5. Calculate the attention weights between nodes through the graph attention network and perform information aggregation to form aggregated fake video prototypes and real video prototypes; S6, perform prototype alignment learning on the target video to generate the final action perception representation; S7. The final operation perception representation is used to output the predicted category of the target video by the existing model in a residual connection manner.

2. The fake news video detection method based on retrieval enhancement and prototype alignment technology according to claim 1 is characterized in that: The method further includes step S8: optimizing the existing model parameters by combining binary cross entropy loss and prototype loss.

3. The fake news video detection method based on retrieval enhancement and prototype alignment technology according to claim 1 is characterized in that: The process in step S3 is expressed as follows: where Ψ(·) represents the pre-trained text encoder, It is a repository for storing diverse videos. represents the first K videos retrieved; The top K retrieved f Fake video samples, represented as fake video samples The top K retrieved r real video samples, represented as real video samples 4. The fake news video detection method based on retrieval enhancement and prototype alignment technology as described in claim 3 is characterized in that: The process in step S4 is expressed as follows: For the retrieved fake video samples Build contextual infographics Each of these nodes Represents a fake video sample Modality-specific features of Where m∈{t, v, a} is the modal type; For the retrieved real video samples Build contextual infographics Each node Represents a real video sample Modality-specific features of 5. The fake news video detection method based on retrieval enhancement and prototype alignment technology according to claim 4 is characterized in that: The fake video prototype process aggregated in step S5 is expressed as follows: Among them, e ij ∈ε f Reflects the pairwise relationship between nodes i and j based on feature semantic similarity; where W and a are trainable parameters, Represents node v i The set of neighbors in the graph; Using attention weight α ij , each node v i Update representation Compute the weighted aggregate of its neighbors: Where σ(·) is a nonlinear activation function; After graph aggregation, The embeddings of all nodes in are pooled to form a false prototype P f，m : The aggregated real video prototype is obtained through the same process as above and is represented as P r，m .

6. The fake news video detection method based on retrieval enhancement and prototype alignment technology according to claim 5 is characterized in that: The process in step 6 is described as follows: First, the modal feature E of the target video m Input the feedforward network to obtain the original manipulation perception representation: M m =FFN(E m )=W2·σ(W1E m +b1)+b2; Among them, M m , m∈{t, v, a} is the manipulation perception representation; FFN is the feedforward network; W2, W1 are weight matrices; b1, b2 are bias terms; Then define the dual prototype alignment loss, and transform a batch of N target videos of Aligned with its true category prototype and away from the opposite category prototype, expressed by the following formula: in express The true category prototype, yes The opposite category prototype.

Images