A Fake News Video Detection Method Based on Retrieval Enhancement and Prototype Alignment Techniques

Abstract

This invention discloses a method for detecting fake news videos based on retrieval enhancement and prototype alignment techniques. First, the target video is processed to construct its multimodal information. Then, the multimodal information is integrated under a large-scale language model to generate a unified text-centric query, and video retrieval is performed to obtain real and fake video samples semantically related to the target video. Next, a dual prototype alignment mechanism is used to construct prototype representations for real and fake categories using a graph attention network, and a final operation-aware representation is generated through prototype alignment learning. Finally, the final operation-aware representation is integrated with an existing model to complete the detection of the target video content. This method can effectively identify subtle differences between real and fake news videos, achieving enhanced representation learning for video content detection and improving the accuracy and robustness of fake news video detection.

Description

Technical Field

[0001] This invention belongs to the field of fake news video detection technology, and relates to a fake news video detection method based on retrieval enhancement and prototype alignment technology. Background Technology

[0002] Fake news video detection is a research area that has become increasingly important in recent years with the rapid development of short video platforms. Video content has become a major medium for information dissemination, with platforms such as TikTok and YouTube Shorts attracting billions of active users every month. Therefore, developing effective methods for detecting fake news videos has become an urgent problem to be solved.

[0003] Existing methods for detecting fake news videos primarily focus on modeling multimodal content and capturing cross-modal correlations. Early detection methods relied mainly on single-modal information to assess video authenticity, such as BERT, ViT, and AST. However, due to the multimodal nature of videos—text, visual, and audio modalities providing complementary information to describe content—single-modal models cannot meet the requirements for accurate video detection. To address this issue, multimodal learning has attracted widespread attention in the field of fake news video detection, as exemplified by works such as FANVM, SV-FEND, NEED, and FakingRec.

[0004] In summary, current methods for detecting fake news videos suffer from problems such as over-reliance on information from individual videos, neglect of the referential value between related videos, and insufficient distinguishing ability of detection representations. This results in limited detection accuracy when models struggle to effectively distinguish between extremely subtle alterations to genuine news videos. Therefore, there is an urgent need for a fake news video detection method based on retrieval enhancement and prototype alignment techniques to improve detection performance. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a method for detecting fake news videos based on retrieval enhancement and prototype alignment techniques, which effectively solves the problems in the prior art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a method for detecting fake news videos based on retrieval enhancement and prototype alignment technology, comprising the following steps:

[0007] S1. Process the multimodal content of the target video and construct the multimodal information of the target video;

[0008] S2. Integrate multimodal information through a large-scale language model to generate a unified text-centric query;

[0009] S3. Based on the generated text-centered query, perform video-to-video retrieval to obtain real video samples and fake video samples that are semantically related to the target video.

[0010] S4. Using the real video samples and fake video samples obtained in step S3, construct real video category prototype representations and fake video category prototype representations based on context information graphs;

[0011] S5. Calculate the attention weights between nodes and aggregate information through a graph attention network to form an aggregated fake video prototype and a real video prototype.

[0012] S6. Perform prototype alignment learning on the target video to generate the final operation-aware representation;

[0013] S7. The final operational perception representation is used to predict the target video output by the existing model in the form of residual connections.

[0014] Furthermore, step S8 is included: optimizing the existing model parameters by combining the binary cross-entropy loss and the prototype loss.

[0015] Furthermore, the process in step S3 is described by the following formula:

[0016]

[0017] in This represents a pre-trained text encoder. It is a repository for storing diverse videos. This represents the first K videos retrieved;

[0018] The first retrieved A fake video sample, denoted as a fake video sample. ;

[0019] The first retrieved A real video sample is represented as a real video sample. .

[0020] Furthermore, the process in step S4 is described by the following formula:

[0021] For the retrieved fake video samples Build a context information graph Each node Indicates fake video samples Modal-specific features ,in Modal type;

[0022] For the retrieved real video samples Build a context information graph Each node Represents real video samples Modal-specific features .

[0023] Furthermore, the process of aggregating fake video prototypes in step S5 is described by the following formula:

[0024] ;

[0025] ;

[0026] in, This reflects the pairwise relationship between nodes i and j based on feature semantic similarity; where and These are trainable parameters. Represents a node The neighbor set in the graph;

[0027] Using attention weights Each node The update indicates Calculate the weighted aggregation of its neighbors:

[0028] ;

[0029] in It is a non-linear activation function;

[0030] After graph aggregation The embeddings of all nodes are pooled to form a spurious prototype. :

[0031] ;

[0032] The aggregated real-world video prototypes are obtained through the same process described above and represented as follows: .

[0033] Furthermore, the process in step 6 is described as follows:

[0034] First, the modal features of the target video are... The input feedforward network obtains the raw manipulation-aware representation:

[0035]

[0036] in It is a manipulation perception representation; FFN is a feedforward network; , This is the weight matrix; , For bias terms;

[0037] Then define the dual-prototype alignment loss, and batch One target video of Aligning itself with its true class prototype and distancing itself from its opposite class prototype, as expressed by the following formula:

[0038]

[0039] in express The true category prototype, yes The opposite category prototype.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This invention introduces a large-model-based video retrieval mechanism and a dual-prototype alignment learning framework, which can effectively identify subtle differences between real and fake news videos. It achieves enhanced representation learning for video content detection, improving the accuracy and robustness of fake news video detection. It is applicable to social media platforms, news organizations, and content moderation systems. Attached Figure Description

[0042] Figure 1 This is an overall framework diagram of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, this invention proposes a method for detecting fake news videos based on retrieval enhancement and prototype alignment techniques, comprising the following steps:

[0045] S1. Process the multimodal content of the target video and construct its multimodal information; specifically, given a target video... The video consists of text, visual, and audio modalities, represented as First, uniform sampling is performed from the target video. Each frame is processed, and then a pre-trained BLIP model is used to generate a text description for each frame, converting the visual information into text form, represented as... At the same time, the Whisper model is used to analyze the target video. The audio is converted into text format and represented as .

[0046] S2. Integrate multimodal information through a large-scale language model to generate a unified text-centric query; specifically, the text description in step S1... Original text and audio text Input a large language model to generate a unified text-centric query. It is expressed as follows:

[0047] ;

[0048] in Represents a large language model, This is a prompt input into a large language model, a concise version of which is: "Suppose you are a multimodal information organization expert. Your task is to summarize information from the visual, textual, and audio content of a given target video: [] ]、[ ]and[ "Provide a concise and accurate description to effectively represent the content of the video." This large-scale language model can employ models such as the Beanbag model, Deepseek model, or ChatGPT model.

[0049] S3. Based on the generated text-centric query, perform video-to-video retrieval to obtain real and fake video samples semantically related to the target video; specifically, as expressed in the following formula:

[0050]

[0051] in This represents a pre-trained text encoder. It is a repository for storing diverse videos. This represents the first K videos retrieved;

[0052] The first retrieved A fake video sample, denoted as a fake video sample. ;

[0053] The first retrieved A real video sample is represented as a real video sample. .

[0054] Using both fake and real videos as context-dependent reference sets, we can guide the target video... Manipulation perception representation learning.

[0055] S4. Using the real and fake video samples obtained in step S3, construct prototype representations of real and fake video categories based on context information graphs; specifically, as expressed in the following formula:

[0056] For the retrieved fake video samples Build a context information graph Each node Indicates fake video samples Modal-specific features ,in Modal type;

[0057] For the retrieved real video samples Build a context information graph Each node Represents real video samples Modal-specific features .

[0058] S5. Calculate the attention weights between nodes using a graph attention network and aggregate the information to form the aggregated fake video prototype and the real video prototype; specifically, the process of aggregating the fake video prototype is expressed by the following formula:

[0059] ;

[0060] ;

[0061] in, This reflects the pairwise relationship between nodes i and j based on feature semantic similarity; where and These are trainable parameters. Represents a node The neighbor set in the graph;

[0062] Using attention weights Each node The update indicates Calculate the weighted aggregation of its neighbors:

[0063] ;

[0064] in It is a non-linear activation function;

[0065] After graph aggregation The embeddings of all nodes are pooled to form a spurious prototype. :

[0066] ;

[0067] The aggregated real-world video prototypes are obtained through the same process described above and represented as follows: .

[0068] S6. Perform prototype alignment learning on the target video to generate the final operation-aware representation; specifically, the process is described as follows: First, the modal features of the target video are... The input feedforward network obtains the raw manipulation-aware representation:

[0069]

[0070] in It is a manipulation perception representation; FFN is a feedforward network; , This is the weight matrix; , For bias terms;

[0071] Then define the dual-prototype alignment loss, and batch One target video of Aligning itself with its true class prototype and distancing itself from its opposite class prototype, as expressed by the following formula:

[0072]

[0073] in express The true category prototype, yes The opposite category prototype.

[0074] S7. The final operational perception representation is used for predicting the target video output by the existing model using residual connections; specifically, the process is described by the following formula:

[0075] ;

[0076] in The target video Prediction categories, and These represent the multimodal fusion network and the prediction network in various fake news video detection methods, respectively.

[0077] This invention also includes step S8: optimizing the existing model parameters by combining the binary cross-entropy loss and the prototype loss; specifically, the process is described by the following formula:

[0078]

[0079] in yes The tag, Indicates batch size. and This parameter balances the two losses. By optimizing the joint loss, the performance of existing methods in detecting fake news videos is improved.

[0080] Finally, the model trained using this invention is applied to test data to effectively identify fake news videos by judging their authenticity. Specifically, the experimental results are shown below:

[0081] 1. Experimental setup

[0082] 1.1. Dataset

[0083] To verify this invention, experiments were conducted on two real-world video datasets: FakeSV and FakeTT. These datasets contain a large number of real and fake news videos, covering different languages ​​and video types. Detailed statistics of the datasets are shown in the table below:

[0084] Dataset language Positive sample size negative sample number Total sample size Average length (seconds) FakeSV Chinese 1,810 1,814 3,624 39.88 FakeTT English 1,172 819 1,991 47.69

[0085] 1.2. Measurement Standards

[0086] Two metrics were used to evaluate performance: Accuracy (ACC) and Macro-F1 Score (M-F1). ACC measures the overall accuracy of the model's predictions, representing the proportion of correctly classified samples among all predictions, with a value ranging from [0,1]. A higher value indicates more accurate overall predictions. The M-F1 score is calculated by averaging the F1 scores for each class. In cases of class imbalance, it more objectively reflects the model's comprehensive ability to distinguish between classes. Its value also ranges from [0,1], with a higher value indicating a better balance between precision and recall.

[0087] 1.3. Baseline Method

[0088] To evaluate the performance of this invention (REAL), five baseline detectors were selected. The invention (REAL) was then plugged into these baseline detectors in a plug-and-play manner (i.e., residual connection), and the improvement in performance was used to determine the effectiveness of the model. These baseline detection methods were divided into two groups:

[0089] (1) Single-modal detection methods: BERT, ViT and AST. These methods only use information from one modality in the video to detect whether it contains false information.

[0090] (2) Multimodal detection methods: FANVM and SV-FEND. These methods use multimodal interaction and fusion to detect false information content by utilizing multiple modal information in the video.

[0091] 2. Overall performance

[0092] In the experiments, the performance of each baseline model was evaluated with and without the method of this invention. The table below shows the performance comparison of the baseline models before and after integrating this invention on two datasets:

[0093] Model Dataset ACC% ACC% (+REAL) M-F1% M-F1% (+REAL) BERT FakeSV 76.88 79.81 (+3.8%) 76.40 79.52 (+4.1%) BERT FakeTT 63.54 69.23 (+8.9%) 63.01 67.16 (+6.6%) ViT FakeSV 70.84 76.19 (+7.6%) 70.84 75.95 (+7.2%) ViT FakeTT 64.21 71.90 (+12.0%) 63.89 70.51 (+10.4%) AST FakeSV 68.00 70.06 (+3.0%) 67.12 69.34 (+3.3%) AST FakeTT 60.87 62.87 (+3.3%) 60.73 62.39 (+2.7%) FANVM FakeSV 75.70 79.70 (+5.3%) 75.18 79.13 (+5.3%) FANVM FakeTT 72.24 74.24 (+2.8%) 70.53 72.66 (+3.0%) SV-FEND FakeSV 77.22 80.07 (+3.7%) 76.46 79.67 (+4.2%) SV-FEND FakeTT 71.57 74.58 (+4.2%) 70.30 72.56 (+3.2%)

[0094] Table 1: Comparison of ACC and M-F1 values ​​of different baseline models and those after incorporating the REAL model of this invention.

[0095] Based on the comparison of the above experimental results, the following observations can be drawn:

[0096] (1) The results show that, after integrating the present invention, all five baseline models exhibit significant performance improvements. These results highlight the effectiveness and versatility of the present invention, which enhances the discriminative power of baseline models in detecting fake news videos by generating more discriminative and expressive representations.

[0097] (2) This invention provides a more significant improvement over poorly performing models. This is assumed to stem from the tendency of these models to misclassify versions of real news videos with extremely subtle changes. By utilizing manipulation-aware representations, this invention alleviates this limitation and establishes a stronger lower bound on the detection performance of these poorly performing methods.

[0098] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting fake news videos based on retrieval enhancement and prototype alignment techniques, characterized in that, Includes the following steps: S1. Process the multimodal content of the target video and construct the multimodal information of the target video; S2. Integrate multimodal information through a large-scale language model to generate a unified text-centric query; S3. Based on the generated text-centered query, perform video-to-video retrieval to obtain real video samples and fake video samples that are semantically related to the target video. S4. Using the real video samples and fake video samples obtained in step S3, construct real video category prototype representations and fake video category prototype representations based on context information graphs; S5. Calculate the attention weights between nodes and aggregate information through a graph attention network to form an aggregated fake video prototype and a real video prototype. S6. Perform prototype alignment learning on the target video to generate the final operation-aware representation; S7. The final operational perception representation is used to predict the category of the target video output by the existing model in the form of residual connections; The process in step S3 is described by the following formula: in This represents a pre-trained text encoder. It is a repository for storing diverse videos. This represents the first K videos retrieved; The first retrieved A fake video sample, denoted as a fake video sample. ; The first retrieved A real video sample is represented as a real video sample. ; The process in step S4 is described by the following formula: For the retrieved fake video samples Build a context information graph Each node Indicates fake video samples Modal-specific features ,in Modal type; For the retrieved real video samples Build a context information graph Each node Represents real video samples Modal-specific features ; The process of aggregating fake video prototypes in step S5 is expressed by the following formula: ； ； in, This reflects the pairwise relationship between nodes i and j based on feature semantic similarity; where and These are trainable parameters. Represents a node The neighbor set in the graph; Using attention weights Each node The update indicates Calculate the weighted aggregation of its neighbors: ； in It is a non-linear activation function; After graph aggregation The embeddings of all nodes are pooled to form a spurious prototype. : ； The aggregated real-world video prototypes are obtained through the same process described above and represented as follows: ; The process in step 6 is described as follows: First, the modal features of the target video are... The input feedforward network obtains the raw manipulation-aware representation: in It is a manipulation perception representation; FFN is a feedforward network; , This is the weight matrix; , For bias terms; Then define the dual-prototype alignment loss, and batch One target video of Aligning itself with its true class prototype and distancing itself from its opposite class prototype, as expressed by the following formula: in express The true category prototype, yes The opposite category prototype.

2. The method for detecting fake news videos based on retrieval enhancement and prototype alignment technology according to claim 1, characterized in that, It also includes step S8: combining binary cross-entropy loss and prototype loss to optimize the existing model parameters.

Images