Deep counterfeit content detection system and method based on multi-modal large model
The deepfake content detection system based on a multimodal large model utilizes video Swin Transformer and ERes2Net to extract visual and audio features, and improves detection accuracy and speed through an audio-visual local-global interaction module. This solves the problems of insufficient interpretability of detection results and lack of long video understanding ability in existing technologies, and achieves efficient and accurate deepfake detection.
Patent Information
- Application Number
- CN202610082717.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-21
- Publication Date
- 2026-05-26
AI Technical Summary
Existing multimodal learning-based deepfake detection solutions are insufficient in terms of interpretability of detection results and understanding of long video content, making it difficult to adapt to the complex and ever-changing needs of real-world scenarios.
A deep forgery content detection system based on a multimodal large model is adopted, including a video Swin Transformer visual feature extraction branch, an ERes2Net interactive audio feature extraction branch, and an audio-visual local-global interaction and detection module. The feature extraction and detection accuracy are improved through a multi-level fusion mechanism and an attention mechanism.
It achieves accurate and rapid detection of multimodal deepfake content, improves the interpretability of detection and system efficiency, reaches an accuracy of 99.9998%, and has a single inference time of less than 0.006 seconds on an RTX3090 machine.
Smart Images
Figure CN122087323A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal risk content recognition technology, and relates to a deepfake content detection system and method based on a multimodal large model. Background Technology
[0002] In the field of deepfake detection using multimodal learning, existing deepfake detection schemes such as SyncNet, CLIP, and AudioCLIP primarily focus on facial forgery recognition. They utilize the inherent correlation between mouth articulation (visual position) and speech units (phonemes), as well as the alignment of subtle emotional differences in facial and vocal expressions to achieve detection. However, the detection results obtained are usually limited to single binary or multi-class classification, such as simply judging true or false, lacking detailed guidance on the detection process, resulting in poor interpretability and difficulty in meeting the increasingly complex and ever-changing needs of real-world scenarios.
[0003] Meanwhile, the current mainstream large language models are still limited in their contextual support. The maximum supported length of 32K to 200K is insufficient for fine-grained understanding of long videos. Mainstream video large language models convert each frame into a 32-bit tag length. This coarse-grained representation is still effective in simple video question answering and video description tasks, but its effectiveness is greatly reduced in scenarios that require fine-grained understanding of each frame's content.
[0004] from Figure 1 As can be seen, the video big language model series algorithms based on multimodal big models have low accuracy on multiple test sets. Figure 1 Adapted from “One Million Deepfake Audio and Video: A Large-Scale, Model-Driven Dataset for Deepfake Audio and Video” (Zhixi Cai et al., published in the Proceedings of the 32nd ACM International Multimedia Conference, 2024, pp. 11355-11359).
[0005] In summary, existing multimodal learning-based deepfake detection schemes and large language models have significant shortcomings in terms of the interpretability and adaptability of detection results, as well as their ability to understand long video content when handling related tasks, which seriously restricts the further development and application of this field. Summary of the Invention
[0006] This invention provides a deepfake content detection system and method based on a multimodal large model, aiming to achieve more accurate, faster and more interpretable detection of multimodal deepfake content through technological innovation, while improving the system's efficiency and applicability through intelligent resource management.
[0007] Specifically, this system consists of a deepfake expert detection system and a multi-level descriptive video understanding and generation system based on a multimodal large model. Based on a lightweight, high-performance deepfake expert detection system and a multi-level descriptive multimodal large model, it provides different levels of interaction according to different user needs. It integrates the excellent modal understanding and text generation capabilities of the multimodal large model with the accuracy of the expert system, optimizing the reasoning process to achieve fast and accurate detection results.
[0008] The technical solution of the present invention is as follows:
[0009] A deepfake content detection system based on a multimodal large model includes:
[0010] The visual feature extraction branch based on video Swin Transformer: This part takes video segments as input and uses a staged Video Swin Transformer as the backbone network; through 3D block partitioning, linear embedding and hierarchical processing in four stages, combined with block merging operation, it aims to extract multi-scale deep spatiotemporal visual features from video data.
[0011] Interactive audio feature extraction branch based on ERes2Net: This part takes Mel frequency cepstral coefficient segments as input, uses cascaded ERes2Net modules to construct an audio feature extraction network, introduces a multi-level fusion mechanism, and injects the extracted audio features into the visual feature extraction branch step by step to form fused features.
[0012] The audio-visual local-global interaction and detection module is used to perform the final processing of the fused features. First, the features are aggregated through parallel spatiotemporal pooling and temporal attention mechanisms. Then, through the core local-global interaction module, the intrinsic correlation of audio-visual features is deeply mined in three dimensions: local region, global region, and local-global. Finally, the deep forgery detection results are output through a fully connected layer.
[0013] A deepfake content detection method based on a multimodal large model includes:
[0014] Step S110, Visual feature extraction based on video Swin Transformer: Taking video segments as input, a staged Video Swin Transformer is used as the backbone network; through 3D block partitioning, linear embedding and hierarchical processing of four stages, combined with block merging operation, the aim is to extract multi-scale deep spatiotemporal visual features in video data.
[0015] Step S120: Interactive audio feature extraction based on ERes2Net: Using Mel frequency cepstral coefficient segments as input, an audio feature extraction network is constructed using cascaded ERes2Net modules; a multi-level fusion mechanism is introduced to inject the extracted audio features into the visual feature extraction branch step by step to form fused features;
[0016] Step S130: Audio-visual local-global interaction and detection: The fused features are processed in the final stage. First, the features are aggregated through parallel spatiotemporal pooling and temporal attention mechanisms. Then, through the core local-global interaction module, the intrinsic relationship between audio-visual features is deeply mined in three dimensions: local region, global region, and local-global. Finally, the deep forgery detection results are output through a fully connected layer.
[0017] A computing device includes: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device performs the method.
[0018] Beneficial effects:
[0019] (1) High reasoning accuracy
[0020] Figure 1 The dataset related to deepfakes is shown, quoted from "One Million Deepfake Audio and Video: A Large-Scale, Model-Driven Dataset for Deepfake Audio and Video".
[0021] Figure 2 The technical solution of the present invention achieves optimal results on AV-Deepfake1M, the industry's largest deepfake detection dataset, with an accuracy of 99.9998%.
[0022] (2) Fast reasoning speed
[0023] Based on user input, the multimodal large language model invokes the deepfake expert system. The expert model has only 46.3M weights, and a single inference operation takes less than 0.006 seconds on an RTX 3090 machine. Based on user input commands, it determines whether to activate multi-level description and interaction services, employing a layered design to accelerate inference and optimize resource consumption. Furthermore, the multimodal large language model can also perform inference on a single RTX 3090 consumer-grade graphics card with 8-bit quantization. Attached Figure Description
[0024] Figure 1 This demonstrates a performance comparison of various algorithms on a deepfake dataset.
[0025] Figure 2 This demonstrates that the technical solution of the present invention achieves optimal results on the industry's largest deepfake detection dataset;
[0026] Figure 3 A structural diagram of the deepfake content detection system based on a multimodal large model according to the present invention is shown.
[0027] Figure 4 This illustrates the mechanisms of global attention and local attention.
[0028] Figure 5 This illustrates a login interface according to an example of the technical solution of the present invention;
[0029] Figure 6 A screenshot of the main page in an example of the technical solution according to the present invention is shown;
[0030] Figure 7 A screenshot of the upload interface in an example of the technical solution according to the present invention is shown;
[0031] Figure 8 A screenshot showing the display effect in an example of the technical solution according to the present invention is shown. Detailed Implementation
[0032] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0033] While traditional self-attention and cross-modal self-attention can capture global information, they still have shortcomings in processing local details. The AV-DeepFake1M challenge generates video modalities by modifying the lip region, making the capture of fine-grained information in the lip region crucial. Furthermore, traditional attention mechanisms ignore fine-grained intermodal relationships during intermodal interactions, which is detrimental to audiovisual DeepFake tasks.
[0034] To address this problem, this invention proposes a deepfake content detection system based on a multimodal large model. Figure 3 A structural diagram of the deepfake content detection system based on a multimodal large model according to the present invention is shown. Figure 3 As shown, the system includes:
[0035] The visual feature extraction branch based on video Swin Transformer: This part takes video segments as input and uses a staged Video Swin Transformer as the backbone network. Through 3D block partitioning, linear embedding, and hierarchical processing in four stages, combined with block merging operations, it aims to extract multi-scale deep spatiotemporal visual features from video data.
[0036] The interactive audio feature extraction branch based on ERes2Net: This part takes MFCC (Mel-frequency cepstral coefficients) segments as input and uses cascaded ERes2Net modules to construct an audio feature extraction network. The key is the introduction of a multi-level fusion mechanism, which injects the extracted audio features into the visual feature extraction branch step by step to form fused features, realizing early interaction and complementarity between audio and video modalities in the feature extraction stage.
[0037] The Audio-Visual Local-Global Interaction (AV-LGI) and Detection Module is used for the final processing of the fused features. First, features are aggregated through parallel spatiotemporal pooling and temporal attention mechanisms. Then, the core AV-LGI module deeply mines the intrinsic correlations of audio-visual features in three dimensions: local region, global region, and local-global. Finally, the deep forgery detection results are output through a fully connected layer.
[0038] In one embodiment, the system further includes a modeling module, which may include:
[0039] A time-aware frame encoder is used to capture frame and timestamp information in a video.
[0040] A sliding feature extractor is used to capture inter-frame temporal information and simultaneously achieve modal alignment between language, video, and audio.
[0041] In one implementation, such as Figure 4 As shown, the Audio-Visual Local-Global Interaction (AV-LGI) and Detection Module includes the Audio-Visual Local-Global Interaction Module (AV-LGI Module). This module mainly consists of three cascaded processing stages: interaction within a local region (S1), interaction between global regions (S2), and region-to-global interaction (S3).
[0042] Specifically, the input features are first organized as A number of independent regions, in which any one of the... Each region ( ) contains local token sequences and the initial global token Based on this, the module executes the following processing flow:
[0043] Interaction within a local region (S1): This stage aims to extract fine-grained features within the region. First, the input local token ( ) and the global token of the initialization or previous stage ( The features are concatenated and fed into a self-attention mechanism to capture long-range dependencies within the region. To enhance the model's ability to perceive local details and introduce inductive bias, features are fed into parallel 1D convolutional layers after the self-attention mechanism. These 1D convolutional layers are configured to operate in the sequence dimension, extracting multi-scale local contextual information through different convolutional kernels, and finally outputting updated local and global tokens. ).
[0044] Global Inter-Region Interaction (S2): This stage aims to establish connections between different regions. The system extracts global tokens from the output of stage S1 for all regions, constructing a global token sequence. These tokens are input into a global self-attention mechanism, enabling the global representation of each region to interact with the global representations of all other regions, thereby fusing the global semantic information of the entire video / audio segment and generating an updated global token set. ).
[0045] Region-Global Interaction (S3): This stage aims to inject global context back into local features. This module employs a cross-attention mechanism, where the global token ( ) as the key and value, local tokens for each region ( This module acts as a query (or an interaction between the two) to guide local features with global information. Structurally, this stage includes two independent feedforward neural network (FFN) paths: a global path where the global token undergoes feature transformation and nonlinear mapping through an FFN layer; and a local path where local features, after cross-attention fusion, enter another independent FFN layer for final feature integration and output. Through this design, the module effectively completes the full feature evolution process from local perception to global aggregation and back to local enhancement.
[0046] Specifically, local self-attention is used to divide the video and audio tensors into local regions, and a self-attention mechanism is applied within each region. Applying self-attention within each region can include: adding a representative term within each region and applying the local self-attention mechanism to it. The representative term compresses the local region features through the self-attention mechanism.
[0047] Global inter-region self-attention is used to extract representative tokens from video and audio after local self-attention calculation, and then perform global inter-region self-attention calculation to achieve information exchange between audiovisual modalities.
[0048] Local-global interaction is used to achieve fine-grained interaction between visual and audio modalities based on cross-attention mechanisms after self-attention between global regions.
[0049] The above mechanisms can effectively integrate information from video and audio modalities, improving the performance of multimodal tasks.
[0050] The training dataset for this system is constructed in two steps:
[0051] 1) Instruction construction and 2) Response construction. Instruction construction refers to manually designing prompt word templates for different tasks, then enriching the expressions based on an AI language model, followed by manual selection, ultimately resulting in 6 templates for each task. The AI language model can be GPT-4.
[0052] According to embodiments of the present invention, a deepfake content detection method based on a multimodal large model is also provided. This method is based on an Audio-Visual Local-Global Neural Network (AV-LGNN). The method includes:
[0053] Step S110, Visual Feature Extraction Based on Video Swin Transformer: Taking video segments as input, a staged Video Swin Transformer is used as the backbone network. Through 3D block partitioning, linear embedding, and hierarchical processing in four stages, combined with block merging operations, the aim is to extract multi-scale deep spatiotemporal visual features from the video data.
[0054] Step S120: Interactive audio feature extraction based on ERes2Net: Using MFCC (Mel-frequency cepstral coefficients) segments as input, an audio feature extraction network is constructed using cascaded ERes2Net modules. The key is the introduction of a multi-level fusion mechanism, which injects the extracted audio features into the visual feature extraction branch step by step to form fused features, realizing early interaction and complementarity between audio and video modalities in the feature extraction stage.
[0055] Step S130: Audio-visual Local-Global Interaction (AV-LGI) and Detection: The fused features are then processed. First, features are aggregated through parallel spatiotemporal pooling and temporal attention mechanisms. Then, the core AV-LGI module is used to deeply mine the intrinsic correlations between audio-visual features in three dimensions: local region, global region, and local-global. Finally, the deep forgery detection results are output through a fully connected layer.
[0056] Step S130 may include the following sub-steps:
[0057] Step a: Divide the video and audio tensors into local regions using self-attention within each region, and apply self-attention within each region.
[0058] Step b: After local self-attention calculation, extract representative tokens of video and audio through global inter-region self-attention, perform global inter-region self-attention calculation, and realize information exchange between audiovisual modalities.
[0059] Step c: After self-attention between global regions, fine-grained interaction between visual and audio modalities is achieved through local-global interaction based on the cross-attention mechanism.
[0060] In one embodiment, the method may further include: modeling the video using two modules:
[0061] 1) A time-aware frame encoder, used to capture frame and timestamp information in video;
[0062] 2) Sliding feature extractor, used to capture inter-frame temporal information and simultaneously achieve modal alignment between language, video, and audio.
[0063] The following is combined with Figures 5 to 8 Description 1: Specific example:
[0064] like Figure 5 As shown, first open your browser and visit the website homepage: http: / / 47.96.228.3 / , enter your username and password, and click the "Login" button.
[0065] After successful login, you will be redirected to the main page, such as... Figure 6 As shown. The main page contains two main functional areas:
[0066] 1. Video preview window (left side)
[0067] 2. Output results window (right side)
[0068] Upload the video and test it, such as Figure 7 As shown:
[0069] 1. On the main page, the left window is used to upload and play videos. You can find the "Upload Video" button at the bottom of the left window:
[0070] 2. Click the "Upload Video" button, and the system will open a file selection dialog box.
[0071] 3. Select the video file you want to upload, and then click "Open".
[0072] 4. After selecting a video, the video will automatically play a preview in the left window and begin detection.
[0073] The output results show:
[0074] 1. After uploading and previewing the video, the system will display the processing results in the right-hand window.
[0075] 2. The processing results will show whether the video is fake and the fake range.
[0076] Figure 8 A screenshot showing the display effect is provided.
[0077] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of explaining or limiting the subject matter of the invention.
Claims
1. A deepfake content detection system based on a multimodal large model, characterized in that, include: The visual feature extraction branch based on video Swin Transformer: This part takes video segments as input and uses a staged Video Swin Transformer as the backbone network; through 3D block partitioning, linear embedding and hierarchical processing in four stages, combined with block merging operation, it aims to extract multi-scale deep spatiotemporal visual features from video data. Interactive audio feature extraction branch based on ERes2Net: This part takes Mel frequency cepstral coefficient segments as input, uses cascaded ERes2Net modules to construct an audio feature extraction network, introduces a multi-level fusion mechanism, and injects the extracted audio features into the visual feature extraction branch step by step to form fused features. The audio-visual local-global interaction and detection module is used to perform the final processing of the fused features. First, the features are aggregated through parallel spatiotemporal pooling and temporal attention mechanisms. Then, through the core local-global interaction module, the intrinsic correlation of audio-visual features is deeply mined in three dimensions: local region, global region, and local-global. Finally, the deep forgery detection results are output through a fully connected layer.
2. The deepfake content detection system based on a multimodal large model according to claim 1, characterized in that, It also includes a modeling module, which includes: A time-aware frame encoder is used to capture frame and timestamp information in a video. A sliding feature extractor is used to capture inter-frame temporal information and simultaneously achieve modal alignment between language, video, and audio.
3. The deepfake content detection system based on a multimodal large model according to claim 1, characterized in that, The audio-visual local-global interaction and detection module includes: the audio-visual local-global interaction module, which includes: self-attention within local regions, self-attention between global regions, and local-global interaction.
4. The deepfake content detection system based on a multimodal large model according to claim 3, characterized in that, Self-attention within local regions is used to divide video and audio tensors into local regions and apply a self-attention mechanism within each region. Global inter-region self-attention is used to extract representative tokens from video and audio after local self-attention calculation, and then perform global inter-region self-attention calculation to realize information exchange between audiovisual modalities. Local-global interaction is used to achieve fine-grained interaction between visual and audio modalities based on cross-attention mechanisms after self-attention between global regions.
5. The deepfake content detection system based on a multimodal large model according to claim 4, characterized in that, The use of self-attention mechanism within each region includes: adding a representative word in each region and applying a local self-attention mechanism to it; the representative word compresses local region features through the self-attention mechanism.
6. The deepfake content detection system based on a multimodal large model according to claim 1, characterized in that, The training dataset for this system is constructed in two steps: instruction construction and response construction. Instruction construction refers to manually designing prompt word templates for different tasks, enriching the expression based on artificial intelligence language models, and then manually filtering them, resulting in 6 templates for each task.
7. A method for detecting deepfake content based on a multimodal large model, characterized in that, include: Step S110, Visual feature extraction based on video Swin Transformer: Taking video segments as input, a staged Video Swin Transformer is used as the backbone network; through 3D block partitioning, linear embedding and hierarchical processing of four stages, combined with block merging operation, the aim is to extract multi-scale deep spatiotemporal visual features in video data. Step S120: Interactive audio feature extraction based on ERes2Net: Using Mel frequency cepstral coefficient segments as input, an audio feature extraction network is constructed using cascaded ERes2Net modules; a multi-level fusion mechanism is introduced to inject the extracted audio features into the visual feature extraction branch step by step to form fused features; Step S130: Audio-visual local-global interaction and detection: The fused features are processed in the final stage. First, the features are aggregated through parallel spatiotemporal pooling and temporal attention mechanisms. Then, through the core local-global interaction module, the intrinsic relationship between audio-visual features is deeply mined in three dimensions: local region, global region, and local-global. Finally, the deep forgery detection results are output through a fully connected layer.
8. The deepfake content detection method based on a multimodal large model according to claim 7, characterized in that, Step S130 includes the following sub-steps: Step a: Divide the video and audio tensors into local regions using self-attention within each region, and apply self-attention within each region. Step b: After local self-attention calculation, extract representative tokens of video and audio through global inter-region self-attention, perform global inter-region self-attention calculation, and realize information exchange between audiovisual modalities. Step c: After self-attention between global regions, fine-grained interaction between visual and audio modalities is achieved through local-global interaction based on the cross-attention mechanism.
9. The deepfake content detection method based on a multimodal large model according to claim 7, characterized in that, The method also includes: modeling the video using two modules: 1) A time-aware frame encoder, used to capture frame and timestamp information in video; 2) Sliding feature extractor, used to capture inter-frame temporal information and simultaneously achieve modal alignment between language, video, and audio.
10. A computing device, characterized in that, include: At least one processor and a memory storing program instructions; When the program instructions are read and executed by the processor, the computing device performs the method as described in any one of claims 7-9.