Deep forgery detection method and system based on audio and video

Abstract

The invention provides a deep forgery detection method and system based on audio and video, and belongs to the field of digital content evidence obtaining. The method aims to solve the problems that an existing method is insufficient in sensitivity to local artifacts, neglects the independent discrimination ability of an acoustic level, generally depends on a static fusion strategy, and is difficult to fully model the difference of real and forged contents in multiple scales of time and space. The method comprises the following steps: extracting visual features by using a CLIP model; a large model Whisper is adopted to extract semantic features of the audio, an acoustic feature encoder AFE is designed to obtain acoustic features, and the semantic features and the acoustic features are fused to obtain optimized semantic features; the method comprises the following steps of: establishing an MV-Block, and modeling a cross-modal feature by utilizing the capabilities of Mama and Vision Transform at the same time; a multi-modal fusion module MMFB is introduced, multi-modal features are integrated, and cross-modal interaction alignment is promoted; and optimizing the network by using the loss function.

Description

Technical Field

[0001] This invention relates to the field of digital content forensics technology, and more specifically, to a method and system for detecting deepfakes based on audio and video. Background Technology

[0002] Against the backdrop of accelerated globalization and digital transformation, and the deep integration of social media, social networks have become the main battleground for information dissemination. Video, with its high spatiotemporal resolution and powerful expressiveness, has secured its dominant position as the mainstream media on these platforms. However, videos contain sensitive information such as facial and behavioral details, leading to significant privacy breaches and security risks in the digital age. With the widespread adoption of various open-source tools and applications, the barrier to entry for creating fake content has been significantly lowered, allowing even non-professional users to quickly generate high-quality fake videos. In this context, deepfake detection technology targeting AI-generated edits has become a crucial line of defense for protecting digital content.

[0003] In audio-visual-based deepfake video detection research, many works have leveraged supervised contrastive learning to characterize the correspondence between audio and visual features, demonstrating some success in cross-modal association modeling and overall performance. However, existing multimodal approaches still suffer from several key shortcomings, limiting their applicability and robustness in complex real-world scenarios. First, current methods often focus on shallow alignment between audio and visual elements, frequently resulting in coarse alignment and averaging, leading to insufficient sensitivity to local artifacts. Second, the importance of the acoustic dimension has not been fully recognized: audio manipulation not only disrupts linguistic coherence but also distorts acoustic consistency. Many methods compress audio into a single global vector or use it merely as an auxiliary signal for the visual branch, neglecting the independent discriminative power at the acoustic level. Finally, these methods generally rely on static fusion strategies, making it difficult to fully model the differences between real and fake content across multiple temporal and spatial scales. Therefore, researching how to better utilize intramodal information and achieve cross-modal deep alignment, strengthening intramodal and intermodal consistency, is a key path to improving the effectiveness of deepfake detection models. Summary of the Invention

[0004] The technical problem to be solved by this invention is:

[0005] To address the shortcomings of existing deepfake detection methods in audio and video, such as insufficient sensitivity to local artifacts, neglect of independent acoustic discrimination, and reliance on static fusion strategies which make it difficult to fully model the differences between real and fake content across multiple temporal and spatial scales.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] This invention provides a method for detecting deepfakes based on audio and video, comprising the following steps:

[0008] S100: An image encoder using the CLIP contrastive language-image pre-trained model extracts visual features from video frames;

[0009] S200. Extract the audio Mel spectrogram from the video, input it into the Whisper model to encode semantic features, design an AFE acoustic feature encoder to capture latent acoustic properties with multi-scale convolution and attention mechanisms, and fuse the acoustic features and semantic features to obtain the overall audio features.

[0010] S300. Construct the MV-Block component, and at the same time utilize the capabilities of Mamba and Vision Transformer to efficiently model cross-modal features for visual features and overall audio features respectively; the MV-Block includes a LayerNorm, a Mamba Block, another LayerNorm and a Vision Block connected in sequence. The Mamba Block is used to process feature maps within the channel, and the Vision Block is used to perform information interaction between channels.

[0011] S400. Introducing the MMFB multimodal fusion module to fuse information from different modalities; including shallow interaction and deep feature fusion of inputs from visual and semantic modalities. The MMFB multimodal fusion module is used to establish two feature processing paths, namely the main path and the auxiliary path. The main path is used to enhance the interaction between features and retain key information, while the auxiliary path is used to assign attention weights to each channel. The main path and the auxiliary path are combined to obtain the final fused feature representation.

[0012] S500: Use the real labels to supervise the final output, that is, reduce the difference between the predicted output and the real labels by minimizing the loss function.

[0013] Further, in step S100, the extracted visual features are: Where C is the first dimension of the two-dimensional feature; and L is the second dimension of the two-dimensional feature.

[0014] Further, in step S200, the audio Mel-spectrum is... The semantic features obtained are Where H represents the first dimension of the extracted Mel map; W represents the second dimension of the extracted Mel map.

[0015] Further, in step S200, when obtaining the optimized semantic features, low-level acoustic features are first extracted through convolutional and linear layers, followed by embedding: tokens are added to the sequence, and positional embeddings are introduced to capture the temporal structure; The input is fed into the AFE acoustic feature encoder to obtain the acoustic features. .

[0016] Further, in step S300, in the Mamba Block, the input features first pass through the normalization layer Norm(·), then enter the Mamba unit Mamba(·), then pass through another normalization layer Norm(·), and finally pass through the multilayer perceptron Mlp(·); the specific calculation is as follows:

[0017]

[0018] in, , and These represent the features input to the Mamba Block, the intermediate features processed by the Mamba Block, and the output features of the Mamba Block, respectively.

[0019] Further, in step S300, within the Vision Block, the Vision Transformer consists of three information flow paths: query Q, key K, and value V; the attention score is obtained by taking the dot product of Q and K, and its attention function is defined as:

[0020]

[0021] in, d represents the dimension of the Q, K, and V vectors.

[0022] Further, in step S400,

[0023] During shallow interaction, the image features processed by the cross-attention module are added to the original audio features, and the audio features are added to the original image features. The cross-attention module calculates the similarity between the visual features and the audio features, and weights the cross-modal features accordingly to generate a new representation. Subsequently, the audio and visual features are integrated, and feature fusion is completed in high-dimensional space through convolutional layers and Batch Norm layers.

[0024] When performing deep feature fusion, two independent MV-Blocks are introduced to process audio features and image features respectively. After convolutional layers, the feature shape remains consistent with the initial encoded features. The MV-Blocks process features both within and between channels to obtain deeper feature representations. A sigmoid-based gating mechanism assigns weights to each feature channel, and element-wise addition and matrix multiplication are used to learn interactive information between different modal features.

[0025] Furthermore, in step S400, for the MLFB multi-level fusion module,

[0026] In the main path, all features are concatenated along the channel dimension, and then the number of channels is reduced by a convolutional layer. Subsequently, a channel attention mechanism is introduced to enhance the interaction between features. Then, features from the final layer are added to the main path to optimize the fusion process. At the end of the main path, a sigmoid-based gating mechanism is used to highlight the most relevant features, thereby enhancing the model's ability to selectively retain key information.

[0027] In the auxiliary path, attention weights are also assigned to each channel; finally, the final fused feature representation is obtained by matrix multiplication of the main path and the auxiliary path, combined with the skip connections of the final layer features.

[0028] Further, in step S500, it is set that... and These represent the final output and the true label, respectively; total loss. The definition is as follows:

[0029]

[0030] A deepfake detection system based on audio and video, the system having program modules corresponding to the steps described above, and executing the steps in the above-described deepfake detection method based on audio and video when running.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] This invention effectively decouples audio semantic features from acoustic features, emphasizing the semantic-acoustic duality of audio. Audio tampering not only disrupts linguistic consistency but also creates acoustic inconsistencies. This invention successfully captures subtle traces of audio tampering, improving accuracy. Furthermore, while effectively eliminating redundant information, this invention utilizes diverse discriminative features to integrate multimodal information and promote cross-modal interaction alignment. Attached Figure Description

[0033] Figure 1 This is a flowchart of a deepfake detection method based on audio and video in an embodiment of the present invention;

[0034] Figure 2 This is a Class Activation Map (CAM) generated from intermediate features before the classifier input in an embodiment of the present invention. Detailed Implementation

[0035] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] Specific Implementation Plan 1: Combining Figure 1 As shown, this invention provides a deepfake detection method based on audio and video, comprising the following steps:

[0037] S100. For visual modalities, CLIP (Contrastive Language-Image Pretraining) is used to extract features from video frames.

[0038] CLIP, a widely used multimodal model, can process both image and text information simultaneously. It comprises two cooperating encoders: one for image processing and the other for text processing. To extract visual features, this invention utilizes CLIP's image encoder. Thanks to pre-training on massive amounts of image data, it can extract more abstract visual representations and exhibits good generalization ability. Furthermore, CLIP's image and text encoders are jointly trained within a multimodal framework, enabling it to capture rich cross-modal correspondences, thus facilitating effective interaction with audio features. The visual features extracted in this invention... This can be used in subsequent audio and video task processing workflows; where C is the first dimension of the two-dimensional feature; L is the second dimension of the two-dimensional feature; the actual value is 50×768;

[0039] S200 uses the large-scale Whisper model to extract semantic features from audio; Whisper is an Automatic Speech Recognition (ASR) system.

[0040] To effectively capture semantic and acoustic features, this invention proposes two independent encoders. Traditional audio deep artifact detection methods mainly focus on anomalies in semantic information, but often ignore acoustic artifacts left by audio manipulation. Audio tampering not only destroys language consistency, but also causes inconsistencies at the acoustic level, making it difficult for many methods to capture subtle artifacts, thereby reducing detection accuracy.

[0041] Because Whisper is specifically designed for speech recognition and transcription and trained on large-scale internet audio-text pairs, it can accurately capture the language content and context in audio, exhibits good generalization ability in multilingual scenarios, and is highly robust to background noise and distortion; this invention uses audio Mel-spectral mapping... Input into the Whisper model to obtain semantic features Where H represents the first dimension of the extracted Mel image; W represents the second dimension of the extracted Mel image; the actual size is 224×224.

[0042] Simultaneously, this invention designs an Acoustic Feature Encoder (AFE) to capture latent acoustic properties using multi-scale convolution and attention mechanisms. Specifically, it first extracts low-level acoustic features through convolutional and linear layers, then performs embedding: special tokens are added to the sequence, and positional embedding is introduced to capture the temporal structure; Input into AFE to obtain acoustic features ;

[0043] Finally, acoustic features and semantic features are fused together to combine acoustic attributes and semantic content, forming a more comprehensive representation of the audio input, thereby improving the robustness and accuracy of subsequent forgery detection.

[0044] S300. As a basic component of the CWCNet (CLIP and Whisper Cooperation Network) proposed in this invention, MV-Block utilizes the capabilities of both Mamba and Vision Transformer to efficiently model cross-modal features.

[0045] Each MV-Block consists of a LayerNorm, a Mamba Block, another LayerNorm, and a Vision Block, which are connected in sequence.

[0046] The architecture of the Mamba Block: Input features first pass through a normalization layer Norm(·), then into a Mamba unit Mamba(·), then through another normalization layer Norm(·), and finally through a multilayer perceptron Mlp(·); its specific computation can be described as follows:

[0047]

[0048] in, , and These represent the features input to the Mamba module, the intermediate features processed by the Mamba unit, and the output features of the Mamba Block module, respectively.

[0049] Given that the Mamba Block primarily processes feature maps within a single channel, achieving efficient information exchange across multiple channels is crucial. Inspired by the traditional Transformer architecture, this invention introduces the Vision Block to meet this requirement. The Vision Transformer consists of three information flow paths: query (Q), key (K), and value (V). The attention score is obtained by performing a dot product on Q and K, and its attention function is defined as:

[0050]

[0051] in d represents the dimensions of the Q, K, and V vectors;

[0052] Thus, the Mamba-Vision Block can facilitate information exchange between channels, thereby enabling a more comprehensive fusion of different modal information within the spatial domain;

[0053] S400. In order to fuse information from different modalities, this invention introduces a multi-modal fusion block (MMFB), which can effectively integrate multimodal features and promote cross-modal interaction alignment.

[0054] include,

[0055] First, a shallow interaction is performed on the inputs from the two modalities: specifically, the image features processed by the cross-attention module are added to the original audio features, and the audio features undergo the same processing; the cross-attention module calculates the similarity between the visual and audio features and weights the cross-modal features accordingly to generate a new representation; this module enables the model to extract additional information related to the image content from the audio and achieve mutual transfer between image and audio information; subsequently, the audio and visual features are integrated, and feature fusion is completed in high-dimensional space through convolutional layers and BatchNorm layers;

[0056] To further achieve deep feature fusion, this invention introduces two independent MV-Blocks after the convolutional layer to process audio and image features respectively. After the convolutional layer, the feature shape remains consistent with the initial encoded features. The MV-Blocks process features both intra-channel and inter-channel, thereby obtaining a deeper feature representation. Next, a sigmoid-based gating mechanism assigns weights to each feature channel, enabling the model to dynamically learn which features should be emphasized and which should be suppressed or discarded, thus improving the overall feature fusion effect. Finally, element-wise addition and matrix multiplication are used to learn the interactive information between different modal features.

[0057] To fully utilize feature information from different levels, a Multi-Level Fusion Block (MLFB) is introduced. The input consists of cross-modal fused features from different levels. Two feature processing paths are established: a main path and an auxiliary path. In the main path, all features are first concatenated along the channel dimension, and then the number of channels is reduced through convolutional layers. Subsequently, a channel attention mechanism is introduced to enhance the interaction between features. Afterward, features from the final layer are added to the main path to further optimize the fusion process. At the end of the main path, a sigmoid-based gating mechanism is used to highlight the most relevant features, thereby enhancing the model's ability to selectively retain key information.

[0058] In the auxiliary path, the CA mechanism is also applied to assign attention weights to each channel; finally, the final fused feature representation is obtained by matrix multiplication of the main path and the auxiliary path, combined with the skip connections of the final layer features.

[0059] S500, To optimize CWCNet, real labels are used to supervise the final output; let... and These represent the final output and the true label, respectively; total loss. The definition is as follows:

[0060]

[0061] This loss function is equivalent to binary cross-entropy with logits, which optimizes the network by minimizing the difference between the predicted output and the true label.

[0062] Specific implementation scheme two: The present invention provides a deepfake detection system based on audio and video, which has a program module corresponding to the above steps, and executes the steps in the above-mentioned deepfake detection method based on audio and video when running.

[0063] Simulation Experiment

[0064] To evaluate the effectiveness of the proposed audio / video deepfake detection method, comparative experiments were conducted using the FakeAVceleb and FakeTIMIT datasets. The effectiveness of this method was also compared with classic unimodal and multimodal methods to further validate its effectiveness.

[0065] These comparison methods can be divided into three categories: visual (V), audio (A), and audiovisual (AV). Visual deepfake detection methods include: CVIT (Deepfake video detection using convolutional visiontransformer), FTCN (Exploring temporalcoherence for more general video face forgery detection), RF (Leveraging real talking faces via self-supervision for robust forgery detection), and NPR (Rethinking the up-sampling operations in cnn-based generative network for generalizable deepfake detection). Audio methods include: RawNet (Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification) and Ecapa (Emphasized channel attention, propagation and aggregation in tdnn-based speaker). Verification based on delay neural networks emphasizes channel attention, propagation, and aggregation in speaker verification; AASIST (Source: Attention-based Cascaded Capsule Network for Audio Deepfake Detection); Caps (Source: Attention-based Cascaded Capsule Network for Audio Deepfake Detection).Advanced audio-visual joint methods include: AVFakeNet (Source: A unified end-to-end Dense Swin Transformer deep learning model for audio-visual deepfakes detection), AVDD (Source: Joint audio-visual deepfake detection), AVOID (Source: Audio-Visual Joint Learning for Detecting Deepfake), and AVFF (Source: Audio-Visual Feature Fusion for Video Deepfake Detection).

[0066] Commonly used evaluation metrics for prediction results include accuracy (ACC), average precision (AP), area under the curve (AUC), and F1 score. The comparative experimental results are shown in Tables 1 and 2 below (bold indicates best performance, and underlined indicates second-best performance):

[0067] Table 1 Summary of Evaluation Indicators (Accuracy, Average Precision, and Area Under the Curve)

[0068]

[0069] As shown in Table 1, compared to single-modal methods, the present invention achieves the best performance in all cases. This indicates that the proposed method can accurately detect deepfake videos involving multimodal manipulation. This capability enables the method to be applied to a wide range of application scenarios. Furthermore, the present invention outperforms other competing methods in terms of ACC, AP, and AUC metrics. The method demonstrates superior performance, revealing its great potential for practical applications in real-world scenarios. Compared to other multimodal methods, the present method exhibits superior performance, particularly on RAFV. This demonstrates the method's effectiveness in addressing the challenges of complex forgery detection and its robustness in handling complex manipulations. Moreover, the results clearly demonstrate the superior performance of utilizing cross-modal corresponding audiovisual methods, thereby significantly improving detection metrics. Therefore, the present invention achieves state-of-the-art performance compared to existing multimodal methods.

[0070] Table 2 Summary of Cross-Dataset Testing

[0071]

[0072] As shown in Table 2, the AVFakeNet and AVDD methods fail to detect forgeries in other cases due to their significantly poor accuracy. Furthermore, the detection performance of AVOID and AVFF deteriorates significantly. Although this invention also experiences some performance degradation, it remains effective in certain situations, demonstrating its ability to handle unknown forgeries.

[0073] Ablation experiments were conducted on the key module AFE of the invention. The first model (denoted as "w / o AFE") does not include the AFE component; the second model (denoted as "w AFE") integrates the proposed AFE. The results are shown in Table 3 below:

[0074] Table 3 Summary of Evaluation Indicators for Ablation Experiments

[0075]

[0076] As shown in Table 3, the necessity of AFE in effectively extracting acoustic features from audio is highlighted. Furthermore, the ablation results emphasize the effectiveness of segmenting acoustic and semantic features, providing fine-grained audio features crucial for subsequent cross-modal fusion processes.

[0077] To verify its effectiveness, four structural variants were constructed: the first was "w / o Fusion," where all fusion modules were replaced with simple feature concatenation followed by convolutional transformation; the second and third were "w / o MMFB" and "w / o MLFB," respectively, where only the corresponding single module was replaced; the final model adopted a multi-level fusion strategy. The ablation experiment results are shown in Table 4 below:

[0078] Table 4 Summary of Ablation Experiment Results

[0079]

[0080] As shown in Table 4, the results reveal a significant decrease in FARV performance when the fusion block is removed. Similarly, removing the MMFB or MLFB modules also leads to a significant drop in detection performance, indicating that each module contributes to the overall effectiveness of the model. This strongly demonstrates the effectiveness of the fusion strategy in aligning audio and image information, accurately identifying inconsistencies between cross-modal features.

[0081] This invention generates class activation maps (CAMs) on intermediate features before classifier input to reveal the decision-making basis of CWCNet. For real faces, feature embedding focuses more on the harmony of the global structure, highlighting symmetry and overall consistency; for fake faces, the response is concentrated in easily tampered local regions, as shown in the visualization results. Figure 2As shown. By comparing with various verification indicators, the method proposed in this invention can obtain more accurate deepfake detection results.

[0082] Specific implementation scheme two: The present invention provides a deepfake detection system based on audio and video, which has a program module corresponding to the above steps, and executes the steps in the above-mentioned deepfake detection method based on audio and video when running.

[0083] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.

[0084] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. An audio-video based deepfake detection method, characterized in that, The method comprises the following steps: S100, extracting visual features from video frames by using an image encoder of a CLIP contrastive language-image pre-training model; S200, extracting an audio mel-spectrogram in the video, inputting a Whisper model to encode to obtain semantic features, designing an AFE acoustic feature encoder to capture potential acoustic properties through multi-scale convolution and attention mechanism, and fusing the acoustic features and the semantic features to obtain overall audio features; S300, constructing an MV-Block component, and simultaneously using the capabilities of Mamba and Vision Transformer to efficiently model cross-modal features of the visual features and the overall audio features; the MV-Block comprises a LayerNorm layer, a Mamba Block, another LayerNorm layer, and a Vision Block connected in sequence, the Mamba Block is used to process feature maps within channels, and the Vision Block is used for information interaction between channels; S400, introducing an MMFB multi-modal fusion module to fuse information of different modalities; S400, introducing an MMFB multi-modal fusion module to fuse information of different modalities; S400, introducing an MMFB multi-modal fusion module to fuse information of different modalities; 2. The audio-video based deepfake detection method of claim 1, wherein: In step S100, the extracted visual features are ; wherein C is a first dimension of the two-dimensional feature; and L is a second dimension of the two-dimensional feature.

3. The audio-video based deepfake detection method of claim 2, wherein: In step S200, the audio mel-spectrogram is , and the obtained semantic feature is ; wherein H represents the first dimension size of the extracted mel-spectrogram; and W represents the second dimension size of the extracted mel-spectrogram.

4. The audio-video based deepfake detection method of claim 3, wherein: In step S200, when the optimized semantic features are obtained, first, low-level acoustic features are extracted through a convolutional layer and a linear layer, and then embedding is performed: tokens are added in the sequence, and position embedding is introduced to capture the time structure; and The input is input to the AFE acoustic feature encoder to obtain acoustic features .

5. The audio-video based deepfake detection method of claim 4, wherein: S500, using real labels to supervise the final output, that is, reducing the difference between the predicted output and the real labels by minimizing the loss function. wherein, , and respectively represent the features input to the Mamba Block, the intermediate features after processing by the Mamba Block, and the output features of the Mamba Block.

6. The audio-video based deepfake detection method of claim 5, wherein: In step S300, in the Mamba Block, the input features first pass through the normalization layer Norm(·), then enter the Mamba unit Mamba(·), then pass through another normalization layer Norm(·), and finally pass through the multi-layer perceptron Mlp(·); the specific calculation expression is: wherein, : d is the dimension of the Q, K, V vectors.

7. The audio-video based deepfake detection method of claim 6, wherein: In step S300, in the Vision Block, the Vision Transformer is composed of three information flow paths: query Q, key K and value V; the attention score is obtained by dot product of Q and K, and the attention function is defined as: In step S400, In the shallow interaction, the image features processed by the cross-attention module are added to the original audio features, and the audio features are added to the original image features; The cross-attention module calculates the similarity between the visual features and the audio features, and weights the cross-modal features according to the similarity, thereby generating new representations; then, the audio and visual features are integrated, and the feature fusion is completed in a high-dimensional space through a convolution layer and a Batch Norm layer; The cross-attention module calculates the similarity between the visual features and the audio features, and weights the cross-modal features according to the similarity, thereby generating new representations; then, the audio and visual features are integrated, and the feature fusion is completed in a high-dimensional space through a convolution layer and a Batch Norm layer; In the deep feature fusion process, two independent MV-Blocks are introduced to process audio features and image features respectively. After the convolution layer, the feature shape remains consistent with the initial encoding feature. The MV-Block processes the features within and between channels to obtain a deeper feature representation. The Sigmoid-based gating mechanism assigns weights to each feature channel, and the element-wise addition and matrix multiplication realize the learning of interactive information between different modal features.

8. The audio-video based deepfake detection method of claim 7, wherein: In step S400, for the MLFB multi-level fusion module, In the main path, all features are concatenated along the channel dimension, and then the convolution layer is used to reduce the number of channels. Subsequently, a channel attention mechanism is introduced to enhance the interaction between features. Then, the features from the final layer are added to the main path to optimize the fusion process. At the end of the main path, a Sigmoid-based gating mechanism is used to highlight the most relevant features, thereby enhancing the model's selective retention ability for key information. In the auxiliary path, attention weights are also assigned to each channel. Finally, through matrix multiplication operations of the main path and the auxiliary path, and combined with the jump connection of the final layer features, the final fused feature representation is obtained.

9. The audio-video based deepfake detection method of claim 8, wherein: In step S500, set With respectively represent the final output and the true label; Total loss Is defined as follows:

10. An audio-visual based deepfake detection system, characterized in that: The system has program modules corresponding to the steps of any one of claims 1-9, and when running, it executes the steps of the above-mentioned deep audio-video-based forgery detection method.

Images