Deep pseudo video detection method based on multi-modal contrast learning

Abstract

The invention relates to a deep pseudo video detection method based on multi-modal comparative learning, and is suitable for the field of deep counterfeit detection. The method comprises the following steps: inputting a to-be-detected video into a trained video authentic identification model, and outputting a detection result of the to-be-detected video; the video authentic identification model comprises a visual encoder which is taken from a trained audio and video contrast learning model and is used for extracting visual features of a face sequence in a to-be-detected video; the audio encoder is taken from the trained audio and video comparison learning model and is used for extracting audio features of the to-be-detected video; the cross-modal feature fusion module is used for fusing the visual features extracted by the visual encoder and the audio features extracted by the audio encoder to obtain a feature fusion result; the spatio-temporal feature extraction module is used for extracting spatio-temporal features from the feature fusion result; the fusion features not only comprise visual features, but also comprise audio features, so that the features are richer; and the classifier is used for classifying the to-be-detected video as a real video or a forged video based on the spatial-temporal characteristics.

Description

Technical Field

[0001] The present invention relates to a deep fake video detection method based on multimodal contrast learning, which is applicable to the field of deep fake detection. Background Art

[0002] There are currently several main methods for deep fake detection of videos: A frame-based deepfake video detection method first extracts the video into image frames, then uses an image-based deepfake detection method to make authenticity judgments, and finally uses a certain strategy to fuse the detection results of each frame to obtain the video detection results. This method takes a single frame as input, ignores the temporal semantic relationship and global coherence between video frames, and only focuses on low-level features such as texture details. It is sensitive to interference and performs poorly on new types of forgeries.

[0003] The deep fake video detection method based on temporal coherence adds the time dimension through RNN or LSTM to directly detect the video. This method introduces temporal information, pays more attention to abnormal facial movements and expressions that are often shown in fake videos, and is more resistant to detection interference caused by compression or blur.

[0004] However, both of the above methods only use a single modality in the video, namely the visual modality, without considering the audio information in the video and without fully utilizing the difference information between vision and audio, resulting in low overall detection accuracy.

[0005] In recent years, research has begun on the joint learning of audiovisual information, by training relatively independent authentication modules for video and audio, and making decisions based on the correlation between the learned features. Although this method takes into account the multimodal characteristics of the video, it does not fully integrate and interact the two modal information, and the detection accuracy is not significantly improved. In addition, since the model directly trains the classifier on the deep fake video dataset, it lacks prior knowledge and is prone to overfitting. It achieves high accuracy on the same distribution data, but the performance plummets on videos generated using new operation methods, and the generalization ability across datasets is low. Summary of the invention

[0006] The technical problem to be solved in the present invention is: to provide a deep fake video detection method based on multimodal contrast learning in response to the above-mentioned problems.

[0007] The technical solution adopted by the present invention is: a deep fake video detection method based on multimodal contrast learning, characterized by comprising: Input the video to be detected into the trained video authentication model, and output the detection result of the video to be detected; The video authentication model includes: The visual encoder is taken from the trained audio and video contrast learning model and is used to extract the visual features of the face sequence in the video to be detected; The audio encoder is taken from the trained audio and video contrast learning model to extract the audio features of the video to be detected; A cross-modal feature fusion module, used to fuse the visual features extracted by the visual encoder and the audio features extracted by the audio encoder to obtain a feature fusion result; The spatiotemporal feature extraction module is used to extract spatiotemporal features from the feature fusion results; the fused features include not only visual features but also audio features, which are richer in features; The classifier is used to classify the video to be detected as a real video or a forged video based on spatiotemporal features.

[0008] The training of the audio and video comparison learning model includes: The real audio and video are used as positive samples, and the audio in the real audio and video is moved forward or backward by a random time as a negative sample; Separate the audio and video in the positive and negative samples into audio and video, perform face detection and tracking on the video to extract the face, and obtain the face sequence; The visual features of the face sequence are extracted through the visual encoder, and the audio features of the audio corresponding to the face sequence are extracted through the audio encoder; Based on visual and audio features, the contrast loss is calculated to reduce the distance between positive samples and increase the distance between negative samples.

[0009] The cross-modal feature fusion module is used to extract features from stage3 and stage4 of the visual encoder and audio encoder respectively for feature fusion; the deeper the network from stage1 to stage4, the larger the receptive field of the extracted features and the richer the semantics, but the underlying detail information may be missing; extracting features from stage3 and stage4 ensures the effective combination of high-level semantic information and low-level detail information.

[0010] The cross-modal feature fusion module includes: , , , in, To fusion features, For the The audio features output by each stage, For the The visual features output by each stage, is the dimension of the feature, is the weight coefficient of the audio feature, is the weight coefficient of the video feature; , , in, is the feature fusion result, is the weight coefficient of the fusion feature corresponding to the third stage.

[0011] The visual encoder adopts ResNet50 and uses 3D convolution; the audio encoder adopts ResNet18.

[0012] The training of the video counterfeit detection model includes: The visual encoder and the audio encoder are extracted from the trained audio-video contrastive learning model, and the parameters of the visual encoder and the audio encoder are frozen.

[0013] A storage medium stores a computer program executable by a processor, wherein the computer program implements the steps of the deep fake video detection method when executed.

[0014] A deep fake video detection device comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and the computer program implements the steps of the deep fake video detection method when executed.

[0015] The beneficial effect of the present invention is that the video encoder and the audio encoder in the present invention are taken from a trained audio and video contrast learning model, and audio and video contrast learning pre-training is performed on a large number of unlabeled real speaker videos, so that the model can pre-understand the static and dynamic features of the natural speaking face, thereby solving the generalization problem of the detection model.

[0016] The present invention utilizes a cross-modal feature fusion module to cross-fuse the visual features extracted by a video encoder and the audio features extracted by an audio encoder, and finds complementary information between the modalities through feature cross-fusion to solve the accuracy problem of the detection model.

[0017] The present invention utilizes a spatiotemporal feature extraction module to extract spatiotemporal features from feature fusion results, captures temporal semantic relationships through spatiotemporal features, understands changes between frames, and solves the anti-interference problem of the detection model.

[0018] The present invention uses a visual encoder, an audio encoder, a cross-modal feature fusion module, and a spatiotemporal feature extraction module in the video authentication model to enable the present invention to have higher accuracy, generalization, anti-interference ability, and universality, thereby improving the overall performance of deep fake video detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1Flowchart of a deep fake video detection method in an embodiment.

[0020] Figure 2 Schematic diagram of self-supervised pre-training for audio and video comparison in an embodiment.

[0021] Figure 3 4 is a structural block diagram of a cross-modal feature fusion module in an embodiment.

[0022] Figure 4 4 is a structural block diagram of the spatiotemporal feature extraction module in the embodiment. DETAILED DESCRIPTION

[0023] Example 1: Figure 1 As shown, this embodiment is a deep fake video detection method based on multimodal contrastive learning. The method uses a trained video fake identification model to detect the video to be detected. The video fake identification model includes a visual encoder, an audio encoder, a cross-modal feature fusion module, a spatiotemporal feature extraction module and a classifier.

[0024] In this example, the visual encoder and audio encoder are both taken from the trained audio and video contrast learning model. The visual encoder is used to extract the visual features of the face sequence in the video to be detected, and the audio encoder is used to extract the audio features of the video to be detected; the cross-modal feature fusion module is used to fuse the visual features and the audio features to obtain the feature fusion result; the spatiotemporal feature extraction module is used to extract spatiotemporal features from the feature fusion result; the classifier is used to classify the video to be detected as a real video or a fake video based on the spatiotemporal features.

[0025] In this embodiment, the audio and video contrast learning model adopts audio and video contrast learning self-supervised training, such as Figure 2 As shown, the specific steps of the training are as follows: 1) Construction of positive and negative sample pairs: The original audio and video pair is used as the positive sample, and the audio in the original audio and video is randomly moved forward or backward by 500ms to 2s as the negative sample; 2) Audio and video preprocessing: adjust the audio and video FPS to 25, crop them into 0.4s segments, separate the audio and video, perform face detection and tracking on the video to extract the face, and obtain face sequences, each with 10 frames; 3) Sample enhancement: randomly mask 50% of the face sequences in the positive and negative samples, and mask the audio at the corresponding time position. After removing the mask, new positive and negative samples are formed. 4) Training: Preprocess the audio and video in the positive and negative samples to obtain the face sequence and the audio corresponding to the face sequence in time. Input the face sequence and audio into the visual encoder and audio encoder respectively to obtain the corresponding visual features and audio features, calculate the contrast loss between the visual features and the audio features, reduce the distance between the positive samples, and increase the distance between the negative samples.

[0026] This embodiment introduces a self-supervised contrastive learning method into the deep fake video detection task, and randomly masks some audio and video frames during contrastive learning. On the one hand, contrastive learning does not require manually annotated data sets, and audio and video pre-training is performed directly on a large number of unlabeled real speaker videos to mine high-level semantic information of audio and video, and pre-understand the static and dynamic features of natural faces and sounds. The model is not easy to overfit, and at the same time has higher accuracy, and is more robust and generalizable. On the other hand, randomly masking some audio and video frames reduces the amount of calculation, has a high model capacity, stronger anti-interference ability, and higher generalization, which overall improves the applicability of pre-trained features.

[0027] In this embodiment, the cross-modal feature fusion module is used to fuse the visual features extracted by the visual encoder and the audio features extracted by the audio encoder to obtain a feature fusion result.

[0028] In this example, the cross-modal feature fusion module extracts features from stage 3 and stage 4 of the visual encoder and audio encoder respectively for feature fusion. The fusion process is as follows: Figure 3 shown.

[0029] 1) First, multimodal feature fusion, Figure 3 For the two fusion modules on the left, the fusion weights are calculated as follows: , , in, For the The stage outputs audio features, For the Each stage outputs visual features. is the dimension of the feature, is the weight coefficient of the audio feature, is the weight coefficient of the video feature, fusion feature As shown below: , 2) Secondly, the fused multimodal features are fused again, such as Figure 3 For the rightmost fusion module, the fusion weight calculation method is as follows: , , in, In order to obtain the final feature fusion result, is the weight coefficient of the fusion feature corresponding to the third stage.

[0030] In this embodiment, the cross-modal feature fusion module extracts features from stage 3 and stage 4 of the visual feature extractor and audio feature extractor respectively, and performs weighted fusion of features twice. The first weighted fusion is to fuse the multimodal features of stage 3 or stage 4 to mine complementary information between modalities; the second fusion is to fuse the fused multimodal features again, so that the model has both low-level texture feature extraction capabilities and high-level semantic information extraction capabilities. The fused feature information is richer, the detection model has stronger detection capabilities, stronger anti-interference, and higher generalization.

[0031] In this example, the spatiotemporal feature extraction module extracts spatiotemporal features based on the feature fusion result obtained by fusing the cross-modal feature fusion module. Figure 4 As shown on the left, the module adopts a multi-layer and multi-scale design. Each TCN-Block is composed of the right block, and the convolution kernel sizes are 3, 5, and 7 respectively.

[0032] Adversarial interference such as compression affects the robustness of spatial-level features. The forgery traces of the compressed image disappear, but the abnormal face, subtle facial movements and morphological changes in the video can still be captured from the perspective of spatiotemporal regularity and continuity. In this example, the spatiotemporal feature extraction module is based on the temporal convolutional network, and multiple temporal convolutional networks are stacked in depth and width to perform multi-dimensional and multi-scale feature extraction. The coordination of the previous and next frames of the forged video is perceived from the high-level semantic information in the spatiotemporal dimension. The detection model is more robust and less susceptible to post-processing.

[0033] In this embodiment, the spatiotemporal features extracted by the spatiotemporal feature extraction module are input into a classifier, and the classifier classifies the video to be detected as a real video or a forged video based on the spatiotemporal features.

[0034] The training of the video counterfeit detection model in this embodiment includes: 1) Construction of positive and negative sample pairs: Samples in which the audio in the video is fake and the video is real, the audio is real and the video is fake, and both the audio and video are fake are taken as positive samples, and samples in which both the audio and video are real are taken as negative samples; 2) Audio and video preprocessing: adjust the audio and video FPS to 25, and crop them into 0.2s segments, separate the audio and video, perform face detection and tracking on the video to extract the face, obtain face sequences, each sequence has 5 frames, and extract the audio corresponding to the face sequence; 3) Network model construction: extract the visual encoder and audio encoder from the trained audio and video contrast learning model, freeze the parameters of the visual encoder and audio encoder, add a cross-modal feature fusion module, a spatiotemporal feature extraction module and a classifier after the visual encoder and audio encoder, and use the standard cross entropy loss function; 4) Training: The network model inputs audio, video and corresponding authenticity conclusions, calculates the loss, and performs backpropagation until convergence.

[0035] The deep fake video detection method based on multimodal contrast learning in this embodiment specifically includes the following steps: S100, obtaining a video to be detected; S200, inputting the video to be detected into the trained video authentication model, and outputting the detection result of the video to be detected; S210, the video authentication model separates the audio and video of the video to be detected to obtain the audio and video; S220, performing face detection and tracking on the separated video to extract faces, to obtain a face sequence; based on the face sequence, extracting audio corresponding to the time of the face sequence from the audio; S230, extracting visual features from the face sequence obtained in step S220 using a visual encoder; extracting audio features from the audio extracted in step S220 using an audio encoder; S240, using a cross-modal feature fusion module to fuse the visual features and audio features extracted in step S230 to obtain a feature fusion result; S250, extracting spatiotemporal features from the feature fusion result obtained in step S240 using a spatiotemporal feature extraction module; S260 , using a classifier based on the spatiotemporal features extracted in step S250 , classifying the video to be detected as a real video or a forged video.

[0036] Embodiment 2: This embodiment is a storage medium on which a computer program that can be executed by a processor is stored. When the computer program is executed, the steps of the deep fake video detection method in Embodiment 1 are implemented.

[0037] Embodiment 3: This embodiment is a deep fake video detection device, which has a memory and a processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed, the steps of the deep fake video detection method in Embodiment 1 are implemented.

Claims

1. A deep fake video detection method based on multimodal contrastive learning, characterized in that: include: Input the video to be detected into the trained video authentication model, and output the detection result of the video to be detected; The video authentication model includes: The visual encoder is taken from the trained audio and video contrast learning model and is used to extract the visual features of the face sequence in the video to be detected; The audio encoder is taken from the trained audio and video contrast learning model to extract the audio features of the video to be detected; A cross-modal feature fusion module, used to fuse the visual features extracted by the visual encoder and the audio features extracted by the audio encoder to obtain a feature fusion result; A spatiotemporal feature extraction module is used to extract spatiotemporal features from feature fusion results; A classifier is used to classify the video to be detected as a real video or a forged video based on spatiotemporal features; The training of the audio and video comparison learning model includes: The real audio and video are used as positive samples, and the audio in the real audio and video is moved forward or backward by a random time as a negative sample; Separate the audio and video in the positive and negative samples into audio and video, perform face detection and tracking on the video to extract the face, and obtain the face sequence; The visual features of the face sequence are extracted through the visual encoder, and the audio features of the audio corresponding to the face sequence are extracted through the audio encoder; Based on visual and audio features, the contrast loss is calculated to reduce the distance between positive samples and increase the distance between negative samples.

2. The deep fake video detection method based on multimodal contrastive learning according to claim 1 is characterized in that: The cross-modal feature fusion module is used to extract features from stage 3 and stage 4 of the visual encoder and audio encoder respectively for feature fusion.

3. The deep fake video detection method based on multimodal contrastive learning according to claim 2 is characterized in that: The cross-modal feature fusion module includes: ， ， ， in, To fusion features, For the The audio features output by each stage, For the The visual features output by each stage, is the dimension of the feature, is the weight coefficient of the audio feature, is the weight coefficient of the video feature; ， ， in, is the feature fusion result, is the weight coefficient of the fusion feature corresponding to the third stage.

4. The deep fake video detection method based on multimodal contrastive learning according to claim 1, characterized in that: The visual encoder adopts ResNet50 and uses 3D convolution; the audio encoder adopts ResNet18.

5. The deep fake video detection method based on multimodal contrastive learning according to claim 1, characterized in that: The training of the video counterfeit detection model includes: The visual encoder and the audio encoder are extracted from the trained audio-video contrastive learning model, and the parameters of the visual encoder and the audio encoder are frozen.

6. A storage medium having stored thereon a computer program executable by a processor, characterized in that: When the computer program is executed, the steps of the deep fake video detection method according to any one of claims 1 to 5 are implemented.

7. A deep fake video detection device, comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, characterized in that: When the computer program is executed, the steps of the deep fake video detection method according to any one of claims 1 to 5 are implemented.

Images