Multi-modal fusion detection method for deepfake audio-video

Abstract

The application discloses a kind of multi-modal fusion detection methods for depth fake audio-video, belong to multi-modal machine learning field, including using time-space feature extractor, cross attention cross-modal joint learning decoder and multi-modal classification detector constructs a network architecture to carry out audio-video multi-modal identification;Time-space feature extractor is responsible for the unified processing to audio and video modal characteristics;Cross attention cross-modal joint learning decoder makes two kinds of modal information to carry out joint learning by two parallel decoders;Multi-modal classification detector outputs two classification results by fusing the feature information of both.This application utilizes the complementarity of audio-image modal, can distinguish whether the video that human is not easy to perceive is fake, enhances the detection robustness of depth fake data generated by various fake methods, improves the security of face recognition device and false generated content supervision.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal machine learning, and specifically relates to a multimodal fusion detection method for deep fake audio and video. Background Art

[0002] Machine learning models based on deep neural networks have brought significant advancements in various fields, including images, audio, and text. However, this advancement in AI-generated content also poses security risks related to copyright and disinformation, including the widespread production of fake text. With the continuous advancement of AI-generated content, attackers can now generate large quantities of high-quality, difficult-to-identify fake videos in a short period of time. Human monitoring is both time-consuming and expensive. Therefore, many organizations urgently need methods to detect high-quality fake videos.

[0003] Deepfake video detection methods can be considered subtasks of anomaly detection and fine-grained detection, so many other detection methods can be applied to this task. Many detection methods targeting different feature information have achieved good detection accuracy on fake face datasets and fake audio datasets. However, most detection methods focus only on data from a single modality, and few simultaneously detect data containing multiple modalities. As a result, they are likely to miss key fake information. Directly integrating the results of multiple models will lose the correlation between different data, leading to false positives. Therefore, with the continuous development of generative methods, it is necessary to design a class of models that can integrate multiple data and complete the detection task without sacrificing recognition accuracy. Summary of the Invention

[0004] The technical problem solved by the present invention is: without losing recognition accuracy, a detection method for deep fake audio and video is realized by constructing a network architecture to perform audio-video multimodal fusion detection.

[0005] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] A multimodal fusion detection method for deepfake audio and video includes: using a spatiotemporal encoder, a multimodal joint decoder, and a cross-modal detector to build a network architecture for audio-video multimodal recognition; the spatiotemporal encoder is responsible for unified processing of audio and video modal features; the multimodal joint decoder uses two parallel decoders to jointly learn the two modal information; and the cross-modal detector outputs a binary classification result by fusing the feature information of the audio feature vector and the visual feature vector.

[0007] As a preferred method, the method of encoding a face video with audio using a spatiotemporal encoder is as follows:

[0008] Two serial Transformer encoders are used as feature extractors. First, the audio and video modalities are uniformly sampled and preprocessed. The preprocessed feature vectors are input into the first Transformer encoder.

[0009] The code generated by the second Transformer encoder represents the specific spatial features of the corresponding time node; the code finally generated by these two encoders contains the spatial information corresponding to the audio and image at each time node;

[0010] Then, the information fused from the two modalities is input into the multimodal joint learning block in parallel for joint learning of the latent space.

[0011] Preferably, the first Transformer encoder model encodes interactive information contained in different video clips and audio clips within the same time window.

[0012] As a preferred method, the method of joint learning of two modal information is as follows:

[0013] The bi-attention method is used to jointly learn the two modal data. A multimodal signal fusion module is used to decode the visual modality feature vector and the audio modality feature vector. Two parallel Transformer modules are used as decoders for the visual modality signal and the audio modality signal. A bidirectional cross-attention mechanism is used to share and jointly learn different modal information.

[0014] The audio feature vector can be expressed as:

[0015]

[0016] The image feature vector can be expressed as:

[0017]

[0018] The self-attention feature vector can be expressed as:

[0019]

[0020] Among them, Q is the query vector, K is the key vector, and V is the value vector;

[0021] P v and P a Represents the visual modality feature vector and audio modality signal feature vector encoded and output from the spatiotemporal encoder, W q , W k , W v The weight matrices corresponding to Q, K, and V vectors, d k To Q*(K)T The product result is normalized.

[0022] Preferably, the cross-modal detector outputs a binary classification result by fusing feature information of the audio feature vector and the visual feature vector as follows:

[0023] The output vector is input into the classifier, the fully connected layer, and the final binary classification value is output. The specific representation is as follows:

[0024] Y=FC(concat(F Audio ,F Visual ))

[0025] Among them, FC stands for fully connected layer, F Audio 、F Visual Represent the audio feature vector and visual feature vector output by the decoder respectively, concat(F Audio ,F Visual ) represents concatenation of two vectors.

[0026] The final result is a probability value of the audio-video matching correlation degree. The higher the probability value, the higher the probability of being judged as true.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0028] (1) This paper proposes a universal and generalizable model, which improves the accuracy of general deep fake video detection by fusing information from different modalities into model training.

[0029] (2) The model uses a Transformer encoder to encode the corresponding video and audio information at the same time point, which can effectively integrate information at different time sequences and combine visual and auditory signals for detection. Compared with traditional forgery detection methods, it does not simply discard valid information for detection.

[0030] (3) The detection of unknown modality forgery can effectively distinguish whether the video is forged or not, thus improving the robustness of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of a multimodal fusion detection method for deep fake audio and video. DETAILED DESCRIPTION

[0032] The present invention will be further illustrated below with reference to specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0033] like Figure 1As shown, the present invention is a multimodal fusion detection method for deep fake audio and video, including: using a spatiotemporal encoder, a multimodal joint decoder and a cross-modal detector to construct a network architecture for audio-video multimodal recognition; the spatiotemporal encoder is responsible for unified processing of audio and video modal features; the multimodal joint decoder uses two parallel decoders to jointly learn the two modal information; the cross-modal detector outputs a binary classification result by fusing the feature information of the audio feature vector and the visual feature vector.

[0034] The method of encoding face-containing video with audio using a spatiotemporal encoder is as follows:

[0035] First, two serial Transformer encoders are used as feature extractors. First, the two different modalities of audio and video are uniformly sampled and preprocessed. The preprocessed feature vectors are input into the first Transformer encoder, i.e., the multimodal temporal encoder. The first Transformer encoder model encodes the interactive information contained in different video and audio clips within the same time window.

[0036] The second Transformer encoder, the multimodal spatial encoder, generates a code that represents the specific spatial features of the corresponding time node. The code generated by these two encoders contains the spatial information corresponding to the audio and image at each time node. The information fused from the two modalities is then input in parallel into the multimodal joint learning block for joint learning of the latent space.

[0037] Secondly, the two modal data are jointly learned through the Bi-attention method: a multimodal signal fusion module is used to decode the visual modal feature vector and the audio modal feature vector, and two parallel Transformer modules are used as decoders for the visual modal signal and the audio modal signal; since information from different modalities is needed to assist in detection, a bidirectional cross-attention mechanism is adopted to share and jointly learn information from different modalities.

[0038] The audio feature vector can be expressed as:

[0039]

[0040] The image feature vector can be expressed as:

[0041]

[0042] The self-attention feature vector can be expressed as:

[0043]

[0044] Where Q is the query vector, K is the key vector, and V is the value vector.

[0045] P v and P a Represents the visual modality feature vector and audio modality signal feature vector encoded and output from the spatiotemporal encoder, W q , W k , W v The weight matrices corresponding to Q, K, and V vectors, d k To Q*(K) T The product result is normalized.

[0046] The output vector is input into the classifier, that is, the fully connected layer, and the final binary classification value is output. The specific representation is as follows:

[0047] Y=FC(concat(F Audio ,F Visual ))

[0048] Among them, FC stands for fully connected layer, F Audio 、F Visual Represent the audio feature vector and visual feature vector output by the decoder respectively, concat(F Audio ,F Visual ) means concatenating two vectors.

[0049] The final result is a probability value of the audio-video matching correlation degree. The higher the probability value, the higher the probability of being judged as true.

[0050] Example 1

[0051] The effectiveness and efficiency of the method of the present invention are verified by the following experiments:

[0052] The following measures are used to evaluate the effectiveness and efficiency of the proposed attacks:

[0053] Classification Accuracy — It is defined as the accuracy of classifying real samples and fake samples at the video level.

[0054] AUC – It is defined as the probability that the true positive rate is greater than the false positive rate.

[0055] This paper selects the Faceforensics++ dataset and the Deepfake Detection Challenge dataset. The Faceforensics++ dataset is a deep fake benchmark dataset, including 5,000 video sequences with audio channels. The video channel operations include the following four methods: Deepfakes, Face2Face, FaceSwap and NeuralTexture.

[0056] The DFDC dataset is currently the largest video deepfake dataset, with over 100,000 videos and English audio sequences. Voices that come from cameramen rather than actors are removed to ensure audio-visual synchronization.

[0057] The backbone network includes Transformer and Bi-attention models.

[0058] 1. Transformer

[0059] The Transformer network is the backbone of our model. It consists of two parts: an encoder and a decoder, each containing six blocks. It calculates the weights of different positions in a feature vector using region-value encoding, position encoding, and a weight matrix to generate new feature vectors.

[0060] 2. Bi-attention

[0061] In the parallel Transformer model, one of the Q, K, and V weight matrices can be swapped with the weight matrix value of another model to promote the fusion of different features.

[0062] The comparison classification models include INDP, Late-fuse and different types of two-stream.

[0063] 1. Indp

[0064] The video stream and audio stream are trained independently, that is, the backbone networks of the three are used separately during the training of the two.

[0065] 2. Late-fuse

[0066] The model is trained directly on the video stream and audio stream. First, the backbone network of the three is used to extract the features of the two modalities respectively, and then the two networks are fused through late fusion.

[0067] 3. Two-stream

[0068] This represents a joint detection network using multimodal fusion learning, the method proposed in this paper. However, this paper considers removing different structures as part of an ablation experiment. First, the video and audio modalities are aligned, and then a cross-attention method is applied. This method is called synchronized flow. The present invention considers the individual detection results of different modalities to reflect whether any one mode has been manipulated. The experimental results are shown in Tables 1 and 2.

[0069] Table 1 Classification accuracy of the present invention on the Faceforensics++ dataset, with AUC in brackets

[0070]

[0071] Table 2 Classification accuracy of the present invention under DFDC dataset, with AUC in brackets

[0072]

[0073] The results in Tables 1 and 2 show that our joint detection network, Two-Streams, consistently outperforms Indp and Late-Fuse on the single tasks of whole-sequence detection and video and audio deep-fake detection. Comparing the results across different types of attention mechanisms, we found that our Bi-Attention approach outperformed the Joint-Attention approach, particularly on whole-sequence and audio stream prediction. This suggests that cross-modal attention contributes more significantly to learning synchronization patterns than self-attention.

[0074] This paper proposes a multimodal fusion detection method for deep fake audio and video. By fusing information from different modalities for model training, the accuracy of general deep fake video detection is improved. By utilizing the complementarity of audio and image modalities, it can effectively distinguish whether the video is forged, thereby enhancing the robustness of detection of deep fake data generated by various forgery methods.

[0075] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal fusion detection method for deep fake audio and video, characterized by: include: Build a network architecture using spatiotemporal encoder, multimodal joint decoder, and cross-modal detector for audio-video multimodal recognition; The spatiotemporal encoder is responsible for unified processing of audio and video modal features; the multimodal joint decoder uses two parallel decoders to jointly learn the two modal information; The cross-modal detector outputs a binary classification result by fusing the feature information of the audio feature vector and the visual feature vector; The method of encoding face-containing video with audio using a spatiotemporal encoder is as follows: Two serial Transformer encoders are used as feature extractors. First, the audio and video modalities are uniformly sampled and preprocessed. The preprocessed feature vectors are input into the first Transformer encoder. The encoding generated by the second Transformer encoder represents the specific spatial features of the corresponding time node; The code generated by these two encoders contains the spatial information corresponding to the audio and image at each time node; Then, the information fused from the two modalities is input into the multimodal joint learning block in parallel to perform joint learning of the latent space. The method for joint learning of two modal information is as follows: The bi-attention method is used to jointly learn the two modal data. A multimodal signal fusion module is used to decode the visual modality feature vector and the audio modality feature vector. Two parallel Transformer modules are used as decoders for the visual modality signal and the audio modality signal. A bidirectional cross-attention mechanism is used to share and jointly learn different modal information. The audio feature vector can be expressed as: The image feature vector can be expressed as: The self-attention feature vector can be expressed as: Among them, Q is the query vector, K is the key vector, and V is the value vector; P v and P a Represents the visual modality feature vector and audio modality signal feature vector encoded and output from the spatiotemporal encoder, W q , W k , W v The weight matrices corresponding to Q, K, and V vectors, d k To Q*(k) T The product result is normalized; The cross-modal detector outputs a binary classification result by fusing the feature information of the audio feature vector and the visual feature vector as follows: The output vector is input into the classifier, the fully connected layer, and the final binary classification value is output. The specific representation is as follows: Y=FC(concat(F Audio ,F Visual )); Among them, FC stands for fully connected layer, F Audio 、F Visual Represent the audio feature vector and visual feature vector output by the decoder respectively, concat(F Audio ,F Visual ) represents concatenation of two vectors; The final result is a probability value of the audio-video matching correlation degree. The higher the probability value, the higher the probability of being judged as true.

2. The multimodal fusion detection method for deep fake audio and video according to claim 1 is characterized by: The first Transformer encoder model encodes the interactive information contained in different video and audio clips within the same time window.

Images