A specific person deep fake detection method based on audio-video multi-modal

Abstract

The application discloses a specific person deep fake detection method based on audio and video multi-modal, belongs to the field of security and digital image forensics, adopts an audio and video multi-modal fusion algorithm, and combines a lip reading method to capture dynamic characteristics of a face based on time sequence.The steps of the application are as follows: (1) extracting audio and video and lip reading characteristics of a person; (2) fusing lip and time sequence characteristics into facial semantic characteristics; (3) fusing facial semantic characteristics and audio characteristics into a neural network; (3) fusing the characteristics in a classifier to detect real and fake faces.The application is based on a neural network model fusing audio and video multi-modal, innovatively designed for AI face changing detection of a specific person, and meets the requirement of high accuracy for specified person detection.Meanwhile, the application extracts facial time sequence information to capture dynamic characteristics of a face, and refers to the translation and rotation of different faces in the time dimension, so as to make up for the limitation of only referring to the characteristics of the image content itself, and greatly improve the accuracy of fake face detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence security and digital image forensics, and relates to a fake video detection method, in particular, a specific person deepfake detection method based on audio-video multi-modal. BACKGROUND

[0002] Face deepfake originates from a machine learning algorithm released by Reddit user deepfakes in 2017, and claims that the algorithm can help him convert celebrity faces into pornographic videos. The algorithm has been widely discussed by the public and the media since its release, followed by a surge of research on visual deepfake algorithms. In 2018, BuzzFeed released a deepfake video of Barack Obama giving a speech, which was made using the FakeApp software created by Reddit users. From 2017 to 2020, the number of papers related to deepfake increased from 3 to more than 250, and FakeApp, Faceswap, Zao, FaceApp, and other fast deepfake software that can be realized without technical costs have also been developed in turn. Various categories of fake videos made by visual deepfake technology have also raised concerns about identity theft, impersonation, and the spread of false information on social media.

[0003] Face deepfake has undergone great changes over the years. Relying on the latest generative adversarial network architecture such as ProGAN and StyleGAN, face deepfake can greatly restore real face angles, lighting, occlusion, and other changes.

[0004] Face deepfake detection is a technology that has emerged with the improper use of face deepfake. Its main process is divided into:

[0005] a) Data processing. Preprocessing operations such as scaling, detection, alignment, etc. are performed on the face to be detected.

[0006] b) Feature extraction. The processed standardized data is extracted in the model according to prior knowledge or image processing methods.

[0007] c) Model establishment. Build a neural network model that matches the detected person, so that the extracted features can be better classified in the classifier.

[0008] d) Detection and classification. Output results in the classifier.

[0009] It is worth mentioning that the key to determining the detection performance lies in how to select relevant features that can effectively distinguish between real and fake faces, and how to establish a model with good classification effect.

[0010] Common feature extraction methods are as follows: the specific artifact-based detection method focuses on the feature determination part in the flowchart, and captures the abnormal phenomena such as blur, jitter and ghosting in the generated image or video from the image processing point of view with pixel-level granularity; the data-driven detection method focuses on the model establishment part in the flowchart, and uses a carefully designed neural network to train and classify the time domain and frequency domain information extracted from the counterfeit; the information inconsistency-based detection method focuses on capturing the inconsistency between the counterfeit and the objective law from the advanced semantics of biological inherent characteristics, time continuity and motion vector. In 2019, Agarwal team found that different people have obvious pattern differences in facial expressions and head movements when speaking. Among the existing counterfeit methods, this pattern is damaged, that is, the face area in the video is tampered with, causing the pattern of facial expressions and head movements of the person when speaking to be inconsistent with the identity of the person. Therefore, modeling a person using his identity and his exclusive action and expression habits, especially establishing a specific person model, and using these models to distinguish the authenticity of the video has become one of the feasible methods. The team also proposed a soft-biometric model, that is, a leader modeling method that combines facial expressions and head movements. SUMMARY

[0011] Since deepfake videos often come with rich temporal information and audio information, facial deepfake detection is not perfect by extracting semantic information from the image itself as the main basis for discrimination; therefore, the present application proposes a specific person deepfake detection method based on audio-video multi-modal, which selects and extracts information of audio-video multi-modal and performs deep fusion, significantly improving the detection efficiency of specific person AI face changing.

[0012] The specific person deepfake detection method based on audio-video multi-modal of the present application has the following steps:

[0013] First, the data set of the target person's face and audio is extracted to obtain the target person's audio and lip reading features in all positive and negative samples, specifically:

[0014] A, frame-by-frame detection of the target person's face key point coordinate information in the video, according to the extracted coordinate information, the face motion unit intensity F face .

[0015] B, lip motion timing feature extraction.

[0016] Detect the lip key points in the 68 key points of all positive and negative samples, and extract the lip motion feature F lip ;

[0017] C, audio timing feature extraction.

[0018] The audio in the video is converted into frequency domain information, and then the frequency domain information is converted into 512-dimensional mel-frequency cepstral coefficients as the audio feature F mel .

[0019] Further, F face and F lip are fused into the face semantic feature F img .

[0020] Further, the face semantic feature F img is fused with the audio feature F mel to obtain a fused feature F.

[0021] Finally, the fused feature F is subjected to true and false face detection in a classifier, in which the fused feature data is first subjected to dimension reduction through two layers of fully connected feedforward neural network layers, and then the dimension-reduced feature data is contracted to 2 dimensions through a softmax layer for binary classification.

[0022] The present application has the following advantages:

[0023] 1. The specific person deep fake detection method based on audio-video multi-modal is based on a neural network model fusing audio-video multi-modal, and is innovatively designed for AI face changing detection of a specific person, so as to meet the requirement of high accuracy for specified person detection.

[0024] 2. The specific person deep fake detection method based on audio-video multi-modal extracts face time sequence information to capture face dynamic features, and refers to the translation and rotation of different faces in the time dimension, so as to make up for the limitation of only referring to the features of the image content itself, and greatly improve the accuracy of false face detection.

[0025] 3. The specific person deep fake detection method based on audio-video multi-modal adopts a false face detection algorithm based on audio-video multi-modal, uses the fused features of multiple modalities, improves the detection efficiency of AI face changing of a specific person, and timely prevents the use of Deepfake technology to fake the face of a person, so as to fundamentally solve the blackening and rumor-making of public figures by using face changing technology, and improve the threshold of the realistic degree of AI face changing. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The specific person deep fake detection method based on audio-video multi-modal is based on audio-video multi-modal. DETAILED DESCRIPTION

[0027] The present application will be further described in detail below with reference to the accompanying drawings.

[0028] The application is based on a specific person deep fake detection method of audio and video multi-modal, and the specific steps are designed as follows:

[0029] Step 1: Collect and make a data set for the face and audio of the target person, such as a speech video of the target person.

[0030] In order to learn the facial features of a certain person, a large number of real videos of the target person need to be collected first as positive samples for model training; and using the real speech video of the target person, three mainstream open source deep fake tools with different fake methods are used to make deep fake videos of the target person: faceswap, wav2lip, first-order-motion-model, which are used as negative samples for model training.

[0031] Step 2: Extract audio and lip reading features of the person.

[0032] The following steps 201, 202 and 203 are performed on all positive and negative samples to obtain the audio and lip reading features of all positive and negative samples:

[0033] 201, motion unit time sequence feature extraction.

[0034] According to the 68 person face key points defined in the widely used open source tool kit OpenFace in the computer vision industry, the target person face key point coordinate information in the video is detected frame by frame. And according to the extracted coordinate information, the pre-trained Resnet-18 model is used to extract the face motion unit intensity F face (Facial Action Coding System) proposed by Ekman and Friesen in 1978).

[0035] 202, lip motion time sequence feature extraction.

[0036] The lip key points in the 68 key points of all positive and negative samples are detected using Face Alignment, and the pre-trained Resnet-18 model is used to extract the lip motion feature F lip . F lip and the aforementioned F face are 256-dimensional feature vectors.

[0037] 203, audio time sequence feature extraction.

[0038] In the audio data preprocessing stage, the audio in the video is converted into frequency domain information by fast Fourier transform, and then the frequency domain information is converted into 512-dimensional mel cepstral coefficients which are more in line with the hearing habits of human ears by using mel filter and discrete cosine transform as audio features Fmel .

[0039] Step 3: F face and F lip are fused into facial semantic features.

[0040] F face and F lip are fused into facial semantic features by transverse splicing of feature matrixes:

[0041]

[0042] In the formula, represents matrix transverse splicing, F img is 1x256, F mel is 1x256, and after splicing, F img is 1x512.

[0043] Step 4: The facial semantic features F img obtained in Step 2 are fused with the audio features F mel to realize the fusion of F face , F lip and F mel .

[0044] Since F face and F lip both belong to features extracted from facial pixel information, the same type of features are aggregated in Step 3 by using the transverse vector splicing method; and F mel belongs to features extracted from audio information, which is different from F face , which belongs to features of different attributes, so the longitudinal splicing method is used here to fuse the facial semantic features F img and the audio features F mel , and the following is obtained:

[0045] F = [F img , F mel ]

[0046] In the formula above, [·,·] represents matrix longitudinal splicing, such as F img is 1x512, F mel is 1x512, and after splicing, F is 2x512.

[0047] Step 5: The fused features F obtained in Step 4 are subjected to true and false face detection in a classifier. The fused features are subjected to deep fake face detection through the classifier. In the classifier model, the fused feature data is first reduced in dimension through two layers of fully connected feedforward neural network layers, and then the reduced feature data is contracted to 2 dimensions using a softmax layer for binary classification.

[0048] In step 5, when the classifier model is trained, since the Resnet-18 for image and lip movement feature extraction has been pre-trained, the learning rate lr1 is 1e-6 during the training of the classifier model, and the learning rate lr2 of the subsequent classifier of the network (two layers of fully connected feedforward neural network layers and softmax) is set to 1e-5. The loss function calculation method is cross entropy, that is,

[0049]

[0050] Where N is the number of samples, y i is the true value of the i-th sample, p i is the binary classification prediction result of the i-th sample. Where the loss function is used to evaluate the model prediction value p i (represented by 0, 1, true or false), and the video true value y i (represented by 0, 1, true or false) hit degree, the model will update the weight parameters in the network according to the output value L using gradient descent, so that the model prediction value better fits the true value.

[0051] The method of the application is based on a neural network model, which includes an image feature extraction module and an audio feature extraction module. The image feature extraction module uses a pre-trained Resnet-18 network, and the audio feature extraction module uses Fourier fast transform and mel-spectrum extraction. Since the application uses the above-mentioned multi-modal consideration method, the traditional single-modal model is obviously not applicable, so the application adjusts the algorithm based on the above two differences to better utilize the advantages of each modal feature. The extracted facial features and lip reading features are equal-dimensionally spliced, and the spliced features are equal-dimensionally spliced with the audio features to obtain the final fused feature vector. The difference between the audio-video feature fusion used in the application and the single-image modal model is as follows:

[0052] 1) The single-image modal model only considers the semantic content of the image itself, while the deepfake video is often dynamic to make it more convincing, so it cannot fully utilize the rich dynamic information in the video itself.

[0053] 2) Since the model is for a person, audio is a way to distinguish one person from others, so the features it represents can be fully utilized in the detection method of the application.

[0054] In summary, the application focuses on the characteristics of the current traditional general detection dataset of the person, which has too many people, consumes a large amount of resources, has weak security attention, and has weak black box detection ability. The limited computing resources are concentrated on the subject with higher value, emphasizing higher recognition rate and higher black box detection performance.

Claims

1. A method for detecting deep fakes of specific individuals based on multimodal audio and video, characterized by: First, extract the target person's face and audio datasets to obtain the target person's audio and lip reading features in all positive and negative samples, specifically: A. Detect the coordinate information of the target person's facial key points in the video frame by frame, and extract the facial motion unit intensity F based on the extracted coordinate information. face ; B. Extraction of temporal features of lip movement; Detect the lip key points among the 68 key points of all positive and negative samples and extract the lip motion features F lip ; C. Audio time series feature extraction; The audio in the video is converted into frequency domain information, and then the frequency domain information is converted into 512-dimensional Mel cepstral coefficients as audio features F mel ; Furthermore, F face and F lip Fusion into facial semantic features F img ; The facial semantic feature fusion method is: Through the horizontal splicing of the feature matrix, F face and F lip Fusion into facial semantic features: Where, Represents horizontal splicing of matrices; Furthermore, the facial semantic features F img With audio feature F mel Fusion, get the fusion feature F: F=[F img ,F mel ] In the above formula, [·,·] represents the vertical splicing of matrices; Finally, the fused feature F is used in the classifier to detect real and fake faces. In the classifier, the fused feature data is first reduced in dimension through two layers of fully connected feedforward neural network layers, and then the reduced feature data is shrunk to 2 dimensions using the softmax layer for binary classification.

2. The method for detecting deep fakes of a specific person based on multimodal audio and video according to claim 1, characterized in that: When training the classifier, the learning rate lr1 is 1e-6, and the learning rate lr2 of the subsequent classifier of the network is set to 1e-5; the loss function is calculated by cross entropy, that is, Where N is the number of samples, y i is the true value of the i-th sample, p i is the binary classification prediction result of the i-th sample; the above loss function is used to evaluate the model prediction value p i and the true value y of the video i Hit degree; according to the output value L, the weight parameters in the network are updated backtrackingly using the gradient descent method.

Images