A deep fake video detection method based on three-dimensional space-time trajectory

Abstract

The present application relates to a kind of deep fake video detection method based on three-dimensional space-time trajectory.The invention mainly includes: propose a kind of deep fake video detection method based on three-dimensional space-time feature learning, on this basis, by combining the feature details of 2D frame and 3D frame, decoupling face expression and head movement, construct space-time motion feature from new angle, and design the time sequence analysis method based on phase space motion trajectory, the feature difference between real face and fake face in deep fake video is mined, to realize the detection of compressed deep fake video.The present application adopts robust three-dimensional model to construct space-time motion feature, can avoid the influence of frame with large head rotation angle or insufficient light, model training and detection speed are fast, with high efficiency and timeliness, can well deal with the detection task of compressed deep fake video in real scene.

Description

Technical Field

[0001] This invention relates to the field of video forensics, specifically to a method for detecting deepfake videos based on three-dimensional spatiotemporal trajectories. Background Technology

[0002] Deepfake is a portmanteau of "deep learning" and "fake" in the field of artificial intelligence (AI), often used to describe fake media generated using technologies such as deep learning. Deepfake technology can replace the identity of a target person in images and videos, and can also make the target person perform corresponding facial expressions and actions based on driving video or specified audio. Furthermore, deepfake technology can edit the facial attributes of a target person, and even generate faces that do not exist in real life. In recent years, deep learning algorithms have been continuously iterating, and AI-generated content has flourished, with AI-forged high-quality images and videos reaching a level indistinguishable to the human eye. Deepfake technology has certain positive applications, promoting the emerging development of the entertainment and cultural exchange industries, and possessing strong entertainment and dissemination potential. However, some criminals use these technologies for telecommunications fraud and other purposes, posing a serious threat to individuals, society, and the nation. Therefore, research on deepfake detection technology is particularly crucial.

[0003] To detect deepfake videos, researchers have proposed four detection techniques based on image forensics, data-driven methods, biosignals, and frequency to determine whether a video has been forged. The paper "Yang X, Li Y, Lyu S. Exposing deepfakes using inconsistent head poses. In: Proc. of the IEEE Int'l Conf. on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019. 8261-8265" proposes a detection method based on 3D head pose evaluation. The paper "Hu J, Liao X, Wang W, and Qin Z, Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(3): 1089-1102" proposes a two-stream network to detect hard-coded compressed videos. The paper "Woo S. ADD: Frequency Attention and Multi-View Based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Images. In: Proc. of AAAIConference on Artificial Intelligence. 2022, 36(1): 122-130" proposes a knowledge distillation method based on frequency domain learning and optimal transmission theory to detect low-quality deepfake videos. The paper "D. Zhang, C. Fu, D. Lu, J. Li and Y. Zhang," Bi-source Reconstruction based Classification Network for Face Forgery Video Detection," in IEEE Transactions on Circuits and Systems for Video Technology, doi:10.1109 / TCSVT.2023.3330390" proposes a classification network based on dual-source reconstruction, using 2D and 3D reconstructions as supervision to learn the optimal feature representation.

[0004] Video compression is a common phenomenon on social networks. When users upload videos to social media platforms (such as Facebook, Instagram, Twitter, etc.), these platforms typically compress the videos to reduce file size and speed up upload and playback. This compression usually employs different compression algorithms and parameters to balance video quality and file size. Compression can lead to loss of video detail, reduced video resolution, frame rate, and image quality, and cause compression artifacts to overlap with tampering artifacts, thereby reducing the performance of existing deepfake video detection methods. To effectively address compressed deepfake videos in real-world scenarios, this invention aims to improve the detection performance and robustness of deepfake videos by proposing a deepfake video detection method based on three-dimensional spatiotemporal trajectory. Summary of the Invention

[0005] This invention proposes a method for detecting deepfake videos based on three-dimensional spatiotemporal trajectories, aiming to determine the authenticity of compressed deepfake videos. The method mainly includes: proposing a deepfake video detection method based on three-dimensional spatiotemporal feature learning; on this basis, by combining feature details from 2D and 3D frames, decoupling facial expressions and head movements, constructing spatiotemporal motion features from a new perspective, and designing a temporal analysis method based on phase space motion trajectories to mine the feature differences between real and fake faces in deepfake videos, thereby achieving the detection of deepfake videos.

[0006] It mainly includes three parts:

[0007] (1) A method for detecting deepfake videos based on three-dimensional spatiotemporal motion features is proposed.

[0008] (2) A spatiotemporal feature construction method based on a three-dimensional model is proposed.

[0009] (3) A time-series feature analysis method based on phase space motion trajectory is proposed.

[0010] The details are as follows:

[0011] (1) A method for detecting deepfake videos based on three-dimensional spatiotemporal motion features is proposed: by studying facial motion, temporal and spatial motion features are constructed, and temporal analysis of phase space motion trajectories is performed to capture the difference features between real and fake faces in deepfake videos, thereby achieving the determination of the authenticity of deepfake videos. This method includes a three-dimensional spatiotemporal feature construction module and a phase space motion trajectory analysis module. The overall detection framework is as follows: Figure 1As shown. The key to this method is to use a robust 3D landmark localization and tracking method to construct spatiotemporal motion features, decouple head motion from facial expressions, consider a wide range of head movements, and evaluate the facial muscle movements of the test subject in a more granular way, so as to effectively resist the impact of compressed video on the model detection performance.

[0012] (2) A spatiotemporal feature construction method based on a 3D model is proposed: This method uses a robust 3D model for facial landmark localization and tracking, while simultaneously tracking and decoupling head motion. Then, facial points and head poses are selected to construct phase-space motion trajectories to characterize the relative movement patterns of facial landmarks generated by facial muscles in the temporal and spatial domains. Specifically, firstly, a 2D facial alignment algorithm is used to automatically locate 68 landmarks for each frame of the facial video. Secondly, a 3D facial model is used to estimate depth information from 2D frames, thereby achieving 3D landmark tracking. Finally, spatiotemporal motion features are constructed through the spatial dynamics and temporal combination of facial action units (AUs). A schematic diagram of the features is shown below. Figure 2 As shown.

[0013] 1) 2D Facial Landmark Localization. This method uses a cascade of trained regressors to localize facial landmarks in each video frame. Gradient tree augmentation is used to train each regressor, employing a cumulative squared error loss approach. Assume the training dataset is {(I1,S1),(I2,S2),…(I…} n ,S n )}, where each I i It is a facial image, S i It is its shape vector. An initial shape estimate is set for each face image. In each regression tree, the gradient tree augmentation algorithm is used to learn the regression function r. t Then the estimate for each shape will be updated as follows:

[0014]

[0015] Initial shape of each frame The mean shape of the training dataset is simply chosen, and the landmarks are centered and scaled using Histogram of Oriented Gradients (HOG) features based on the detected full face bounding boxes. At each level of the cascade, the estimated landmarks are refined using the residuals generated by the previous regression tree. It's important to note that all frames in the video are normalized to ensure that the pixels between the two eyes have the same Euclidean distance. Therefore, the movement of landmarks is comparable within the same individual.

[0016] 2) 3D Facial Landmark Tracking. The facial landmark tracking algorithm maintains invariance to head movements, including translation, scaling (approaching or moving away from the camera), and rotation (rolling, yaw, pitch). To eliminate interference from head movements, depth information for each facial pixel is first extracted from 2D video frames using a 3D deformable facial model. This model includes a principal component analysis (PCA) model of the facial shape, which can be used to reconstruct a 3D face from a single 2D image. The PCA model consists of principal components V = [υ1,υ2,…,υ]. n Average value of all facial grids Standard deviation σ n Composition. Then, the shape of a new face is generated as follows:

[0017]

[0018] Where n is the number of principal components, α n It is S i Representation in PCA shape space coordinates. The 3D facial shape is reconstructed by fitting 68 detected landmarks to the PCA shape model. To perform model fitting, we use the gold-standard algorithm to find the least-squares approximation of the affine camera matrix for a given 2D-3D point pair.

[0019] By utilizing a 3D geometric transformation matrix, each frame can be transformed into a frontal face, enabling the tracking and comparison of facial landmark motion throughout the video. Simultaneously, a geometric constraint method for facial landmarks is applied to address the issue of insufficient accuracy in facial landmark detection algorithms on frames with large head rotation angles or insufficient lighting.

[0020] 3) Constructing spatial and temporal features based on a 3D model. Studies have shown that by combining the spatial dynamics and temporal aspects of facial action units (AUs), facial muscle movements can be quantified more reliably and specifically, while also helping to reduce random noise in landmark localization. We reduced the dimensions of facial landmark features from 2×68 to 12 features, as follows: (1-2) y-values ​​of the left and right eyebrows (corresponding to AU 1, 2, and 5); (3) difference in the inner angles of the eyebrows (corresponding to AU 4); (4) horizontal distance between the corners of the mouth (corresponding to AU 12 and 20); (5) vertical distance between the lips (corresponding to AU 25, 26, and 27); (6) average vertical position of the two corners of the mouth (corresponding to AU 15); (7-9) rigid body displacement of the head in the X, Y, and Z directions; (10-12) rigid body rotation of the head in the roll, pitch, and yaw directions.

[0021] Compared to other open-source tools that can directly detect AUs, our method can provide more information related to facial motion: most AU detection algorithms can only handle faces in frontal views, while our 3D landmark tracking method can directly extract head motion from videos; our method provides continuous measurements of facial landmarks, rather than just six discrete values ​​representing AUs and their intensity.

[0022] (3) A temporal feature analysis method based on phase space motion trajectory is proposed: This method first processes the previously constructed features by introducing time-delay embedding to reconstruct the phase space trajectory. Second, a recursive graph (RP) is created to capture the relationships between features. Finally, a lightweight Transformer architecture is adopted to mine the differences in feature distribution in the temporal and spatial domains between real and fake videos. Existing detection methods mainly focus on the temporal differences between adjacent frames. Our method focuses on modeling the facial change patterns between the first frame and each subsequent frame within a continuous period, emphasizing holistic and global video feature analysis.

[0023] A recursive graph (RP) is a visualization method that uses a binary square matrix to represent the temporal dependencies between all states in time series data. Suppose that at times i and j, the states of system (or mode) X are determined by X... i and X j This indicates that recurrence can be recorded using a binary function, as shown below:

[0024]

[0025] Here, Θ is a Heaviside function. For two time points i and j in a time series, if their similarity exceeds a certain predetermined threshold ∈ X Then a point will be displayed at the corresponding position in the recursion graph (i.e., By repeating this process over the entire time series, a matrix can be created where each element represents the similarity between the corresponding time points.

[0026] First, we extract the feature motion trajectories of the first and subsequent frames of the video under test to construct a recursive graph. Then, we transform the recursive matrix into the network's adjacency matrix to represent the spatiotemporal neighborhood relationships between system states throughout the time series. Finally, to fully explore the spatiotemporal feature differences between real and fake videos, we design a lightweight Transformer classification model: First, the initial part of the model includes a linear layer to embed the input data into the hidden representation space. This allows the model to learn useful representations of the data without prior manual feature engineering. Second, we achieve a more lightweight Transformer model by setting a smaller hidden dimension (128), encoder layer number (2), and attention head number (2). This design is more suitable for practical applications, especially when tasks require efficient deployment and resources are limited. Next, we use average pooling to generate the final classification probability label, rather than a sequence-to-sequence output, to reduce the computational complexity of the model. Finally, we calculate the loss between the predicted label and the actual label, and then complete the training by updating the network parameters.

[0027] This method employs robust 3D landmark localization and tracking techniques to construct more holistic and global temporal and spatial motion features. By using a temporal analysis method based on phase space motion trajectories, it mines the feature differences between real and fake faces in deepfake videos, thereby achieving effective detection of deepfake videos.

[0028] Compared with existing technologies, the above solution has at least the following significant advantages:

[0029] 1. This invention provides a method for detecting deepfake videos based on three-dimensional spatiotemporal trajectories. It analyzes facial motion from a new perspective, uses a robust three-dimensional model for landmark localization and tracking, and combines the feature details of 2D and 3D frames to construct spatiotemporal motion features. These features can avoid the influence of frames with large head rotation angles or insufficient lighting, and can well cope with the detection task of deepfake videos in real-world scenarios.

[0030] 2. Based on the spatial dynamics and temporal combination characteristics of facial action units (AU), this invention utilizes phase space motion analysis to mine the differences in facial features between real and fake compressed videos. The model training and detection speed is fast, and it has high efficiency and timeliness, meeting the needs of compressed depth-spoofing video detection in real-world scenarios. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the overall detection framework of the present invention, "A method for detecting deepfake videos based on three-dimensional spatiotemporal trajectories".

[0032] Figure 2 This is a schematic diagram illustrating the features of the present invention, "A method for constructing spatiotemporal features based on a three-dimensional model". Detailed Implementation

[0033] To more clearly demonstrate the features and advantages of this patent, detailed descriptions of embodiments are provided below. It should be understood that the following detailed descriptions are merely examples and are intended to provide further explanation of this application. Unless otherwise stated, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0034] This invention is a method for detecting depth-spoofed videos based on three-dimensional spatiotemporal trajectories.

[0035] The development language is Python, the development environment is Linux, and the datasets are Deepfake Detection Challenge (DFDC), Faceforensics++, and FakeAVCeleb. The DFDC dataset contains over 119,154 video clips from 3,426 actors, using methods such as DFAE, MM / NN face swapping, NTH, StyleGAN, FSGAN, and TTS Skins voice conversion to create deepfake clips. Each video is 10 seconds long. The Faceforensics++ dataset contains 4,000 video clips generated by four forgery methods: DeepFakes, Face2Face, FaceSwap, and NeuralTextures, as well as 1,000 real video clips. FakeAVCeleb is a multimodal deepfake detection dataset proposed in 2021, consisting of 500 real videos and 19,500 forged videos, containing deepfake visual content and corresponding synthetic lip-synced fake audio. In this paper's experiments, the training, validation, and test sets are divided in an 8:1:1 ratio. The overall framework of the algorithm is as follows... Figure 1 As shown. The specific steps are as follows:

[0036] Step 1: Facial Landmark Localization and Tracking. Input a face video, outputting 68 3D landmarks representing the face and head pose. Specifically, first, frame-level evaluation is performed on the input video to confirm the presence of a face within the frame. Next, a semantic model (dlib library) is used to localize the 68 facial landmarks, performing frame segmentation and annotation of the face region. A cascaded regression tree framework is used, refining the estimated landmarks at each cascade level by adding the residuals from the previous regression tree. Finally, the 3D facial model is used to estimate depth information and pose from the 2D frames, thus enabling the tracking of the 3D landmarks.

[0037] Step 2: Spatiotemporal Motion Features and Phase Space Trajectory Construction. The input consists of 68 3D landmarks representing the facial and head poses, and the output is a spatiotemporal motion feature vector. Specifically, firstly, to ensure the randomness of the video segments, a specific window size is given, and the video frame set is sampled multiple times through smooth window movement. Secondly, corresponding landmarks are selected based on the 12 features designed earlier. Next, the horizontal distance, vertical distance, Euclidean distance, and interior angle feature vectors of the corresponding features are calculated. Finally, based on the feature vectors of the facial action units (AUs) combined in space and time, the phase space trajectory is reconstructed to obtain the corresponding RP map, outputting the RP matrix and constructing the positive and negative samples of the network.

[0038] Step 3: Network Training. Inputting the positive and negative phase space motion samples constructed in Step 2, we use single-center loss and Softmax loss to construct the total loss function to train the point cloud network. The parameterized centers of the single-center loss are randomly initialized and updated based on a mini-batch dataset (not the entire dataset), which leads to unstable network training. Therefore, we introduce Softmax loss with global information to guide the update of the center points. The network uses the Adam optimization algorithm to update the model parameters, with an initial learning rate of 0.0001. When the loss function no longer decreases during training, the learning rate scheduler is used to adjust the learning rate based on the total loss of the current epoch. The batch size is 128, and the epoch is 30. In each epoch, the loss, ACC, and AUC are calculated, and the model parameters are saved based on the optimal loss.

[0039] Step 4: Joint Prediction. This paper uses a voting method as the joint prediction approach, combining multiple prediction results to reduce variance and thus improve model robustness. We first perform binary labeling on the prediction results, and then select the ones with higher probabilities. The pseudocode for the specific implementation is shown below:

[0040]

[0041] Step 5: Algorithm Testing. Input the video to be tested, and output the predicted label of the video. If the algorithm outputs 0, it means the video is predicted as true. If the algorithm outputs 1, it means the video is predicted as false. First, train and test on the Faceforensics++, FakeAVCeleb, and DFDC datasets respectively to verify the detection performance of the algorithm. Second, test using Faceforensics++ low-quality data and Faceforensics++ Social data to verify the model's compression resistance. Finally, to verify the algorithm's generalization ability, set up cross-dataset experiments. Specifically, train on the FakeAVCeleb dataset and test on the DFDC dataset. Train on the DFDC dataset and test on the FakeAVCeleb dataset.

[0042] In summary, this invention presents a method for detecting deepfake videos based on three-dimensional spatiotemporal trajectories. Compared to existing detection algorithms, this method demonstrates competitive detection performance and efficiency. Furthermore, it exhibits good detection performance even with datasets featuring large head rotation angles, insufficient lighting, and compressed real-world scenes, making it more suitable for real-world detection needs.

[0043] Those skilled in the art will understand that the scope of protection of this invention is not limited to the specific embodiments described. Without departing from the principles of this invention, those skilled in the art can make equivalent modifications or substitutions to the relevant technical features. It should be noted that the modified or substituted technical solutions will all fall within the scope of protection of this invention.

Claims

1. A method for detecting depth-spoofed videos based on three-dimensional spatiotemporal trajectories, characterized in that: This paper constructs spatiotemporal motion features from a new perspective, combining feature details from 2D and 3D frames to decouple head movement from facial expressions. It also designs a temporal analysis method based on phase-space motion trajectories, primarily including: The system comprises a 3D spatiotemporal feature construction module and a phase space motion trajectory analysis module. In the 3D spatiotemporal feature construction module, a robust 3D model is used for facial landmark localization and tracking, while simultaneously tracking head movements decoupled from facial expressions. Spatiotemporal motion features are then constructed through the spatial dynamics and temporal combination of facial action units (AUs). In the phase space motion trajectory analysis module, motion features are embedded with time delays to reconstruct phase space trajectories and create a recursive graph (RP). A lightweight Transformer architecture is then trained to mine the differences in feature distributions between real and fake videos in the temporal and spatial domains. Finally, a voting method is used to fuse the model results. The three-dimensional spatiotemporal feature construction module directly locates and tracks facial and head landmarks in the video using a robust three-dimensional model, and constructs spatiotemporal motion features based on the spatial dynamics and temporal combination of facial action units (AUs). Specifically, this includes: The system comprises a 3D landmark tracking and localization module and a facial motion unit spatiotemporal combination module. In the 3D landmark tracking and localization module, depth-spoofing video detection is performed by combining detailed feature changes from 2D and 3D frames. First, a face video is input into the 3D landmark tracking and localization module, and a regression tree cascade framework is used to refine and estimate the facial landmarks in each video frame. Second, a 3D deformable face model extracts the depth information of each facial pixel from the 2D video frames. Then, a 3D geometric transformation matrix and geometric constraint method are used to convert each frame into a frontal face, thereby tracking and comparing the motion of facial landmarks throughout the video. In the facial motion unit spatiotemporal combination module, 12 spatiotemporal motion features are constructed between the first frame and each subsequent frame over a continuous period: the y-values ​​of the left and right eyebrows, the difference in the inner angles of the eyebrows, the horizontal distance between the corners of the mouth, the vertical distance between the lips, the average vertical position of the two corners of the mouth, the rigid body displacement of the head in the X, Y, and Z directions, and the rigid body rotation of the head in the roll, pitch, and yaw directions, thus obtaining the 3D spatiotemporal motion features. The phase space motion trajectory analysis module models the facial change patterns between the first frame and each subsequent frame over a continuous period, analyzing the overall and global features of the video, specifically including: The system consists of a phase space motion feature module and a representation learning module. In the phase space motion feature module, 12 facial features from the first frame and subsequent frames of the video under test are first introduced into time-delay embedding to reconstruct the phase space motion trajectory. Then, a recursive graph is created based on the phase space motion trajectory to represent the relationship between features. In the representation learning module, the recursive matrix is ​​transformed into the adjacency matrix of the network and input into a lightweight Transformer architecture to learn the spatiotemporal neighborhood relationship between system states throughout the time series.

Images