A method for detecting compressed depth fake videos based on facial muscle motion

Abstract

This invention relates to a method for detecting compressed depth-spoofing videos based on facial muscle movement, used for identifying compressed depth-spoofing videos on social networks. The invention mainly includes: proposing a method for detecting compressed depth-spoofing videos based on geometric modeling. Based on this, features are analyzed from two different perspectives: adjacent frame difference analysis and time-series feature analysis, to capture unnatural facial muscle movements, thereby achieving the identification of compressed depth-spoofing videos. This invention models the face from a geometric perspective and constructs facial muscle movement features, which can effectively resist the effects of compression and is suitable for detecting compressed depth-spoofing videos in real-world scenarios, possessing practical application value in real-life situations.

Description

Technical Field

[0001] This invention relates to the field of video forensics, specifically to a method for detecting compressed depth-of-field forged videos based on facial muscle movements. Background Technology

[0002] Video, as a crucial carrier of information, carries significant weight. When videos containing misleading content circulate on social networks, they can seriously threaten personal reputation, social credibility, judicial fairness, and national security. Currently, video compression is widely used on social networks. However, video compression reduces visual quality, and compression artifacts and tampering artifacts coexist, posing new challenges to the compression of deepfake videos. Therefore, research on the detection of compressed deepfake videos is urgently needed.

[0003] To detect deepfake videos, since facial distortion during the forgery process leaves artifacts, the paper "HHNguyen, J. Yamagishi, and I. Echizen, Capsule-forensics: Using capsule networks to detect forged images and videos, in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2019, pp. 2307–2311" proposes a novel capsule network for detecting deepfake videos. Due to the temporal defects inherent in forged videos, the paper "D. Güera and EJDelp, Deepfake video detection using recurrent neural networks, in Proceedings IEEE International Conference on Advanced Video and Signal based Surveillance, 2018, pp. 1–6" proposes a time-aware deepfake video detection method. To address the detection of compressed depth-spoofing videos, the paper "H.Liu, X.Li, W.Zhou, Y.Chen, Y.He, H.Xue, W.Zhang, and N.Yu, Spatial-phase shallow learning: rethinkingface forgery detection in frequency domain, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp.772–781" proposes a depth-spoofing video detection method based on upsampling forgery traces.

[0004] Videos are compressed after being distributed through social networks, resulting in the coexistence of compression artifacts and manipulation artifacts in compressed depthfake videos. The methods mentioned above struggle to extract discriminative features from compressed videos, rendering them ineffective in detecting deepfake videos. Furthermore, these methods, relying on convolutional neural networks, have low detection efficiency, hindering deployment in real-world scenarios. Compressed videos are far more common in real life. Considering these practical application challenges, this invention aims to propose a method for detecting compressed depthfake videos based on facial muscle movement, enabling effective identification of such videos on social networks. Summary of the Invention

[0005] This invention proposes a method for detecting compressed depth-spoofed videos based on facial muscle movement, enabling the identification of the authenticity of compressed videos on social networks. The method mainly includes: proposing a geometric modeling-based method for detecting compressed depth-spoofed videos on social networks; and, based on this, capturing unnatural facial muscle movements in the spoofed videos through adjacent frame difference analysis and time-series feature analysis, thereby achieving the detection of compressed depth-spoofed videos on social networks.

[0006] The details are as follows:

[0007] (1) A method for detecting compressed deepfake videos based on geometric modeling is proposed: by analyzing the facial motion of the face, the face is modeled and features are constructed from a geometric perspective, and the designed features are analyzed from different angles to capture unnatural facial muscle movements in the fake video, thereby realizing the detection of compressed deepfake videos on social networks.

[0008] Specifically, it consists of a facial landmark extraction module, a facial muscle movement feature extraction module, and a prediction probability fusion module. The overall detection framework is as follows: Figure 1 As shown, this method, from a geometric perspective, models the facial features and facial regions to construct facial muscle movement characteristics and analyzes them from different angles to capture unnatural facial muscle movements. Because geometric features possess a certain degree of robustness, this invention can effectively resist the effects of compression.

[0009] (2) A geometric modeling method for constructing facial muscle movement features is proposed: facial feature vectors are constructed by modeling the selected regions of interest on the face to represent the movement of facial muscles.

[0010] This module models the region of interest from a geometric perspective based on the extracted facial landmarks, constructing feature vectors. It calculates the length and angle of the facial feature vectors to jointly represent facial muscle movement. A detailed illustration of the features is shown below. Figure 2 As shown, its region of interest includes the facial features region and the face region.

[0011] For the facial features, four regions with relatively obvious movements—eyebrows, eyes, mouth, and nose—were selected for modeling. Vectors with the same number indicate that the two vectors form an angle, and solid-line vectors indicate that the selected vector has its feature vector length calculated. The feature vector length and the vector angle are used together to effectively represent facial muscle movement, where vector length represents changes in length, and vector angle represents changes in direction. For the eyebrow region, the focus is on the movement relationships between eyebrows and between eyebrows and eyes. For the eye region, the focus is on the eye movement process, including the opening and closing of the eyes. Nose movement is not easily observed, but we can use changes at the tip of the nose to represent its movement. Movement around the lips is relatively obvious; we use the shape of the lips and the process of opening and closing the mouth to describe the movement around the mouth. 8, 12, 6, and 16 features were constructed for the eyebrow, eye, nose, and mouth regions, respectively, for a total of 42 features for the facial features.

[0012] For the facial region, the construction method differs from that of the facial feature region. The left and right facial regions are modeled separately. For the left facial region, the feature vector formed by feature points 0 and 39 is selected as the reference vector. Angles are constructed between these vectors and the remaining vectors, and the length of the solid line vector is calculated. The angles and lengths of the vectors can be calculated using formulas (1) and (2). The construction method for the right facial region is similar to that of the left facial region. The feature vector formed by feature points 42 and 16 is selected as the reference vector, and facial region features are constructed together with the remaining vectors. A total of 24 features are generated for the facial region.

[0013]

[0014]

[0015] At this point, facial muscle movement characteristics can be represented as:

[0016]

[0017] Where n = 60, is the number of frames extracted from a video, m = 33, l i,j θ represents the length of the j-th feature vector in the i-th frame. i,j It represents the angle between the j-th feature vectors in the i-th frame.

[0018] (3) A method based on adjacent frame difference analysis and temporal series feature analysis is proposed: After calculating the facial muscle movement features, the temporal relationship is mined by observing the relationship between features in adjacent frames to capture unnatural facial muscle movements. We obtain the difference features by subtracting the facial muscle movement features of the current frame from the facial muscle movement features of the next frame, which can be represented as:

[0019]

[0020] Where, k = n-1, Δl i,j =l i+1,j -l i,j , △θ i,j =θ i+1,j -θ i+1,j .

[0021] In genuine videos, facial muscle movements occur naturally due to facial expressions. After differencing, these muscle movement features exhibit a certain temporal regularity. However, in fake videos, facial expressions are often stiff due to manipulations such as facial distortion. After differencing, the facial muscle features show an irregular temporal distribution. To effectively capture the regularity of this irregular distribution of facial muscle movements in fake videos—that is, unnatural facial muscle movements—we use a GRU network to train the differencing features.

[0022] To effectively resist compression noise interference in the temporal domain modeling, time-series features of facial muscle characteristics are calculated to enhance unnatural facial muscle movements in fake videos.

[0023] Specifically, the input video frame sequence is treated as a coherent whole, and the absolute energy E(vec) of the facial muscle motion features is calculated separately. j ), first-order absolute difference and A(vec j Time series complexity O(vec) j ), kurtosis K (vec j ) and coefficient of variation C(vec j Five time series features. Among them, vec j ={vec[0][j],vec[1][j],...,vec[n][j]} represents the j-th value of the facial muscle movement feature.

[0024] Absolute energy is the squared fluctuation of a time series from the origin, and can be expressed as:

[0025]

[0026] To describe the absolute fluctuations between adjacent observations in time series data, the first-order absolute difference sum can be expressed as:

[0027]

[0028] To evaluate the complexity of the time series, we calculate the time series complexity:

[0029]

[0030] Kurtosis is the fourth standard moment, which can be expressed as:

[0031]

[0032] Where, σ j The standard deviation of a time series, μ j This represents the mean of a time series.

[0033] The coefficient of variation is the ratio of the standard deviation to the mean, and can be expressed as:

[0034]

[0035] Therefore, the time-series features of facial muscle movement characteristics can be represented as follows:

[0036] T vec ={E(vec j ),A(vec j ),O(vec j ),K(vec j ),C(vec j )} (10)

[0037] An SVM classifier is trained using time-series features of facial muscle movement to enhance unnatural facial muscle movements and reduce the impact of compression noise on temporal modeling.

[0038] This method models the face, designs facial muscle movement features, and analyzes and captures unnatural facial muscle movements in fake videos from different angles, thereby achieving effective detection of compressed depth fake videos spreading on social networks.

[0039] Compared with existing technologies, the above solution has at least the following significant advantages:

[0040] 1. This invention provides a method for detecting compressed depth fake videos based on facial muscle movement. By modeling the face and designing geometric features to represent the movement patterns of facial muscles, the designed features are analyzed from different angles to capture unnatural facial muscle movements in fake videos. This feature effectively resists the influence of compression artifacts and compression noise in compressed videos, meeting the detection needs in real-world scenarios.

[0041] 2. This invention models the face from a geometric perspective, which does not require a large number of samples to train the model, thus reducing the time cost and meeting the deployment requirements of the model in real-world scenarios. Attached Figure Description

[0042] Figure 1 This forms the overall detection framework.

[0043] Figure 2 This is a schematic diagram of facial muscle movement characteristics. Detailed Implementation

[0044] This invention is a method for detecting compressed depth-spoofed videos based on facial muscle movements.

[0045] The development language is Python, the development environment is Win7, and the datasets are Faceforensics++ and Faceforensics++Social. The Faceforensics++ dataset contains 6000 videos, including 1000 real videos, 1000 fake videos generated by DeepFakes, 1000 fake videos generated by Face2Face, 1000 fake videos generated by FaceSwap, 1000 fake videos generated by NeuralTextures, and 1000 fake videos generated by FaceShifter. Faceforensics++Social contains 560 real compressed videos and 2800 fake compressed videos. In the specific experiments, the datasets were divided into training and test sets in an 8:2 ratio. The overall detection framework is as follows: Figure 1 As shown, the specific steps are as follows:

[0046] Step 1: Facial Landmark Extraction. The input is a face video, and the output is the face landmarks in the video. First, download the Faceforensics++ and Faceforensics++Social datasets. Install OpenCV and the face detector Dlib in the Python environment, and use OpenCV to extract frames from all videos. Since the designed features mainly rely on facial landmarks, the classic face detector Dlib is used to detect faces in the videos. Because we need facial landmarks to construct features, we perform a face alignment operation to extract accurate landmarks. Face alignment is performed using human eyes, which calculates the center points of the two eyes and the angle between the two eyes, calculates the affine matrix, performs an affine transformation, and then outputs accurate facial feature points.

[0047] Step 2: Facial Muscle Motion Feature Construction. The input consists of facial landmarks and the region of interest (ROI). The output is a representation vector of the constructed facial muscle motion features. Based on the ROI, the required feature points are selected from the set of facial landmarks. Feature vectors are constructed for the eyebrows, eyes, nose, mouth, and face regions respectively. The length and angle of each feature vector are calculated, and the two together represent the facial muscle motion.

[0048] Step 3: Capturing Unnatural Facial Muscle Movements. The input is facial muscle movement features, and the output is the difference features and temporal series features of the facial muscle movements. For the difference features, we subtract the facial muscle movement features of the current frame from those of the next frame. For the temporal series features, we treat the input video frame sequence as a coherent whole and calculate its absolute energy, first-order absolute difference sum, temporal series complexity, kurtosis, and coefficient of variation. In real videos, the difference features and temporal series features exhibit a certain temporal regularity; in fake videos, they show an irregular temporal distribution, thus capturing unnatural facial muscle movements.

[0049] Step 4: Model Training. The input consists of extracted differential and time-series features of facial muscle movements. The output consists of trained GRU and SVM models. The differential and time-series features from Step 3 are used to train the GRU network and SVM, respectively, to capture and enhance unnatural facial movements. For the GRU network, the output has 64 units, with two fully connected layers containing 64 and 2 units respectively. A Drop layer with dr = 0.25 is established between the input and the RNN, and two more Drop layers are added to split the remaining layers. The learning rate is 0.001, the batch size is 1024, and the Adam optimizer is used to optimize the model. The optimal model is saved during training. For the SVM classifier, gamma = 0.000006, the kernel function is a linear kernel function, and the penalty factor C = 0.2.

[0050] Step 5: Prediction Probability Value Fusion. The input consists of the predicted probability values ​​from the two models in Step 4, and the output is the predicted label for the video. To effectively fuse the results of the two models, the Dempster-Shafer evidence theory is used to effectively fuse evidence knowledge from multiple perspectives. The probability values ​​of the two models are fused to obtain the final prediction result. If the fused probability value is greater than 0.5, the video is judged as genuine; if the probability value is less than 0.5, the video is judged as fake.

[0051] Step Six: Model Testing. The input is the processed facial muscle motion features of the video, and the output is the predicted label of the video. First, we test the model using a low-quality Faceforensics++ dataset to assess its effectiveness on compressed videos. To verify the model's effectiveness on compressed videos in real-world scenarios, we test it on Faceforensics++Social, which features compressed videos circulating on social media platforms Facebook and YouTube. The feature extraction method is consistent with Faceforensics++. Since Faceforensics++Social only provides validation and test sets, we train using high-quality Faceforensics++ data and test using Faceforensics++Social. If the model predicts a video as real, the output is 0; if it predicts a video as fake, the output is 1.

[0052] In summary, this invention presents a method for detecting compressed depth-spoofing videos based on facial muscle movement, achieving effective detection of compressed depth-spoofing videos on social networks. It is effective not only for videos compressed using hard-coding methods but also for distinguishing the authenticity of compressed videos circulating on social networks, meeting the requirements for compressed video detection in real-world scenarios. Furthermore, the model does not require significant time overhead, meeting the requirements for model deployment in practical situations.

[0053] Those skilled in the art will understand that the scope of protection of this invention is not limited to the specific embodiments described. Without departing from the principles of this invention, those skilled in the art can make equivalent modifications or substitutions to the relevant technical features. It should be noted that the modified or substituted technical solutions will all fall within the scope of protection of this invention.

Claims

1. A method for detecting compressed depth-of-field forgery videos based on facial muscle movement, characterized in that, The method includes: This method constructs facial muscle movement features through geometric modeling, and captures unnatural facial muscle movements in forged videos through adjacent frame difference analysis and time series feature analysis, thereby achieving the detection of deepfake videos on social networks. The method includes a facial landmark extraction module, a facial muscle movement feature extraction module, and a probability prediction fusion module. The facial landmark extraction module and the facial muscle movement feature extraction module capture unnatural facial muscle movements. The facial landmark extraction module primarily extracts precise facial landmark points L = [x1, y1, ..., x...] from the input face video. 68 ,y 68 In the facial muscle motion feature extraction module, addressing the issues of coexistence of compression artifacts and tampering artifacts and temporal modeling of compression noise interference in compressed videos, the module models the eyebrows, eyes, mouth, nose, and face regions from a geometric perspective based on extracted facial landmarks, constructs geometric feature vectors, and calculates the length l of each feature vector. i,j and the included angle θ i,j The facial muscle movement vec is jointly represented, where the vector length represents the change in length and the vector angle represents the change in direction; adjacent frame difference analysis then extracts the facial muscle movement features of the next frame. i+1,j (θ i+1,j Subtract the facial muscle motion features of the current frame. i,j (θ i,j The difference feature D is obtained. vec This is used to capture unnatural facial muscle movements in fake videos; temporal series feature analysis treats the input video frame sequence as a coherent whole and calculates the absolute energy E(vec) of the facial muscle movement features. j ), first-order absolute difference and A(vec j Time series complexity O(vec) j ), kurtosis K (vec j ) and coefficient of variation C(vec j Five time series features T vec To enhance unnatural facial muscle movements.

Images