An efficient fake face video detection method based on face key points

Abstract

The application discloses a high-efficiency fake face video detection method based on face key points, and comprises the following steps: 1) micro-expression face feature point collection; 2) video frame feature point tracking; and 3) true and false face feature point optical flow classification. The method improves the detection speed, and the detection precision is 0.99 in a public fake face dataset facefrensic++.

Description

Technical Field

[0001] This invention belongs to the field of cyberspace content security and fake face detection, and relates to a generative method for detecting fake face videos, specifically a high-efficiency method for detecting fake face videos based on facial key points. Background Technology

[0002] With the widespread adoption of mobile payments, more and more merchants are using facial recognition for payment collection. This method is more convenient than scanning QR codes, and payments can even be completed without carrying a mobile phone. While technology brings convenience, it also brings payment risks. Currently, generative adversarial networks can generate fake videos and photos that are indistinguishable from real faces to the naked eye. Criminals are using other people's photos to generate fake facial videos to maliciously attack facial recognition systems, posing significant security risks to various facial authentication scenarios. Therefore, research on facial anti-fraud technology is essential.

[0003] Currently, typical face fraud methods mainly include six types: printing, playing videos, partially obscuring images with other portraits, soft plastic masks, head mold sculptures, and noisy accessories. With the development of generative adversarial neural networks (GANs), a new type of face fraud method has emerged in recent years. This method uses GANs to generate fake faces that are difficult to distinguish from the naked eye. The realistic face images or videos generated by this new method are more difficult to detect than typical face fraud methods. Typical face fraud detection methods mainly utilize image analysis based on texture information, pixel color difference, high and low frequency image features, material classification, hardware multimodal assisted detection, semi-supervised and unsupervised learning, and deep learning. The new face fraud detection methods are mainly divided into those for detecting real and fake faces in single images and those for detecting real and fake faces in video images. In the detection of single images, McCloskey et al. (Scott McCloskey and Michael Albright. Detecting gangenerated imagery using color cues[J]. arXiv preprint arXiv: 1812.08247,2018.) proposed using color cubes to distinguish between real and generated faces. Since generated faces are generated by pixel synthesis, while real face images are generated optically through a camera, their spectral sensitivities are different, thus distinguishing between generated and real faces. Goebel et al. (Goebel M, Nataraj L, Nanjundaswamy T, Mohammed TM, Chandrasekaran S, Manjunath B S. Detection, attribute and localization of GAN generated images[J]. Electronic Imaging,2021(4): 276-1-276-11) proposed a GAN-generated image detection method that combines image features with deep learning methods. By calculating the co-occurrence matrix of neighboring pixels in different directions of the RGB channels for each image, the features of the co-occurrence matrix are learned through deep learning.Building upon the co-occurrence matrix, Barni et al. (Barni M, Kallas K, Nowroozi E and Tondi B. 2020. CNN detection of GAN-generated face images based on cross-band co-occurrences analysis[C]. Proceedings of 2020 IEEE International Workshop on Information Forensics and Security. New York, USA: IEEE: 1-6) proposed a co-occurrence matrix spanning three different color channels, which, compared to feature learning across a single color channel, improves the richness of image features. In generative face detection in video images, LI et al. (Yuezun Li, Ming-Ching Chang, and Siwei Lyu. In ictu oculi: Exposing ai generated fake face videos by detecting eye blinking[J]. arXiv preprint arXiv:1806.02877, 2018.) determine whether a face in a video is a real face or a generated fake face by detecting whether the face blinks in the video image. Agarwal et al. (Protecting world leaders against deep fakes[J]. In IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 38-45, 2019.) found that different people have different speaking habits, and their head movements, lips, and micro-expressions usually have their own style. The regularity of facial motion units of fake faces is quite different from that of real facial motion units. By judging whether the speaking pattern of a person is consistent with the learned real facial video feature regularity, it is possible to determine whether the video is fake, thereby detecting whether the speeches of leaders from various countries have been faked.

[0004] In fact, generative faces have already achieved extremely high accuracy and resolution, with very little difference from real faces in terms of detail. For example, the face generation model TransEditor proposed by SenseTime (TransEditor: Transformer-Based Dual-Space GAN for Highly Controllable Facial Editing[C].IEEE Conference on Computer Vision and Pattern Recognition, 2022) can generate faces with complex attributes. The generated faces can make different poses, change hair, smile, etc. Therefore, for highly realistic generative faces, further anti-spoofing research is needed on the detailed information of the face. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing an efficient method for detecting forged face videos based on facial key points. This method improves the detection accuracy of generative fake faces, resulting in more detailed detection.

[0006] The technical solution to achieve the purpose of this invention patent is:

[0007] An efficient method for detecting fake face videos based on facial key points includes the following steps:

[0008] 1) Train the video dataset by inputting a video dataset containing real human face images and a video dataset containing fake human face images;

[0009] 2) The facial video input in step 1) is processed by extracting facial landmarks from each frame of the facial image using Google's open-source facial landmark detection model, MediaPipe.

[0010] 3) Since the fluctuation of facial key points is different between fake faces and real faces, the movement between frames of facial key points obtained in step 2) is tracked, and the optical flow of facial key points is calculated by the LK optical flow algorithm to describe the dynamics of facial key points.

[0011] 4) Since the LK optical flow method introduces noise that interferes with the stability of image feature marking, a Denoising Diffusion Probabilistic Model (DDPM) is designed to integrate the information from the detection and prediction of facial key points in step 3) to obtain the denoised facial feature sequence.

[0012] 5) Input the denoised face feature sequence from step 4) into the RNN neural network for fake video classification, and predict real and fake faces through a fully connected layer.

[0013] In step 1), the video dataset used is the publicly available anti-spoofing face detection dataset facefrensic++ proposed by Andreas, which includes real face videos and generative fake face videos corresponding to real face videos.

[0014] The step 2) of extracting facial key points for each frame is as follows:

[0015] 2.1) First, the input video data is split into frames. The MediaPipe algorithm package is imported to introduce the FaceDetection facial landmark detection function to detect facial landmarks in each frame. The confidence level of the facial landmark detection is set to 0.5. Each landmark is composed of position coordinates (x, y). The FaceDetection function captures facial landmarks and outlines the distinctive facial contours. Affine transformation is used to align the facial landmarks to the preset positions.

[0016] The steps in step 3) involving tracking facial landmarks and calculating the optical flow of these landmarks using the LK optical flow algorithm are as follows:

[0017] 3.1) Define each frame of the face image mentioned in step 2) as... , The image of the next frame is defined as Each frame of facial images Small feature blocks in an image are defined as Define a small image feature block of the same size as the next frame as... The core of the LK optical flow method is to track facial landmarks, which requires making facial feature blocks... and pixel displacement between Minimum;

[0018] 3.2) will The center point is defined as = From step 3.1), we obtain The tracking prediction points inside Therefore, the displacement vector is represented by formula (1).

[0019]

[0020] In formula (1), the pixel displacement d is initialized to ,pass To update and iterate d, d←d+ Until d converges, Representative with The set of all small feature blocks centered at point . exp is an exponential function with the natural constant e as its base. This is used to reduce the weights of locations far from the center to make a smoother prediction;

[0021] 3.3) This is achieved through formula (2). ,

[0022]

[0023] in It is a Hessian ma matrix. It is by The matrix generated by vertical connection, It is a Jacobian matrix, obtained by weighting x in J using the Jacobian matrix. C represents each small feature block The number of channels, A is a diagonal matrix whose elements are... Composition, thereby through Update d to predict the next frame. middle center point .

[0024] The step in step 4) to design the denoising diffusion probability model to obtain the denoised face feature sequence is as follows:

[0025] 4.1) Prediction point obtained by the LK optical flow method described in step 3). Sequence using functions The predicted values ​​are then denoised using a denoised diffusion probability model.

[0026] 4.2) A Markov chain is used for the forward propagation of the denoising diffusion probability model. This dynamic process is represented by formula (3).

[0027]

[0028] in yes middle center point , T represents the total number of frames, and the amount of noise added in each step is defined using a variance table. , Represents the normal distribution function. Represents the prior probability. ), I represents the pixel block of the small feature map of each frame of the image;

[0029] 4.3) The back propagation process is represented by formula (4),

[0030]

[0031] Represents the posterior probability, for , , Represents a constant parameter. and , The function formula is as follows:

[0032]

[0033] 4.4) The loss function of the designed denoising diffusion probability model is shown in formula (6).

[0034]

[0035] in This represents the KL divergence.

[0036] The steps in step 5) for classifying facial feature sequences and predicting real and fake faces are as follows:

[0037] 5.1) Embed the calibrated facial landmark features from step 3) and the predicted facial landmark feature sequence after denoising in step 4) into... and The two sets of feature vector sequences are then input into a two-stream RNN for pseudo-video classification.

[0038] 5.2) Each facial key point uses express, = Therefore, the first type of eigenvector is defined as The facial landmark sequence is represented as ,

[0039]

[0040] Viewed as a sequence Perform a flattening operation, where n is the number of facial key points in a frame of an image;

[0041] 5.3) The second type of eigenvector is defined as follows: The formula is as follows:

[0042]

[0043] Indicates the difference in landmark positions between consecutive frames;

[0044] 5.4) Two feature vectors A are obtained by embedding them into two RNN neural network models. B= One RNN models facial shape motion patterns on A, and another RNN models facial marker difference patterns on B, to capture temporal discontinuities.

[0045] 5.5) Add a fully connected layer to the output of each RNN to make its own prediction. The average of A and B of the two streams is used as the final prediction. These prediction operations are reduced to a function f(·,·) as shown in the following formula.

[0046] (9),

[0047] This represents the first operation in an RNN neural network. Represents the second RNN neural network operation, so the probability of the face in the final predicted video segment being real or fake is obtained by formula (9).

[0048] Compared with existing methods, the present technical solution has the following advantages:

[0049] 1. The method in this technical solution uses facial key points to predict real and fake faces, which has higher accuracy in detecting real and fake faces;

[0050] 2. The LK optical flow algorithm in this technical solution can effectively predict the marker points in the next frame of the image by calculating the optical flow of facial key points, thus enabling accurate tracking of facial feature key points;

[0051] 3. The method of this technical solution uses a denoising diffusion probability model to obtain the denoised face feature sequence, which can improve the accuracy of face key points.

[0052] 4. The method of this technical solution uses two RNNs to train facial key point features. One RNN learns the facial shape movement pattern, and the other RNN learns the facial marker difference pattern to capture the discontinuity of time.

[0053] This method offers high accuracy and more detailed detection of generative fake faces in videos. Attached Figure Description

[0054] Figure 1 Flowchart for an embodiment;

[0055] Figure 2 A schematic diagram of facial key point extraction in the embodiment;

[0056] Figure 3 This is a schematic diagram of the detection results of the FaceForensic++ dataset in the example. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0058] Example:

[0059] Reference Figure 1 A highly efficient method for detecting fake face videos based on facial key points, comprising the following steps:

[0060] 1) Train the video dataset by inputting a video dataset containing real human face images and a video dataset containing fake human face images;

[0061] 2) The facial video input in step 1) is processed using Google's open-source facial landmark detection model, MediaPipe (https: / / github.com / google / mediapipe), to extract facial landmarks from each frame of the facial image;

[0062] 3) Since the fluctuation of facial key points is different between fake faces and real faces, the movement between frames of facial key points obtained in step 2) is tracked, and the optical flow of facial key points is calculated by the LK optical flow algorithm to describe the dynamics of facial key points.

[0063] 4) Since the LK optical flow method introduces noise that interferes with the stability of image feature marking, a denoising diffusion probability model DDPM is designed to integrate the information from the detection and prediction of facial key points in step 3) to obtain the denoised facial feature sequence.

[0064] 5) Input the denoised face feature sequence from step 4) into the RNN neural network for fake video classification, and predict real and fake faces through a fully connected layer.

[0065] In step 1), the video dataset used is the publicly available anti-spoofing face detection dataset facefrensic++ (FF++) proposed by Andreas et al. (Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images[J]. arXiv preprint arXiv:1901.08971, 2019.). It contains 1000 datasets, including real face videos and 1000 generative fake face videos corresponding to real face videos.

[0066] The step 2) of extracting facial key points for each frame is as follows:

[0067] 2.1) First, the input video data is split into frames. The MediaPipe algorithm package is imported, and the FaceDetection facial landmark detection function is used to detect facial landmarks in each frame. The confidence level for facial landmark detection is set to 0.5. Each landmark is composed of position coordinates (x, y). The FaceDetection function captures a total of 468 facial landmarks, as shown below. Figure 2 As shown, the facial landmarks are outlined using facial key points, and affine transformations are used to align the facial key points to preset positions.

[0068] The steps in step 3) involving tracking facial landmarks and calculating the optical flow of these landmarks using the LK optical flow algorithm are as follows:

[0069] 3.1) Define each frame of the face image mentioned in step 2) as... , The image of the next frame is defined as Each frame of facial images Small feature blocks in an image are defined as Define a small image feature block of the same size as the next frame as... The core of the LK optical flow method is to track facial landmarks, which requires making facial feature blocks... and pixel displacement between Minimum;

[0070] 3.2) will The center point is defined as = From step 3.1), we obtain The tracking prediction points inside Therefore, the displacement vector is represented by formula (1).

[0071]

[0072] In formula (1), the pixel displacement d is initialized to ,pass To update and iterate d, d←d+ Until d converges, Representative with The set of all small feature blocks centered at point . exp is an exponential function with the natural constant e as its base. This is used to reduce the weights of locations far from the center to make a smoother prediction;

[0073] 3.3) This is achieved through formula (2). ,

[0074]

[0075] in It is a Hessian ma matrix. It is by The matrix generated by vertical connection, It is a Jacobian matrix, obtained by weighting x in J using the Jacobian matrix. C represents each small feature block The number of channels, A is a diagonal matrix whose elements are... Composition, thereby through Update d to predict the next frame. middle center point .

[0076] The step in step 4) to design the denoising diffusion probability model to obtain the denoised face feature sequence is as follows:

[0077] 4.1) Prediction points obtained by the LK optical flow method described in step 3). Sequence using functions The predicted values ​​are then denoised using a denoised diffusion probability model.

[0078] 4.2) A Markov chain is used for the forward propagation of the denoising diffusion probability model. This dynamic process is represented by formula (3).

[0079]

[0080] in yes middle center point , T represents the total number of frames, and the amount of noise added in each step is defined using a variance table. , Represents the normal distribution function. Represents the prior probability. ), I represents the pixel block of the small feature map of each frame of the image;

[0081] 4.3) The back propagation process is represented by formula (4),

[0082]

[0083] Represents the posterior probability, for , , Represents a constant parameter. and , The function formula is as follows:

[0084]

[0085] 4.4) The loss function of the designed denoising diffusion probability model is shown in formula (6).

[0086]

[0087] in This represents the KL divergence.

[0088] The steps in step 5) for classifying facial feature sequences and predicting real and fake faces are as follows:

[0089] 5.1) Embed the calibrated facial landmark features from step 3) and the predicted facial landmark feature sequence after denoising in step 4) into... and The two sets of feature vector sequences are then input into a two-stream RNN for pseudo-video classification.

[0090] 5.2) Each facial key point uses express, = Therefore, the first type of eigenvector is defined as The facial landmark sequence is represented as ,

[0091]

[0092] Viewed as a sequence To perform a flattening operation, n is the number of facial key points in a frame of an image; in this example, n=468.

[0093] 5.3) The second type of eigenvector is defined as follows: The formula is as follows:

[0094]

[0095] Indicates the difference in landmark positions between consecutive frames;

[0096] 5.4) Two feature vectors A are obtained by embedding them into two RNN neural network models. B= One RNN models facial shape motion patterns on A, and another RNN models facial marker difference patterns on B, to capture temporal discontinuities.

[0097] 5.5) Add a fully connected layer to the output of each RNN to make its own prediction. The average of A and B of the two streams is used as the final prediction. These prediction operations are reduced to a function f(·,·) as shown in the following formula.

[0098] ) (9),

[0099] This represents the first operation in an RNN neural network. Represents the second RNN neural network operation, so the probability of the face in the final predicted video segment being real or fake is obtained by formula (9).

[0100] Performance evaluation:

[0101] Using the area under the ROC curve (AUC) as the evaluation metric, the method designed in this example achieved an accuracy of 99.91% on the FaceForensic++ dataset. Figure 3 As shown in Table 1, using the FaceForensic++ dataset under the same experimental environment, the method in this example is compared with existing fake face detection models Meso4, FWA, DSP-FWA, Xception, and Capsule. The detection results of real and fake faces in the video are shown in Table 1:

[0102] Table 1 Performance Comparison of Different Detection Models

[0103] .

[0104] As can be seen from the table above, the method presented in this example achieves the highest AUC result. This is because it uses a dense set of 468 facial key points as the learning object and employs a fuzzy probabilistic denoising model to remove interference. Compared to existing models that use only 68 facial key points, this method achieves higher accuracy and has a smaller memory footprint, resulting in greater portability.

Claims

1. An efficient fake face video detection method based on face key points, characterized in that, Comprising the following steps: 1) training by inputting a video dataset with real face images and a video dataset with fake face images; 2) extracting face key points from each frame of the face image by using the open-source face key point detection model mediapipe for the face video input in step 1); 3) tracking the movement between frames of the face key points obtained in step 2) and calculating the optical flow of the face key points by using the LK optical flow algorithm to describe the dynamics of the face key points; 4) designing a denoising diffusion probability model DDPM to integrate the information of face key point detection and prediction in step 3) to obtain a denoised face feature sequence; 5) inputting the denoised face feature sequence in step 4) into an RNN neural network for fake video classification and predicting real and fake faces through a fully connected layer; The steps of face feature sequence classification and real and fake face prediction are: 5.1) embedding the aligned face landmark feature of step 3) and the predicted face landmark feature sequence after denoising by step 4) into and two groups of feature vector sequences, and then inputting into the dual-stream RNN for pseudo video classification; 5.2) Each facial landmark adopts representations, , so the first feature vector is defined as The sequence of facial landmarks is represented as ,​ ， Considered from the sequence Flattening operation is performed, n is the number of facial key points in a frame of image; 5.3) The second feature vector is defined as , where the formula is as follows, ， representing a difference in consecutive inter-frame landmark positions; 5.4) By embedding in two RNN neural network models two feature vectors A= , B= , one RNN models the face shape motion pattern on A and the other RNN models the face landmark difference pattern on B to capture the discontinuity in time; 5.5) adding a fully connected layer to the output of each RNN to make its own prediction, and the average value of two streams A and B is used as the final prediction. These prediction operations are summarized as a function f(·,·) as shown in equation (9), ) (9) ， representing a first RNN neural network operation, representing a second RNN neural network operation, so the likelihood of the face being real or fake for the last predicted video segment is derived by equation (9).

2. The face key point based efficient fake face video detection method of claim 1, wherein, In step 1), the video dataset uses the public anti-fake face detection dataset facefrensic++ proposed by Andreas, which includes real face videos and corresponding generated fake face videos.

3. The face key point based efficient fake face video detection method of claim 1, wherein, The steps of extracting face key points in step 2) are: 2.1) first, split the input video data into frames, import the mediapipe algorithm package, and introduce the FaceDetection face key point detection function to detect the face key points in each frame of the image. The face key point detection confidence is set to 0.5, and each key point is composed of position coordinates (x, y). The face key points are captured by the FaceDetection function, and the face key points are used to outline the facial landmark contours. Affine transformation is used to align the face key points to the preset position.

4. The face key point based efficient fake face video detection method of claim 1, wherein, In step 3), the steps of tracking the face key points and calculating the optical flow of the face key points by using the LK optical flow algorithm are: 3.1) Define each frame of the face image mentioned in step 2) as... , The image of the next frame is defined as Each frame of facial images Small feature blocks in an image are defined as Define a small image feature block of the same size as the next frame as... The core of the LK optical flow method is to track facial landmarks, which requires making facial feature blocks... and pixel displacement between Minimum; 3.2) will The center point is defined as = From step 3.1), we obtain The tracking prediction points inside Therefore, the displacement vector is represented by formula (1). ， In formula (1), the pixel displacement d is initialized to ,pass To update and iterate d, d←d+ Until d converges, Representative with The set of all small feature blocks centered at point . exp is an exponential function with the natural constant e as its base. This is used to reduce the weights of locations far from the center to make a smoother prediction; 3.3) is achieved by equation (2) , ， in It is a Hessian ma matrix. It is by The matrix generated by vertical connection, It is a Jacobian matrix and ,right In By performing Jacobian matrix weighting, we obtain C represents each small feature block The number of channels, A is a diagonal matrix whose elements are... Composition, thereby through Update d to predict the next frame. middle center point .

5. The face key point based efficient fake face video detection method of claim 1, wherein, In step 4), the steps of designing a denoising diffusion probability model to obtain a denoised face feature sequence are: 4.1) the predicted point from the LK optical flow method of step 3) The sequence adopts a function is denoted, and the predicted value is denoised using a denoising diffusion probability model; 4.2) using Markov chain for forward propagation of the denoising diffusion probability model, and the dynamic process is represented by equation (3), ， wherein is in the center point , , T represents the total number of frames, and the amount of noise added at each step is defined by the variance table , represents a normal distribution function, represents a prior probability, ), I represents a small feature map pixel block of each frame image; 4.3) the backward propagation process is represented by equation (4), ， representing the posterior probability, for , , representing a constant parameter, and , the function formula is as follows, ， 4.4) the loss function of the designed denoising diffusion probability model is shown in equation (6), ， wherein denotes the KL divergence.

Images