A deepfake detection method based on face geometry relationship reasoning

Abstract

The application discloses a deep fake detection method based on face geometric relation reasoning, relates to the field of video passive forensics, and is used for improving the generalization ability of a deep fake detection model. First, a face feature point detector is used to detect face key point features, and an explicit face geometric relation graph is constructed according to the internal facial feature structure; a global feature extractor is constructed by using a Transformer to obtain a global feature map of a face image; on the basis of the global feature map of the face, a high-information-area positioning mechanism is positioned by using a self-supervised learning mechanism, and an implicit face geometric relation graph is constructed; a face internal geometric relation reasoning module is constructed by using a graph convolutional neural network, the explicit and implicit face geometric relation graphs are combined in features, and fake detection is performed on a video frame to be tested. The application can effectively improve the accuracy of face deep fake video detection, has good model generalization performance under different data domains, and has practical value.

Description

Technical Field

[0001] The present invention relates to the field of video forensics technology, and in particular to a deep fake detection method based on face geometric relationship reasoning. Background Art

[0002] AI-generated content, as an emerging technology, has been widely disseminated on social media. However, the widespread adoption of AI content generation technology has made facial video manipulation more accessible. Deepfakes, the first widely disseminated facial video manipulation technology, have evolved to the point where they can synthesize incredibly realistic facial videos, making them difficult for humans to discern. The misuse of these generated deepfakes poses serious risks. Therefore, deepfake video detection has become a crucial research issue in multimedia forensics within the information security field. Due to the minimal feature differences between real and forged faces, existing deep learning models are unable to detect any traces of manipulation. To improve detection effectiveness, existing deepfake methods based on deep learning utilize various feature attention mechanisms to enhance the network model's ability to discern subtle differences. However, the continuous advancement of deepfake methods has led to significant differences in the tampering traces generated by different deepfake models. Most existing methods are typically only applicable to a single or small number of tampering traces and have limited generalization performance. For example, the article "DeepFake Detection Based on Discrepancies Between Faces and Their Context," published in the prestigious journal IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 44, Issue 10, 2022, shows that this method achieved an AUC detection score of 99.7 for face images generated using the faseswap method. However, it struggles to detect face images generated by improved versions of deepfakes, with the AUC score dropping by nearly 35%. For emerging deepfake technologies, further design of new feature attention modules is necessary to achieve better detection results. However, in real-world applications such as social media, the need to cope with constantly updated deepfake videos creates a high cost for updating models. Therefore, existing technologies are unable to meet practical needs, and there is an urgent need to improve the generalization capabilities of model design. Summary of the Invention

[0003] The purpose of the present invention is to address the above limitations and provide a deep fake detection method based on facial geometric relationship reasoning to further improve the generalization ability of deep fake video detection.

[0004] The technical solutions for achieving the purpose of the present invention are as follows:

[0005] A deepfake detection method based on facial geometric relationship reasoning uses Transformer to build a global feature extractor to obtain a global feature map of the face image. Based on the global feature map, a self-supervised learning mechanism is used to locate high-information areas and construct an implicit geometric relationship map of the face. A graph convolutional neural network is used to build an internal face geometric relationship reasoning module, which combines the features of the explicit and implicit geometric relationship maps of the face to detect forgeries in the test video frame, thereby improving the accuracy of deepfake face detection. The method includes the following main steps:

[0006] Step 1: Obtain a training video containing both real and fake samples, perform interval sampling on the video frames, and use a face detector to extract the face and a small amount of background area frame by frame as the face image;

[0007] Step 2: For each face image frame, use a face key point detector to detect 468 3D key points of the face, and construct an explicit geometric relationship graph of the face based on the facial relationships;

[0008] Step 3: Construct a global feature extractor to extract the global features of each frame of face image;

[0009] Step 4: Construct a high-information region locator based on a self-supervised learning method to locate feature regions with higher information content in the global features and construct an implicit geometric relationship graph of the face between the feature regions;

[0010] Step 5: Construct a facial geometric relationship reasoning module, extract explicit and implicit facial geometric relationship graph features based on a graph convolutional neural network, and match graph node relationships;

[0011] Step 6: Input the continuous face images in the continuous frames of the test set into the trained model to obtain the probability score of the predicted authenticity, and average the scores of the entire video frame to determine the authenticity of the video.

[0012] Furthermore, in step 1, the specific method of obtaining the face image is:

[0013] (1) Preset a sampling interval of video frames, starting from the first frame of the video, and extract video frames from the video at the sampling interval;

[0014] (2) Use the RetinaFace face detector to determine the face candidate frame in the current video frame and establish a Cartesian coordinate system in the upper left corner of the image. The candidate frame coordinates can be expressed as (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the candidate frame, and w, h represent the length and width of the candidate frame respectively;

[0015] (3) The candidate frame range is expanded according to the preset ratio r, and the face candidate frame becomes (xr×w,yr×h,w+r×w,h+r×h), and the image within the candidate frame range is captured as the face image.

[0016] Furthermore, in step 2, the specific method of constructing the face explicit geometric relationship graph is:

[0017] (1) Inputting the facial image into a pre-trained facial 3D key point extractor Mediapipe to obtain 468 facial key points;

[0018] (2) The facial key points are used as nodes V of the explicit geometric relationship graph L According to the location of the key points in the face, the nodes of the eyebrows, pupils, eye sockets, lips and facial contours are connected in sequence to form the appearance contour. Then, according to the geometric structure of the face, the nodes of the five parts are connected to each other to form the face explicit geometric relationship graph G L .

[0019] Furthermore, in step 3, the global feature extractor is MobileVit or other backbone network implemented based on VisionTransformer.

[0020] Furthermore, in step 4, the specific method of the high-information region locator based on the self-supervised learning method is:

[0021] (1) In the global face feature map, M regions of interest are obtained using a region extraction network;

[0022] (2) Input the local features of the region of interest into a simple binary classifier f p In the classifier, a 1×1 convolution kernel is used to reduce the number of local feature channels to 2 channels, and then an activation function and a batch normalization layer are used to increase the nonlinear expression ability of the features to obtain local semantic information features. Using global average pooling The length and width are downsampled to 1;

[0023] (3) Using the cross entropy loss function L bce Calculate the loss value l between the binary classification result of each local feature and the authenticity of the current face p ;

[0024] (4) Arranging all loss values ​​obtained from each image from largest to smallest;

[0025] (5) The loss value l obtained for each local feature P and the corresponding region proposal score S PCalculate the square difference between them, and calculate the mean after obtaining all the square differences, which is used as the loss function L for self-supervised learning. self .

[0026] Furthermore, in step 4, the implicit geometric relationship graph G of the face is constructed P The specific method is:

[0027] (1) According to the information score S of the high information region locator P , select the N local features F with the highest scores in the high-information region P As a set, each F P The feature matrix of size 2×W×H is converted into a feature vector of size T×2, where T=W×H, and the converted feature vector set is used as the node of the implicit geometric relationship graph of the face

[0028] (2) Convert the size of the node set N×T×2 to N×C P , where C P =T×2, using SoftMax operation to convert the node set into the attention vector V att =softmax(V P );

[0029] (3) Based on the self-attention mechanism, calculate V P and the attention vector transposed The relationship between the nodes of the implicit geometric relationship graph is obtained by Its size is N×N.

[0030] Furthermore, in step 5, the geometric relationship reasoning module comprises an explicit face geometric feature reasoning module, an implicit face geometric feature reasoning module, a graph feature matching module, and a graph classifier:

[0031] (1) The explicit face geometric feature reasoning module uses a point cloud analysis model built based on a graph convolutional neural network to generate an explicit face geometric relationship graph G L Based on this, we obtain the graph feature expression G of the explicit face geometric relationship gr ;

[0032] (2) The implicit face geometric feature reasoning module uses a two-layer graph convolutional network model to generate the implicit geometric relationship graph G P Based on the implicit face geometric relationship, the graph feature expression G is obtained. ir ;

[0033] (3) The graph feature matching module uses a two-layer interactive graph convolutional network model to gr and G irBased on this, the multi-perspective geometric relationships are integrated to obtain the fused geometric relationship graph G F ;

[0034] (4) The graph classifier uses global average pooling and global maximum pooling to obtain the maximum value and average value of graph node features respectively, fuses the graph representation features from two observation angles, and performs classification based on a multi-layer perceptron.

[0035] In implementation, the specific method of the graph feature matching module is as follows:

[0036] (1) Based on the geometric relationship graph G gr and G ir Contains the graph node feature set V gr and V ir , use the mutual attention mechanism to calculate V gr and V ir Transpose The pairwise relationship of G gr Connect G ir The adjacency matrix Its size is N×N;

[0037] (2) Based on the geometric relationship graph G gr and G ir Contains the graph node feature set V gr and V ir , use the mutual attention mechanism to calculate V ir and V gr Transpose The pairwise relationship of G gr Connect G ir The adjacency matrix Its size is N×N;

[0038] (3) Based on the graph node feature set V gr and V ir , the adjacency matrix A gi and A ig ,Use an interactive graph convolutional network model to match graph node features and infer the geometric anomalies in deep fake face images.

[0039] In implementation, the calculation process of the interactive graph convolutional network model is as follows:

[0040] (1) Extraction of G gr Connect G ir Implicit feature relationship graph node feature expression:

[0041] V gi =σ(W1×σ(A gi V gr W gi )+V gr);

[0042] Where W1 and W gi These are all learnable parameters of the interactive graph convolutional network. σ(·) represents the nonlinear activation function, which is ReLu or Leaky ReLu function.

[0043] (2) Extraction of G ir Connect G gr Implicit feature relationship graph node feature expression:

[0044] V ig =σ(W2×σ(A ig V ir W ig )+V ir );

[0045] Among them W2 and W ig These are all learnable parameters of the interactive graph convolutional network. σ(·) represents the nonlinear activation function, which is ReLu or Leaky ReLu function.

[0046] (3) Express the two graph node features V gi and V ig Splicing,obtains the node feature expression of multi-angle geometric relationship,graph.

[0047] In practice, the total loss function for training in step 5 is:

[0048]

[0049] in, is the binary cross entropy loss function and uses label smoothing technology. is the self-supervised loss function. P and P a They are the classification results of local high-information regions and the classification results of global geometric relationship graph node features, S P Region proposal score for high-information region localizer.

[0050] In step 6, the trained model is used to make predictions for all video frames, and all scores are averaged as the prediction result of the authenticity of the face in the video.

[0051] This invention uses deep learning technology to ensure the security of video content containing human faces. It extracts features from the explicit and implicit geometric relationships of the face and infers anomalies based on the inherent geometric structure of the face to determine whether the current image is forged.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention uses three-dimensional facial key points to construct a facial geometric relationship graph as the inherent features of the face, and uses a graph convolutional neural network to extract relationship features.

[0054] 2. Use the Transformer network to extract global features and use self-supervised learning methods to locate high-information areas, regardless of the forgery method, and use the self-attention mechanism to construct an implicit geometric relationship graph.

[0055] 3. Matching implicit and explicit geometric relationship features and inferring facial geometric anomalies effectively avoids relying on specific forgery traces and effectively improves the generalization ability of deep forgery detection models. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of an embodiment of the present invention.

[0057] Figure 2 This is a network structure diagram of an embodiment of the present invention.

[0058] Figure 3 This is the explicit geometric graph of the face according to an embodiment of the present invention.

[0059] Figure 4 The experimental results of the present invention are compared with those of existing methods in a public verification example.

[0060] Figure 5 This is a detection effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0062] like Figures 1-2 As shown, an embodiment of the present invention constructs a deep fake detection network based on facial geometric relationship reasoning, which includes a Visinon Transformer backbone network, a high-information region locator, a high-information region classifier, a point cloud analysis network, a graph convolution module, an interactive graph convolution module and a graph classifier, constituting the entire model framework. Figure 1 Provided is the workflow of the present invention; Figure 2 A specific network structure diagram of the present invention in one embodiment is given.

[0063] Step 1: Obtain a training video containing both real and fake samples, perform interval sampling on the video frames, use a face detector to locate and sample the face in the video frame, and crop the face and a small portion of the surrounding background area as the face image. In some specific implementations, RetinaFace, whose backbone network is Resnet50, is used as the face detector. The detection frame is expanded to 0.1 times the length and width of the detection result, and the range of the four positioning coordinates is used as the face detection result. For each frame of the face image, use the key points provided by RetinaFace to align all images to a uniform size based on the facial key points of the eyes, nose, and mouth corners.

[0064] Step 2: Use a 3D facial key point detector to extract 3D facial key points and construct an explicit geometric relationship graph G containing facial features position information and facial contour information based on the facial relationship. L ;

[0065] In some specific implementation scenarios, using MediaPipe to detect faces can obtain 468 3D facial key points;

[0066] Step 3: Construct a global feature extractor based on Vision Transformer to extract global facial features. In some specific embodiments, the backbone network uses MobileVit, uses pre-trained parameters from the ImageNet dataset as model initialization parameters, retains the first downsampling and subsequent five feature extraction stages of the model, and extracts global features for each frame of the facial image.

[0067] Step 4: Construct a high-information region locator based on a self-supervised learning method, using a region proposal network constructed by convolutional layers and fully connected layers to provide the information score S of each region in the global feature map. P And the corresponding coordinates, each region is sent to a simple binary classifier constructed by a 1×1 convolutional layer to obtain a two-channel semantic feature map containing the image spatial semantic information The global average pooling layer is used to provide the probability that the current area belongs to a fake image. The probability and the label of the current face image are combined through the binary cross entropy loss function to obtain the corresponding loss. The loss value and the information score S are calculated. P The squared difference is used as the self-supervised learning loss function to ensure that the simple local two-classification loss is higher, that is, the area with higher uncertainty is used as the detection result of the high-information area locator. The high-information area features obtained by the detection are used as graph representation feature nodes, and the two-channel semantic feature map of the corresponding area is used The graph adjacency matrix is ​​jointly constructed to form the implicit geometric relationship graph of the face between feature regions.

[0068] In some specific implementations, the high-information region locator will provide 20 candidate regions during training, which are pooled into a uniform size, such as 7×7, using the region of interest. The corresponding binary cross entropy loss is calculated for each candidate region, and the squared difference between the loss and the proposal scores of the 20 candidate regions is calculated. The training of the module is completed by minimizing the binary cross entropy loss and the supervision loss. The two-channel semantic features of the 6 regions with the highest candidate scores are selected. Each region feature can be stretched into a vector. The six feature vectors are concatenated to form a node in the implicit geometric relationship graph of the face. Based on the self-attention mechanism, the adjacency matrix of the implicit geometric relationship graph of the face is Size is 6×6;

[0069] Step 5: Construct a face geometry relationship reasoning module and represent the face explicit and implicit geometry relationship graph G L and G P The graph feature representation is enhanced by two neural networks composed of graph convolution layers to obtain the enhanced geometric relationship graph representation G gr and G ir . Build a multi-layer interactive graph convolutional network model to match G gr and G ir The graph node feature relationship, enhanced geometric relationship graph representation, and the enhanced features are fused into G F , highlighting geometric abnormal feature nodes, using the fully connected layer to convert the graph node feature channels into binary classification channels, and using global average pooling and global maximum pooling to calculate the average binary classification prediction value of all nodes and the maximum binary classification prediction value of all nodes respectively, adding the prediction values ​​as the final classification result, outputting the probability that the current image is a forged image, and using cross entropy loss to calculate the classification loss;

[0070] In some specific implementation scenarios, for the face explicit geometric relationship graph representation G L , the number of channels is 3, the number of nodes is 468, the graph feature enhancement network is composed of a point cloud analysis network based on a graph convolutional network, the backbone network uses CurveNet, and the number of graph nodes is downsampled to G P The number is the same, while deepening the feature representation dimension is to enhance the relationship representation ability, the feature dimension is the same as G P Consistent. Implicit geometric relationship graph representation G P , the number of channels is 98, the number of nodes is 6, and the enhanced network consists of two layers of simple graph convolutional networks combined with nonlinear activation functions to maintain the number of nodes and channels in the original graph.

[0071] The specific steps of interactive graph convolutional network calculation are: 1) input two graph representations G1 and G2 to be matched; 2) calculate the adjacency matrix connecting G1 to G2 Where V1 and V2 are the node features of the corresponding graph; 3) Enhance the G1 graph representation capability and update the graph node features, Where W1 and W 1→2 are all learnable parameters of the interactive graph convolutional network, σ(·) represents the nonlinear activation function; 4) Calculate the adjacency matrix connecting G2 to G1 5) Enhance the G2 graph representation capability and update the graph node features. Among them W2 and W 2→1 are all learnable parameters of interactive graph convolutional networks, σ(·) represents the nonlinear activation function;

[0072] In some specific implementations, the nonlinear activation function is a ReLu or Leaky ReLu function.

[0073] In some specific implementation scenarios, feature nodes enhanced by the interactive graph convolutional network model can be fused by splicing or adding.

[0074] In some specific implementations, the cross entropy loss function can use label smoothing regularization constraints, specifically:

[0075]

[0076] Where y∈[0+α,1-α] is the true label, 0 represents the real image, 1 represents the forged image, and α represents the label smoothing parameter. is the predicted probability value.

[0077] In some specific implementation scenarios, the final loss function of the network constructed in steps 1 to 5 is shown as follows:

[0078]

[0079] in For the binary cross entropy loss function, label smoothing technology can be used to replace it with is the self-supervised loss function. P and P a They are the classification results of local high-information regions and the classification results of global geometric relationship graph node features, S P Region proposal score for high-information region localizer.

[0080] Step 6: Input the continuous face images in the continuous frames of the test set into the trained model, output the probability score of the corresponding frame being true or false, and average the probability scores obtained for all frames corresponding to the video to determine the authenticity of the video.

[0081] Example

[0082] This embodiment includes the following steps:

[0083] S1: Collect training samples;

[0084] S1.1: Input video. For each input video, use the RetinaFace face detector with a Resnet50 backbone network to detect the face position in each frame. During the training phase, use interval sampling to retain a number L of face images.

[0085] S1.2: Align the face images to a uniform size of 380×380 based on the key points provided by RetinaFace, including the center of the eyes, the left and right corners of the mouth, and the center of the face.

[0086] S1.3: Each face image is assigned a category label for the corresponding video, 0 represents a real video, and 1 represents a fake video.

[0087] S2: Build Figure 3 The explicit geometric relationship graph G of the face shown L ;

[0088] S2.1: Use MediaPipe to obtain the 3D coordinates of 468 facial key points;

[0089] S2.2: Connect key points around the eyes, lips, nose, and facial contours based on their locations, and connect each part based on their distribution across facial regions.

[0090] S3: Construct a global feature extractor based on Vision Transformer to extract global facial features;

[0091] S3.1: Using the lightweight MobileVit network as a global feature extractor helps reduce the computational complexity of the algorithm and facilitates the practical application of the present invention. The first convolutional downsampling module of the backbone network and the subsequent five basic modules of MobileViT are selected as feature extractors;

[0092] S3.2: Input a face image of size 380×380 and obtain a global feature map F of size 11×11 a ;

[0093] S4: Construct a high-information region locator based on self-supervised learning method, obtain the 6 regions with the highest information content, and construct the implicit geometric relationship graph G of the face P .

[0094] like Figure 1 and Figure 2 As shown, the implicit geometric relationship graph G of the face P The specific steps are:

[0095] S4.1: Using the region proposal network commonly used in two-stage target detectors as a high-information region locator, based on the anchor box approach, in the global feature map F a Provide 20 candidate boxes of the region of interest, and the candidate scores corresponding to the candidate boxes are regarded as information scores;

[0096] S4.2: Use the region of interest pooling method to unify the size of the features at the location of the candidate box to 7×7;

[0097] S4.3: Send the 20 region of interest features into the 1×1 convolution layer, change the feature channel of each region to 2, and obtain the two-channel semantic features Its size is 7×7×2, and then global flat pooling is used to downsample the size of each feature map to obtain the true or false prediction probability score;

[0098] S4.4: Calculate the binary cross entropy loss between the predicted probability score of each region of interest and the corresponding face image label;

[0099] S4.5: Calculate the square difference between the cross entropy loss and the candidate score of the candidate box. The area with higher candidate scores corresponds to a higher local classification loss, which means that the corresponding uncertainty is higher, or it is called higher uncertainty.

[0100] S4.6: Select the six region features with the highest candidate scores as the high-information regions of the current image;

[0101] S4.7: Each regional feature is stretched into a vector and spliced ​​to form the implicit geometric relationship graph node of the face

[0102] S4.8 calculates the connection relationship between nodes based on the self-attention mechanism and constructs the graph adjacency matrix as Size is 6×6;

[0103] S5: Construct a facial geometric relationship reasoning module to analyze and reason about facial geometric anomalies;

[0104] S5.1: Use CurveNet based on graph convolutional network to remove the classifier as the point cloud analysis network Ψ(·);

[0105] S5.2: Input face explicit geometric relationship graph G L , use the point cloud analysis network to analyze the explicit geometric relationship of the face, extract the 6 most important feature nodes, and output the feature enhanced relationship graph G gr =Ψ(G L ), the output feature dimension is 96;

[0106] S5.3: Use graph convolution layers and nonlinear activation functions to form a graph convolution module, and use two layers of graph convolution modules as the face implicit geometric relationship graph enhancement network Ω(·);

[0107] S5.4: Input face implicit geometry G P , use the implicit geometric relationship graph of the face to enhance the network to enhance the feature representation, and output the enhanced relationship graph G ir =Ω(G P )

[0108] S5.5: Use interactive graph convolution layers and nonlinear activation functions to form an interactive graph convolution module, and use two layers of interactive graph convolution modules as the geometric relationship graph matching network Φ(·);

[0109] S5.6: Input geometric relationship diagram G gr and G ir , obtain the node matching relationship reasoning graph (G gi G ig )=Φ(G gr ,G ir ) and the corresponding feature node V gi and V ig ;

[0110] S5.7: Splice according to channel direction or fuse node features

[0111] S5.8: Use the fully connected layer to fuse the node features V F The feature channel is converted to

[0112] S5.8: Use global average pooling to obtain the average node classification score P1∈[0,1];

[0113] S5.8: Use global max pooling to obtain the maximum node classification score P2∈[0,1];

[0114] S5.9: As the true or false probability score obtained by geometric anomaly reasoning, the binary cross entropy loss is calculated with the image label;

[0115] S6: Obtain the video to be tested, use a face detector to detect all face images contained in the video, input the images into the trained model in sequence, obtain the authenticity probability score of each image, calculate the average prediction score of all images in the video, and obtain the authenticity probability score of the video;

[0116] This example uses the area under the ROC curve (AUC) as an evaluation metric. The ROC curve is composed of the true positive rate (TPR) and the false positive rate (FPR) as the horizontal and vertical coordinates, respectively. The true positive rate refers to the proportion of samples that are correctly predicted as positive among samples that are actually positive; the false positive rate refers to the number of samples that are incorrectly predicted as negative among samples that are actually negative. The closer the AUC score is to 1, the better the model performance. The AUC score is not affected by the classifier threshold setting and is a robust evaluation metric.

[0117] Figure 4 This is a comparison of the experimental results of the present invention with existing methods in a public verification example. This embodiment chooses to use the high-quality (C23) dataset in the FaceForensices++ (FF++) dataset to train the model, and verifies the effectiveness of the method on the FF++ (C23) and CelebDF v2 dataset test sets. The results show that the method proposed in the present invention performs well on both the FF++ (C23) and CelebDF v2 public datasets. While maintaining a high detection performance in the data domain, it effectively improves the detection performance for unknown deep fake methods. Compared with the comparison algorithm, it has better detection effect.

[0118] Figure 5 This is a detection effect diagram of the verification example of the present invention on different data sets, which specifically shows the explicit geometric relationship diagram, implicit geometric relationship diagram and corresponding feature Grad-CAM focus area effect of the method proposed in the present invention.

[0119] The above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to fall within the scope of protection of the present invention.

Claims

1. A deepfake detection method based on facial geometric relationship reasoning. This method uses Transformer to construct a global feature extractor to obtain a global facial feature map. Based on this global facial feature map, a self-supervised learning mechanism is used to locate high-information areas and construct an implicit facial geometric relationship map. A graph convolutional neural network is used to construct an internal facial geometric relationship reasoning module. The explicit and implicit facial geometric relationship maps are combined to perform deepfake detection on the test video frame. The method includes the following main steps: Step 1: Obtain a training video containing both real and fake samples, perform interval sampling on the video frames, and use a face detector to extract the face and a small amount of background area frame by frame as the face image; Step 2: For each face image frame, use a face key point detector to detect 468 3D key points of the face, and construct an explicit geometric relationship graph of the face based on the facial relationships; Step 3: Construct a global feature extractor to extract the global features of each frame of face image; Step 4: Construct a high-information region locator based on a self-supervised learning method to locate feature regions with higher information content in the global features and construct an implicit geometric relationship graph of the face between the feature regions; Step 5: Construct a facial geometric relationship reasoning module, extract explicit and implicit facial geometric relationship graph features based on a graph convolutional neural network, and match graph node relationships; Step 6: Input the continuous face images in the continuous frames of the test set into the model trained by steps 1 to 5 to obtain the probability score of the predicted authenticity, and average the scores of the entire video frame to determine the authenticity of the video.

2. A deep fake detection method based on face geometric relationship reasoning as claimed in claim 1, characterized in that In step 1, the specific method of obtaining the face image is: (1) Preset a sampling interval of video frames, starting from the first frame of the video, and extract video frames from the video at the sampling interval; (2) Use the RetinaFace face detector to determine the face candidate frame in the current video frame and establish a Cartesian coordinate system in the upper left corner of the image. The candidate frame coordinates can be expressed as (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the candidate frame, and w, h represent the length and width of the candidate frame respectively; (3) The candidate frame range is expanded according to the preset ratio r, and the face candidate frame becomes (xr×w,yr×h,w+r×w,h+r×h), and the image within the candidate frame range is captured as the face image.

3. The deep fake detection method based on face geometric relationship reasoning according to claim 1, characterized in that: In step 2, the specific method of constructing the face explicit geometric relationship graph is: (1) Inputting the facial image into a pre-trained facial 3D key point extractor Mediapipe to obtain 468 facial key points; (2) The facial key points are used as nodes V of the explicit geometric relationship graph L According to the location of the key points in the face, the nodes of the eyebrows, pupils, eye sockets, lips and facial contours are connected in sequence to form the appearance contour. Then, according to the geometric structure of the face, the nodes of the five parts are connected to each other to form the face explicit geometric relationship graph G L .

4. The deep fake detection method based on face geometric relationship reasoning according to claim 1, characterized in that: In step 3, the global feature extractor is a backbone network implemented based on Vision Transformer, including MobileVit.

5. The deep fake detection method based on face geometric relationship reasoning according to claim 1, characterized in that: In step 4, the specific method of constructing a high-information region locator based on the self-supervised learning method is: (1) Obtain M regions of interest using a region proposal network in the global face feature map; (2) Input the local features of the region of interest into a simple binary classifier f p In the classifier, a 1×1 convolution kernel is used to reduce the number of local feature channels to 2 channels, and then an activation function and a batch normalization layer are used to increase the nonlinear expression ability of the features to obtain local semantic information features. Using global average pooling The length and width are downsampled to 1; (3) Using the binary cross entropy loss function L bce Calculate the loss value l between the binary classification result of each local feature and the authenticity of the current face p ; (4) Arrange all loss values ​​obtained from each image from large to small; (5) The loss value l obtained for each local feature P and the corresponding region proposal score S P Calculate the square difference between them, and calculate the mean after obtaining all the square differences, which is used as the loss function L for self-supervised learning. self .

6. The deep fake detection method based on face geometric relationship reasoning according to claim 5, characterized in that: In step 4, construct the implicit geometric relationship graph G of the face P The specific method is: (1) Based on the region proposal score S of the high-information region locator P , select the N local features F with the highest scores in the high-information region P As a set, each F P The feature matrix of the region proposal with a size of 2×W×H is converted into a feature vector of size T×2, where T=W×H, and the converted feature vector set is used as the node of the implicit geometric relationship graph of the face (2) Convert the size of the node set N×T×2 to N×C P , where C P =T×2, using SoftMax operation to convert the node set into the attention vector V att =softmax(V P ); (3) Based on the self-attention mechanism, calculate V P and the attention vector transposed The relationship between the nodes of the implicit geometric relationship graph is obtained by Its size is N×N.

7. The deep fake detection method based on face geometric relationship reasoning according to claim 1, characterized in that: In step 5, the geometric relationship reasoning module consists of an explicit face geometric feature reasoning module, an implicit face geometric feature reasoning module, a graph feature matching module, and a graph classifier: (1) The explicit face geometric feature reasoning module uses a point cloud analysis model built based on a graph convolutional neural network to generate an explicit face geometric relationship graph G L Based on this, we obtain the graph feature expression G of the explicit face geometric relationship gr ; (2) The implicit face geometric feature reasoning module uses a two-layer graph convolutional network model to generate the implicit geometric relationship graph G P Based on the implicit face geometric relationship, the graph feature expression G is obtained. ir ; (3) The graph feature matching module uses a two-layer interactive graph convolutional network model to gr and G ir Based on this, the multi-perspective geometric relationships are integrated to obtain the fused geometric relationship graph G F ; (4) The graph classifier uses global average pooling and global maximum pooling to obtain the maximum value and average value of graph node features respectively, fuses the graph representation features from two observation angles, and performs classification based on a multi-layer perceptron.

8. The deep fake detection method based on face geometric relationship reasoning according to claim 7, characterized in that: The specific method of the graph feature matching module is: (1) Based on the geometric relationship graph G gr and G ir Contains the graph node feature set V gr and V ir , use the mutual attention mechanism to calculate V gr and V ir Transpose The pairwise relationship of G gr Connect G ir The adjacency matrix Its size is N×N; (2) Based on the geometric relationship graph G gr and G ir Contains the graph node feature set V gr and V ir , use the mutual attention mechanism to calculate V ir and V gr Transpose The pairwise relationship of G gr Connect G ir The adjacency matrix Its size is N×N; (3) Based on the graph node feature set V gr and V ir , the adjacency matrix A gi and A ig ,Use an interactive graph convolutional network model to match graph node features and infer the geometric anomalies in deep fake face images.

9. The deep fake detection method based on face geometric relationship reasoning according to claim 7, characterized in that: The calculation process of the interactive graph convolutional network model is as follows: (1) Extraction of G gr Connect G ir Implicit feature relationship graph node feature expression: V gi =σ(W1×σ(A gi V gr W gi )+V gr ); Where W1 and W gi These are all learnable parameters of the interactive graph convolutional network. σ(·) represents the nonlinear activation function, which is ReLu or Leaky ReLu function. (2) Extraction of G ir Connect G gr Implicit feature relationship graph node feature expression: V ig =σ(W2×σ(A ig V ir W ig )+V ir ); Among them W2 and W ig These are all learnable parameters of the interactive graph convolutional network. σ(·) represents the nonlinear activation function, which is ReLu or Leaky ReLu function. (3) Express the two graph node features V gi and V ig Splicing,obtains the node feature expression of multi-angle geometric relationship,graph.

10. The deep fake detection method based on face geometric relationship reasoning according to claim 1, characterized in that: In step 5, the total training loss function is: in, is the binary cross entropy loss function and uses label smoothing technology. is the self-supervised loss function, P P and P a They are the classification results of local high-information regions and the classification results of global geometric relationship graph node features, S P Region proposal score for high-information region localizer.

11. The deep fake detection method based on face geometric relationship reasoning according to claim 1, characterized in that: In step 6, the model trained in steps 1 to 5 is used to make predictions on all video frames, and all scores are averaged as the prediction result of the authenticity of the face in the video.

Images