Face video deepfake detection method and device based on face timing information

Abstract

The present application provides a kind of face video depth forgery detection method and device based on face timing information, comprising: obtaining face video with standard true or false label as training data;Extract features from multiple video frames in training data by basic network, obtain original features, reconstruct three-dimensional face with original features, and obtain reconstructed image;The basic network extracts features from the reconstructed image, and obtains reconstructed features;Difference module carries out difference to original features and reconstructed image, and dimension reduction is carried out to difference result, and difference features are obtained;Reconstructed features and difference features are used as query features, and original features are used as to-be-queried features, and are respectively input into query module, according to the similarity of query feature and to-be-queried feature, to-be-queried feature is reweighted, and weighted difference feature and weighted reconstructed feature are obtained;After adaptive fusion module is weighted and fused, original features, weighted difference features and weighted reconstructed features are classified into true or false.

Description

Technical Field

[0001] This invention relates to the fields of machine learning, computer vision, and image recognition and classification, and particularly to the problem of deepfake detection in computer vision. Background Technology

[0002] With the rapid development of generative adversarial network technology, the Internet is filled with more and more generated content. This content spreads widely on the Internet without being marked as true or false, posing a great challenge to social stability and personal reputation. Therefore, the academic and industrial communities have carried out research on the detection of deepfake face content, which is the most harmful among them, and have achieved certain research results. However, there are still some problems to be solved in this field, such as the generalization problem of detection.

[0003] Improving the generalization ability of deepfake detection has always been a research hotspot. Taking the FaceForensics++ dataset as an example, it contains a video set consisting of four forgery methods (DeepFakes, FaceSwap, Face2Face, and NeuralTexture) and real videos. Normal testing should also include these four forgery methods and real images. However, in actual deepfake system deployment, videos collected from the internet are likely to contain forgery methods other than these four. The ability to effectively detect forgery methods not present in the training set becomes crucial for the successful deployment of deepfake detection systems. For example, in addition to the four forgery methods in the training set, a generalization test method can be added to the test dataset. During testing, the test metrics are calculated separately, and the detection accuracy and AUC can be used as a measure of generalization.

[0004] Current research shows that when the training and test sets are identically distributed (i.e., from the same source), the accuracy can reach over 90%. However, in cases of dissimilar sources, the accuracy is even lower, falling below 60-70%. The difficulty lies in the fact that the fundamental statistical principle of deep neural networks requires that the training and test sets be identically distributed. If the distribution of the test set differs significantly from that of the training set, the neural network struggles to achieve good results on this task. Therefore, directly designing a model for real / fake classification training is unlikely to achieve good generalization. The fundamental reason is that the forgery traces from different forgery methods are inconsistent, especially with newly emerging forgery methods, which have significantly improved in realism compared to previous methods. Therefore, some researchers are searching for forgery traces temporally, such as looking for inconsistencies in the lips during the timeline, or using 3D convolution to find inconsistencies throughout the video. Other researchers are focusing on the final step of face-swapping. While various face-swapping methods exist, they all ultimately need to be integrated back into the source image. Therefore, some researchers have proposed detecting these integration traces for generalization detection. Summary of the Invention

[0005] To address the shortcomings of existing forgery detection methods, such as insufficient facial structure mining and excessive focus on forgery flaws leading to poor generalization, this invention proposes a face forgery detection method based on inconsistent 3D feature reconstruction. This method leverages the restorative properties of 3D face reconstruction images to transform the deep forgery detection problem into an inconsistency problem between the target image and the reconstructed image. It shifts the detection focus from forgery traces to inconsistencies with the reference image, thereby improving the generalization of forgery detection.

[0006] The motivation for proposing this method stems from the observation of 3D face reconstruction results. Through observation, it was found that when an image is re-rendered using 3D features to create a reconstructed image, there are certain differences between the reconstructed image and the image to be detected. The skin of the face becomes smoother, and forgery traces disappear; this difference is as if the forgery traces have been "repaired." Figure 1 As shown, the left side displays the real image and the fake image, while the right side displays the reconstruction of the fake image and the reconstruction of the real image. It can be seen that there is a significant difference between the reconstructed fake image and the fake image itself, while the reconstructed real image is essentially indistinguishable from the real image. The reason for this phenomenon is that the 3D model's representation is based on adding or subtracting elements from a standard (average) human face, which cannot represent the flaws created by the fake face swap. These flaws are not present in a normal human face. Therefore, a neural network can be designed to detect this inconsistency. Furthermore, since this difference exists in multiple fakery methods simultaneously, it does not overfit to deepfake traces compared to existing methods. Therefore, this invention believes that this difference can serve as a breakthrough for improving the generalization of deepfake detection.

[0007] The motivation behind this invention is to identify inconsistencies between two images. For example, in a genuine video, frames tend to have continuity; for instance, the size of the nose in a video of the same person will not change. However, due to issues like poor production quality, fake videos often exhibit variations in facial shape and features. Since the decoder of a transformation model based on an attention mechanism is naturally suited for such query tasks, this invention proposes an attention-based query network using the decoder of the transformation model as its foundation. Finally, to fuse the query results from multiple networks and obtain the most original image features, this invention proposes an adaptive fusion module for fake feature analysis. This final fusion constitutes the face forgery detection method based on 3D feature reconstruction proposed in this invention.

[0008] Specifically, this invention proposes a method for detecting deepfake facial videos based on temporal facial information, including:

[0009] The initial steps involve constructing a deep forgery detection model that includes a feature extractor as the base network, a feature difference module, a query module, and an adaptive fusion module; and acquiring face videos with standard real / fake labels as training data.

[0010] The image reconstruction step involves extracting features from multiple video frames in the training data using the base network to obtain raw features, then using these raw features to perform 3D face reconstruction to obtain a reconstructed image; finally, the base network extracts features from the reconstructed image to obtain reconstructed features.

[0011] The feature difference step involves the feature difference module performing difference on the original feature and the reconstructed image, and then reducing the dimensionality of the difference result to obtain the difference features.

[0012] In the query weighting step, the reconstructed feature and the difference feature are used as query features, and the original feature is used as the feature to be queried. These are input into the query module. Based on the similarity between the query feature and the feature to be queried, the feature to be queried is reweighted to obtain the weighted difference feature and the weighted reconstructed feature.

[0013] In the model training steps, the adaptive fusion module performs a weighted fusion of the original features, the weighted difference features, and the weighted reconstruction features, and then performs a true / false classification. Based on the classification results and the true / false labels, a loss function is constructed to update the training of the deep forgery detection model.

[0014] The deepfake detection step uses the trained deepfake detection model to detect fake people in face videos and obtains the classification result of whether the face video to be detected belongs to fake videos.

[0015] The aforementioned deepfake detection method for face videos based on face temporal information, wherein the image reconstruction step includes:

[0016] The basic network extracts face-related features and face-independent features based on the following formula:

[0017] α i ,β i ,δ i ,p i ,γ i =RNet(v i )

[0018] Here v i RNet represents the extracted i-th video frame, and α represents the 3D face feature extraction model. i ,β i ,δ i The facial features include facial identity features, facial expression features, and facial texture features; p i ,γ iThese are features unrelated to the face, namely lighting features and pose features;

[0019] The face-related features drive the 3DMM model to obtain the position and color of each vertex in the 3D reconstructed face model. Combined with the face-independent features, the 3D reconstructed face model is rendered to obtain a 2D reconstructed face image. This 2D reconstructed face image is then pasted back into the video frame, replacing the face image in the video frame, to obtain the reconstructed image.

[0020] The aforementioned deepfake detection method for face videos based on face temporal information, wherein...

[0021] The weighting steps for this query include:

[0022] The query module includes a Transformer encoder and a Transformer decoder based on an attention mechanism. Consistency markers are added to the reconstructed feature and the difference feature respectively to obtain the query feature. Consistency markers are added to the original feature to obtain the feature to be queried.

[0023] The query feature is input into the Transformer encoder for feature encoding. The Transformer decoder then completes the decoding by reweighting the query feature based on the similarity between the encoded result and the query feature.

[0024] The training steps for this model include:

[0025] The adaptive fusion module takes the consistency identifier from the weighted difference feature and the weighted reconstruction feature, and then performs a weighted fusion with the classification feature before sending it into the linear classification layer for true / false classification.

[0026] The aforementioned deepfake detection method for face videos based on face temporal information includes a linear classification head for extracting image information of the original features. The model training steps include: weighted fusion of the image information of the original features, the weighted differential features, and the weighted reconstructed features, followed by classification for authenticity.

[0027] This invention also proposes a face video deepfake detection device based on face temporal information, comprising:

[0028] The initial module constructs a deep forgery detection model, which includes a feature extractor as the base network, a feature difference module, a query module, and an adaptive fusion module; and obtains face videos with standard real and fake labels as training data.

[0029] The image reconstruction module extracts features from multiple video frames in the training data through the base network to obtain the original features, and then uses the original features to perform 3D face reconstruction to obtain the reconstructed image; the base network then extracts features from the reconstructed image to obtain the reconstructed features.

[0030] The feature difference module performs difference on the original feature and the reconstructed image, and reduces the dimensionality of the difference result to obtain the difference feature;

[0031] The query weighting module takes the reconstructed feature and the difference feature as query features and the original feature as the feature to be queried, and inputs them into the query module respectively. Based on the similarity between the query feature and the feature to be queried, the feature to be queried is reweighted to obtain the weighted difference feature and the weighted reconstructed feature.

[0032] The model training module, the adaptive fusion module, performs weighted fusion of the original features, the weighted difference features and the weighted reconstruction features, performs true / false classification, and constructs a loss function based on the classification results and the true / false labels to update the training of the deep forgery detection model;

[0033] The deep fake detection module uses the trained deep fake detection model to detect fake people in face videos and obtain the classification result of whether the face video to be detected belongs to fake video.

[0034] The aforementioned deepfake detection device for facial videos based on facial temporal information, wherein the image reconstruction module includes:

[0035] The basic network extracts face-related features and face-independent features based on the following formula:

[0036] α i ,β i ,δ i ,p i ,γ i =RNet(v i )

[0037] Here v i RNet represents the extracted i-th video frame, and α represents the 3D face feature extraction model. i ,β i ,δ i The facial features include facial identity features, facial expression features, and facial texture features; p i ,γ i These are features unrelated to the face, namely lighting features and pose features;

[0038] The face-related features drive the 3DMM model to obtain the position and color of each vertex in the 3D reconstructed face model. Combined with the face-independent features, the 3D reconstructed face model is rendered to obtain a 2D reconstructed face image. This 2D reconstructed face image is then pasted back into the video frame, replacing the face image in the video frame, to obtain the reconstructed image.

[0039] The aforementioned deepfake detection device for facial videos based on facial temporal information, wherein...

[0040] This query weighting module includes:

[0041] The query module includes a Transformer encoder and a Transformer decoder based on an attention mechanism. Consistency markers are added to the reconstructed feature and the difference feature respectively to obtain the query feature. Consistency markers are added to the original feature to obtain the feature to be queried.

[0042] The query feature is input into the Transformer encoder for feature encoding. The Transformer decoder then completes the decoding by reweighting the query feature based on the similarity between the encoded result and the query feature.

[0043] The model training module includes:

[0044] The adaptive fusion module takes the consistency identifier from the weighted difference feature and the weighted reconstruction feature, and then performs a weighted fusion with the classification feature before sending it into the linear classification layer for true / false classification.

[0045] The deepfake detection device for face videos based on face temporal information further includes a linear classification head for extracting image information of the original features; the model training module includes: weighted fusion of the image information of the original features, the weighted differential features and the weighted reconstructed features, and then performing true / false classification.

[0046] The present invention also proposes a server, which includes the aforementioned face video deepfake detection device.

[0047] The present invention also proposes a storage medium for storing a computer program that executes the face video deepfake detection device method.

[0048] As can be seen from the above scheme, the advantages of the present invention are: because the present invention proposes a face image forgery detection method based on the difference between the forged portrait image and its three-dimensional reconstructed image, the generalization of forgery detection is improved. Attached Figure Description

[0049] Figure 1This is a schematic diagram illustrating the "repairing" effect of 3D facial reconstruction on forgery flaws.

[0050] Figure 2 This is a schematic diagram of the face reconstruction process of the present invention;

[0051] Figure 3 This is a schematic diagram of the query network based on the attention mechanism proposed in this invention;

[0052] Figure 4 This is a schematic diagram of the feature difference module proposed in this invention;

[0053] Figure 5 This is a schematic diagram of the adaptive fusion module proposed in this invention;

[0054] Figure 6 This is a visual schematic diagram of the reconstruction branch and difference branch proposed in this invention;

[0055] Figure 7 This is a schematic diagram of the server structure of the present invention. Detailed Implementation

[0056] This invention provides a method for detecting deepfake faces based on temporal inconsistencies in 3D information. Addressing the shortcomings of existing image forgery detection methods, such as insufficient facial structure mining and poor generalization, this invention transforms deepfake traces in the detection image into detecting inconsistencies between the target image and the reconstructed 3D face image. It proposes an attention-based query network to detect inconsistencies between the reconstructed image, the difference map, and the target image, and simultaneously proposes an adaptive fusion module to adaptively mix these inconsistencies and forgery features. On common generalization evaluation criteria, the proposed method achieves high generalization results, and ablation experiments with different base networks verify its plug-and-play nature. Finally, tests on various datasets demonstrate the detection generalization of this invention, and visualization results also show that the proposed method effectively captures image inconsistencies.

[0057] In summary, this invention proposes a method for detecting deepfake faces based on inconsistencies in 3D information reconstruction. Given that 3D face reconstruction can repair deepfake traces, this method improves the generalization of deepfake detection by transforming deepfake traces in the detection image into inconsistencies between the image to be detected and the 3D reconstructed face image. To achieve the above technical effects, this invention includes the following key technical points:

[0058] (1) This invention utilizes the difference between the image to be detected and its three-dimensional reconstructed face image to perform deep face forgery detection. It can be used for the detection of various video images forged by faces and can improve the generalization of deep face forgery detection.

[0059] (2) A dual-branch attention mechanism is proposed. By utilizing the query characteristics of the attention mechanism, the reconstruction graph and the difference graph are used to query the image to be detected, obtain inconsistency features, and then mine inconsistency.

[0060] (3) A difference module is proposed. This module obtains difference features from the image to be detected and the reconstructed image, and performs dimensionality reduction to reduce the subsequent computational load.

[0061] (4) An adaptive fusion module is proposed to supplement the original features. At the same time, the original features, differential features and reconstructed features are fused through the adaptive fusion module, and finally detection and classification are performed.

[0062] (5) The detection method proposed in this invention has high versatility and can be applied to various basic networks. A dual-branch attention mechanism is designed for detection.

[0063] Experimental results demonstrate that the proposed face depth forgery detection method based on inconsistent 3D information reconstruction has high generalization ability, and visualization results also verify the interpretability of the method.

[0064] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0065] The following content provides a detailed introduction to the face forgery detection method based on inconsistent 3D feature reconstruction proposed in this invention. First, the face reconstruction module describes how to obtain a reconstructed image from a target image by extracting 3D facial features and using differentiable rendering. Next, an attention-based query network, which is the main structure of the proposed query mechanism, consists of three branches. Finally, an adaptive fusion module fuses the forgery features from the three branches to obtain the final classification result.

[0066] The steps to reconstruct a 3D image of a face are as follows: Figure 2 As shown, the video is represented as V. The image to be detected first passes through a feature extractor to extract face-related features and face-independent features. This feature extractor comes from the face 3D reconstruction method proposed by Deng et al.

[0067] α i ,β i ,δ i ,p i ,γ i =RNet(v i )

[0068] Here vi Represents the extracted i-th frame image, and RNet represents a face 3D feature extraction model based on ResNet50 architecture. Then, face-related features drive the 3DMM model to obtain the position S and color T of each vertex in the 3D reconstructed face model.

[0069] S i ,T i =3DMM(α) i ,β i ,δ i )

[0070] S i ={x1,y1,z1,x2,y2,z2,x3,y3,z3…}

[0071] T i ={r1,g1,b1,r2,g2,b2,r3,g3,b3…}

[0072] Next, using graphics rendering methods that combine lighting and pose information, the aforementioned 3D model can be rendered. This invention chooses a differentiable rendering method, consistent with the training process. Finally, based on the facial key points detected in the first step, the rendered image is overlaid on the original image to form a reconstructed image containing the background.

[0073] The structure of the attention-based query network is as follows: To detect inconsistencies between two images, this invention proposes an attention-based query network, the structure of which is as follows: Figure 3 As shown, the attention-based query network structure mainly consists of two feature extraction branches and three query feature branches formed by their differences. One branch reconstructs the feature to be detected, another branch uses the differenced feature to query the feature to be detected, and the third branch is the feature to be detected itself, which aims to provide the original forged features. The process is described in detail below: First, feature maps are extracted from the image to be detected and the reconstructed image. The weights of the convolutional neural network are shared and participate in the training process. The reason for sharing weights is that features extracted from different images using the same parameters can highlight their inconsistencies. Furthermore, the structure of this basic network can be replaced, as verified in ablation experiments. This process is expressed by the following formula:

[0074] f i =F(v) i ),

[0075] Where F represents the base network used for feature extraction, v i This represents the image to be tested. This represents the reconstructed image. At this point, fi and Where w, h, and c are the sizes of the extracted feature maps, the two feature maps are first processed through a feature difference module to obtain difference features, then dimensionality reduction is performed, and finally expansion is performed. The structure of the feature difference module is as follows: Figure 4 As shown, the formulaic description is as follows:

[0076]

[0077] f i =F p (f i )

[0078]

[0079]

[0080] Among them, F p , and It consists of a single convolutional layer with a kernel size of 1×1, reducing its channel value from c to e. Next, the features need to be expanded, with the goal of expanding them along w and h, into a sequence of length l = w×h and feature dimension e.

[0081]

[0082] and And the f above i Perform the same operation to expand.

[0083] The main function of differencing is to calculate the difference in high-dimensional features between two frames. Similar to the first research point, directly calculating the difference helps the network focus on inconsistencies rather than the facial information contained in the image itself. Secondly, the main function of dimensionality reduction is to reduce the computational cost of the subsequent query mechanism. Finally, the unfolding operation transforms image features into sequence features.

[0084] The following section focuses on the structure of the two query branches. The query structure utilizes the query mechanism within the attention mechanism, where one feature serves as Q, and the other as both K and V. The principle here is to use feature Q as the query, calculate its similarity with K, and thus reweight feature V. The attention mechanism can be formally expressed as:

[0085]

[0086] that place Where l represents the sequence length, e, e represents the feature dimension of each position in the sequence, and d kIn natural language processing, the dimension representing word vectors is 'e', ​​which is used in this invention. In its implementation, this invention uses a Transformer encoder and a Transformer decoder composed of an attention mechanism. The encoder mainly consists of the aforementioned attention mechanism, a layer normalization structure, and a forward propagation network. The three inputs Q, K, and V in the encoder are consistent, thus employing a self-attention mechanism. In the decoder, Q differs from K and V, and the input features are weighted according to the query features.

[0087] Simultaneously, mimicking the class token in the ViT model, used to acquire class features of different image patches during training, this invention designs an inconsistency token. Similar to the class representation, it is added to the beginning of the sequence, thus increasing the sequence length by 1, becoming l+1. This token is learnable and participates in training, and its formulaic description is as follows:

[0088] f i =concat(T,f i )

[0089]

[0090]

[0091] Where T,T reco ,T diff The inconsistency identifier (consistency identifier), as a learnable feature vector, is concatenated before the features. `concat` represents the concatenation operation. After performing the above operation,

[0092] This invention uses reconstructed features and difference features as query features, first inputting them into a Transformer encoder for feature encoding, and then using them as Q to enter a Transformer decoder. The features to be detected are directly used as K and V for decoding.

[0093]

[0094]

[0095] After decoding, the inconsistency level indicator of each part is used for subsequent feature fusion:

[0096]

[0097]

[0098] at this time, 0 represents extracting the first vector from the first feature, which is to be placed in the first inconsistency identifier; through the above feature fusion, the inconsistencies will be gathered into the identifier feature.

[0099] This invention uses the features and difference features of the reconstructed image as query features, and the features of the image to be detected as query features. This design allows for the recombination of features in the image to be detected, thereby highlighting features related to forgery traces. The features extracted by the base network encompass the original image information. Although directly inputting the features extracted by the base network into the adaptive fusion module results in features that primarily represent inconsistencies and lose some original forgery trace information, this invention adds a branch with a linear classification layer (linear classification head) to supplement the classification model with the original image information. For example... Figure 6 As shown, this invention proposes an adaptive fusion module to fuse the fake features of the three branches. This adaptive module performs adaptive weighting for each branch. First, the weights of each branch are obtained through a linear layer and a sigmoid function. Then, the weighted sums of the different branches are fed into a linear classification layer for real / fake classification. The linear layer here uses common parameters, and only the weight matrix is ​​retained, with the bias removed, denoted as W:

[0100]

[0101]

[0102] c i =Sigmoid(W·f i )

[0103]

[0104] p i To identify the final predicted value, this invention uses cross-entropy as the loss function L:

[0105] L = -[ylogp + (1-y)log(1-p)]

[0106] We conducted our experiments on the PyTorch platform. All models were trained using eight GTX 3090 GPUs. Regarding training details, our method used Adam as the optimizer, with a learning rate set to 1e-4, decaying every 10 epochs with a decay factor of 0.1 each time. Training lasted for a total of 20 epochs. In the above implementation, optionally, only 16 consecutive frames from the video are detected each time. The result of a video is achieved by segmenting the video into smaller slices of 16 frames, judging each slice as real or fake, and finally averaging the results to determine the video's real / fake score.

[0107] The training data used in this invention consists of multiple videos of people already labeled with real and fake tags, which are used to train the feature extractor (base network), feature difference module, two pairs of Transformer encoders and decoders, linear classification head, and adaptive fusion module. However, the 3DM model is a model trained using existing technologies and does not require additional training in this invention.

[0108] Experimental results:

[0109] Since the main research motivation of this invention is to improve the generalization of forgery detection, the experimental part of this invention is mainly a generalization detection setting. Specifically, this invention follows the main generalization detection experimental settings in this field, trains on the FaceForensics++[6] dataset, and tests on the test sets of Celeb-DFV2 and FaceShifter

[11] . For the experimental dataset, the official training and validation split is selected for the experiment. For each video to be detected, this method randomly selects 20%, or 54 frames, from its first 270 frames for the experiment. This invention sets the same settings for the training, validation and test sets. This invention takes the average of the results of all segments of a video as the prediction result of the video. This invention first uses RetinaFace

[21] for face detection. In order to retain more information around the face, this invention cropped the area twice the size of the bounding box of each frame, and also saved the five facial key points obtained in the face detection process. Five facial landmarks are used to align these faces with a standard face template, which helps to convert 2D face images into 3D face features. For the extraction of 3D prior information about the face, this invention uses a third-party implementation based on the PyTorch framework. https: / / github.com / sicxu / Deep3DFaceRecon_ pytorch Both the detected face image and the reconstructed image were resized to 224×224. As with previous methods in this field, Accuracy and AUC were used as the primary evaluation metrics.

[0110] We first conducted experiments on multiple datasets and compared our method with existing methods. The experimental results are shown in Table 1.

[0111]

[0112] Table 1 compares the detection results of the method proposed in this invention with those of other methods. Xception* represents the use of more test data.

[0113] In Table 1, we compared our method with some existing methods. As you can see, our method achieved a significant generalization gain on various basic networks. After adding more test data, the method proposed in this paper achieved the best generalization effect.

[0114]

[0115] Table 2. Restorative Effects of 3D Face Reconstruction

[0116] To verify that the 3D face reconstruction image does indeed "repair" forgery traces, this invention designed an ablation experiment to address this issue. The experiment was set up as follows: using ResNet50 as the base network, binary classification experiments were conducted on the FaceForensics++ dataset. The first experiment used the image to be detected, and the second experiment used the 3D face reconstruction image. The experimental results are shown in Table 2, where "Original Images" represents classification using the original image and "Reconstruction Images" represents classification using the reconstructed image. The experimental results show that when classifying directly, the forgery detection performance using the 3D face reconstruction image decreased by 9.3% compared to using the original image. This experimental result demonstrates the "repairing" effect of the reconstruction image on deepfake traces.

[0117]

[0118] Table 3. Effectiveness of generalization of three branches on ResNet50

[0119]

[0120] Table 4. Effectiveness of generalization of the three branches on Xception.

[0121] This invention conducted ablation experiments on different branches of the ResNet50 and Xception basic network structures. To verify the detection results of the three branches proposed in this invention and their fusion effect, the generalization of the proposed spoofing algorithm was tested separately on each of the three branches, and the fusion effect of the three branches was also tested. As shown in Tables 3 and 4, the first column of the table represents the spoofing detection method, where "Baseline (ResNet50)" represents direct classification using ResNet50, and also represents the third branch in the method proposed in this invention; "Reco-branch" represents the branch that uses reconstructed features to query the features to be detected; "Diff-branch" represents the branch that uses difference features to query the features to be detected; and the last "Ours" represents the result of combining the above three branches. Each subsequent column in the table represents the detection results on different generalization datasets. The results on FaceForensics++ represent the results of training and testing on the same source, while the results on the other three datasets are generalization detection results. The purpose of adding FaceForensics++ here is to consider the detection performance on the original dataset while ensuring generalization.

[0122] Experimental results show that, on different base networks, the proposed Reco and Diff branches significantly improve generalization compared to the basic direct binary classification. Fusing the three branches achieves even better generalization. Notably, on the FaceForensics++ dataset, the proposed algorithm shows a slight decrease in performance compared to direct binary classification. Due to the inclusion of more features, the model avoids overfitting the training dataset, resulting in a 1% to 2% decrease in detection performance on that dataset. However, this decrease is considered acceptable compared to the improvement in model generalization.

[0123]

[0124] Table 5 Ablation experiments of the adaptive fusion module

[0125] To verify the effectiveness of the adaptive fusion branch proposed in this invention, experiments were conducted using the direct addition of the three branches as the basis for comparison. The experimental results are shown in Table 5. The table structure is consistent with the other tables mentioned above. In the first column, "Add" represents direct addition, and "Adaptive-add" represents the adaptive fusion module proposed in this invention. The experimental results show that, regardless of whether it's on the same FaceForensics++ dataset or other datasets, the adaptive fusion module proposed in this invention improves the detection results to a certain extent in generalization experiments, proving the effectiveness of the proposed module.

[0126] This invention provides a visual analysis of the inconsistency identifiers in the reconstructed branches, and the results are as follows: Figure 6 As shown on the left, the first column represents the image to be detected, the second column represents the reconstructed image, and the third column represents the visualization analysis image. The visualization image represents the similarity between the first two images; therefore, the dark area of ​​the face in the third image represents inconsistency. In the third example, the forehead of the face has a white headscarf blemish in the image to be detected, but this blemish is repaired in the reconstructed image, resulting in a dark area in the visualization image representing poor similarity at that location; other samples are similar. This invention performs visualization analysis on the inconsistency markers of the difference branches, and the results are as follows. Figure 6As shown on the right, the first column represents the image to be detected, the second column represents the reconstructed image, and the third column represents the visualization analysis chart. The visualization chart represents the similarity corresponding to the difference between the first two images; that is, the greater the difference between the first two images, the brighter the corresponding area will be in the third image. Taking the first row as an example, the first image has obvious glue marks around the mouth of the face. This problem was fixed in the second reconstructed image, and the inconsistent area is highlighted in the third image. In the last row, the image to be detected has square forgery marks around the face. This defect was fixed in the second reconstructed image, and a square area is visualized in the third image to mark the defect.

[0127] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0128] This invention also proposes a face video deepfake detection device based on face temporal information, comprising:

[0129] The initial module constructs a deep forgery detection model, which includes a feature extractor as the base network, a feature difference module, a query module, and an adaptive fusion module; and obtains face videos with standard real and fake labels as training data.

[0130] The image reconstruction module extracts features from multiple video frames in the training data through the base network to obtain the original features, and then uses the original features to perform 3D face reconstruction to obtain the reconstructed image; the base network then extracts features from the reconstructed image to obtain the reconstructed features.

[0131] The feature difference module performs difference on the original feature and the reconstructed image, and reduces the dimensionality of the difference result to obtain the difference feature;

[0132] The query weighting module takes the reconstructed feature and the difference feature as query features and the original feature as the feature to be queried, and inputs them into the query module respectively. Based on the similarity between the query feature and the feature to be queried, the feature to be queried is reweighted to obtain the weighted difference feature and the weighted reconstructed feature.

[0133] The model training module, the adaptive fusion module, performs weighted fusion of the original features, the weighted difference features and the weighted reconstruction features, performs true / false classification, and constructs a loss function based on the classification results and the true / false labels to update the training of the deep forgery detection model;

[0134] The deep fake detection module uses the trained deep fake detection model to detect fake people in face videos and obtain the classification result of whether the face video to be detected belongs to fake video.

[0135] The aforementioned deepfake detection device for facial videos based on facial temporal information, wherein the image reconstruction module includes:

[0136] The basic network extracts face-related features and face-independent features based on the following formula:

[0137] α i ,β i ,δ i ,p i ,γ i =RNet(v i )

[0138] Here v i RNet represents the extracted i-th video frame, and α represents the 3D face feature extraction model. i ,β i ,δ i The facial features include facial identity features, facial expression features, and facial texture features; p i ,γ i These are features unrelated to the face, namely lighting features and pose features;

[0139] The face-related features drive the 3DMM model to obtain the position and color of each vertex in the 3D reconstructed face model. Combined with the face-independent features, the 3D reconstructed face model is rendered to obtain a 2D reconstructed face image. This 2D reconstructed face image is then pasted back into the video frame, replacing the face image in the video frame, to obtain the reconstructed image.

[0140] The aforementioned deepfake detection device for facial videos based on facial temporal information, wherein...

[0141] This query weighting module includes:

[0142] The query module includes a Transformer encoder and a Transformer decoder based on an attention mechanism. Consistency markers are added to the reconstructed feature and the difference feature respectively to obtain the query feature. Consistency markers are added to the original feature to obtain the feature to be queried.

[0143] The query feature is input into the Transformer encoder for feature encoding. The Transformer decoder then completes the decoding by reweighting the query feature based on the similarity between the encoded result and the query feature.

[0144] The model training module includes:

[0145] The adaptive fusion module takes the consistency identifier from the weighted difference feature and the weighted reconstruction feature, and then performs a weighted fusion with the classification feature before sending it into the linear classification layer for true / false classification.

[0146] The deepfake detection device for face videos based on face temporal information further includes a linear classification head for extracting image information of the original features; the model training module includes: weighted fusion of the image information of the original features, the weighted differential features and the weighted reconstructed features, and then performing true / false classification.

[0147] like Figure 7 As shown, the present invention also proposes a server, which includes the aforementioned face video deepfake detection device.

[0148] The present invention also proposes a storage medium for storing a computer program that executes the face video deepfake detection device method.

[0149] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for face video deepfake detection based on face timing information, characterized in that, Comprising: An initial step of constructing a deep fake detection model comprising a feature extractor as a base network, a feature difference module, a query module, and an adaptive fusion module; and obtaining face videos with standard true or fake labels as training data; An image reconstruction step of extracting features of multiple video frames in the training data by the base network to obtain original features, and performing three-dimensional face reconstruction on the original features to obtain reconstructed images; and extracting features of the reconstructed images by the base network to obtain reconstructed features; The feature difference step is that the feature difference module differentiates the original feature and the reconstructed feature to obtain a difference feature. A query weighting step of inputting the reconstructed features and the difference features as query features, and inputting the original features as to-be-queried features into the query module, respectively, re-weighting the to-be-queried features according to similarities between the query features and the to-be-queried features to obtain weighted difference features and weighted reconstructed features; A model training step of performing true or fake classification on the original features, the weighted difference features, and the weighted reconstructed features after weighted fusion by the adaptive fusion module, and constructing a loss function according to a classification result and the true or fake label to update the deep fake detection model; A deep fake detection step of performing face video fake detection on a person by using the trained deep fake detection model to obtain a classification result of whether a to-be-detected face video is a fake video; The image reconstruction step comprises: The base network extracts face-related features and face-unrelated features based on the following formula: Here Ri represents the i-th video frame extracted, RNet represents a three-dimensional feature extraction model of a human face, The features related to the human face include a human face identity feature, a human face expression feature, and a human face texture feature. The features irrelevant to the human face include a light feature and a posture feature. The face-related feature drives a 3DMM model to obtain the positions and colors of each vertex in a three-dimensional reconstructed face model, the face-unrelated feature is combined to render the three-dimensional reconstructed face model, a two-dimensional reconstructed face image is obtained, and the two-dimensional reconstructed face image is pasted back to the video frame to replace the face image in the video frame, so that the reconstructed image is obtained ; The feature difference step comprises: wherein, denotes a feature extraction network, denotes a picture to be detected, denotes a reconstructed image; and wherein w, h, c are the size of the feature map, and and firstly, the feature difference module is used to obtain the difference feature, then dimension reduction is performed, and finally, expansion is performed: wherein, , and is composed of a convolutional layer with a kernel size of , which reduces its channel value from c to e; next, the feature is unfolded in w and h to become a sequence feature with a length of , and the feature dimension e as the differential feature: 。 2. The face video deep fake detection method based on face temporal information according to claim 1, wherein The query weighting step comprises: The query module comprises a Transformer encoder and a Transformer decoder based on an attention mechanism, consistency identifiers are added to the reconstructed features and the difference features to obtain the query features, and consistency identifiers are added to the original features to obtain the to-be-queried features; The query features are input into the Transformer encoder for feature encoding, and the Transformer decoder decodes by re-weighting the to-be-queried features according to similarities between the encoding result and the to-be-queried features; The model training step comprises: The adaptive fusion module takes the consistency identifiers in the weighted difference features and the weighted reconstructed features, and performs weighted fusion on the consistency identifiers and the classification features to send them to a linear classification layer for true or fake classification.

3. The face video deepfake detection method based on face timing information according to claim 1, wherein the deepfake detection model further comprises a linear classification head for extracting image information of the original features; and the model training step comprises: The image information of the original features, the weighted difference features, and the weighted reconstructed features are weighted and fused for true or fake classification.

4. An apparatus for face video deepfake detection based on face timing information, characterized in that, Comprising: An initial module of constructing a deep fake detection model comprising a feature extractor as a base network, a feature difference module, a query module, and an adaptive fusion module; and obtaining face videos with standard true or fake labels as training data; An image reconstruction module of extracting features of multiple video frames in the training data by the base network to obtain original features, and performing three-dimensional face reconstruction on the original features to obtain reconstructed images; and extracting features of the reconstructed images by the base network to obtain reconstructed features; A query weighting module of inputting the reconstructed features and the difference features as query features, and inputting the original features as to-be-queried features into the query module, respectively, re-weighting the to-be-queried features according to similarities between the query features and the to-be-queried features to obtain weighted difference features and weighted reconstructed features; The feature difference module differentiates the original feature and the reconstructed feature and reduces the dimension of the differentiation result to obtain a differentiated feature. The query weighting module inputs the reconstructed feature and the differential feature as query features and the original feature as a to-be-queried feature into the query module, respectively, re-weights the to-be-queried feature according to the similarity between the query feature and the to-be-queried feature, and obtains a weighted differential feature and a weighted reconstructed feature; The model training module performs true or false classification on the original feature, the weighted differential feature and the weighted reconstructed feature after weighted fusion by the adaptive fusion module, and constructs a loss function according to the classification result and the true or false label to update the deep fake detection model. The deep fake detection module uses the trained deep fake detection model to perform face video fake detection, and obtains a classification result of whether the to-be-detected face video is a fake video. The image reconstruction module includes: The base network extracts face-related features and face-unrelated features based on the following formula: Here Ri represents the i-th video frame extracted, RNet represents a three-dimensional feature extraction model of a human face, The features related to the human face are identity features, expression features, and texture features of the human face. The features unrelated to the human face are illumination features and posture features. The face-related feature drives a 3DMM model to obtain the positions and colors of each vertex in a three-dimensional reconstructed face model, the face-unrelated feature is combined to render the three-dimensional reconstructed face model, a two-dimensional reconstructed face image is obtained, and the two-dimensional reconstructed face image is pasted back to the video frame to replace the face image in the video frame, so that the reconstructed image is obtained ; The feature difference module includes: wherein, denotes a feature extraction network, denotes a picture to be detected, denotes a reconstructed image; and where w, h, c are the size of the feature map, and and firstly, the feature difference module is used to obtain the difference feature, then dimension reduction is performed, and finally, expansion is performed: wherein, , and is composed of a convolution layer with a kernel size of , which reduces the channel value from c to e; next, the feature is unfolded in w and h, becoming a sequence feature with a length of , and a feature dimension of e as the differential feature: 。 5. The face video deep fake detection device based on face timing information according to claim 4, wherein The query weighting module includes: The query module includes a Transformer encoder and a Transformer decoder based on an attention mechanism, adds a consistency identifier to the reconstructed feature and the differential feature respectively to obtain the query feature, and adds a consistency identifier to the original feature to obtain the to-be-queried feature. The query feature is input into the Transformer encoder for feature encoding, and the Transformer decoder decodes by re-weighting the to-be-queried feature according to the similarity between the encoding result and the to-be-queried feature. The model training module includes: The adaptive fusion module takes the consistency identifier in the weighted differential feature and the weighted reconstructed feature, and performs true or false classification after weighted fusion with the classification feature and sending to a linear classification layer.

6. The face video deepfake detection device based on face timing information according to claim 4, wherein the deepfake detection model further comprises a linear classification head for extracting image information of the original features; and the model training module comprises: The image information of the original feature, the weighted differential feature and the weighted reconstructed feature are weighted and fused to perform true or false classification.

7. A server, characterized by The face video deep fake detection device according to any one of claims 4-5.

8. A storage medium for storing a computer program for executing the face video deep fake detection device method according to any one of claims 1-3.

Images