Deepfake video detection method based on multi-domain feature region standard score difference

Abstract

The application discloses a Deepfake video detection method based on multi-domain feature region standard score difference, which comprises the following steps: data set division; video frame division and extraction of a region to be detected; construction of a double-branch convolutional neural network; calculation of the RGB feature and NSCT sub-band image of the region to be detected; frequency domain feature obtained by frequency band fusion of the NSCT sub-band image; response of different level texture features obtained through a texture feature extraction module; output features of the space domain and frequency domain feature branches are spliced along the channel dimension, and input into an abnormal feature discrimination module to obtain a tampered region prediction mask; the tampered region prediction mask is subjected to a full connection layer to obtain one-dimensional features, which are spliced with the output features of the texture feature extraction module, and then subjected to a full connection layer and a Softmax activation function to output a binary classification prediction result. The application can better combine the feature information of the space domain and the frequency domain, strengthen the response to the texture feature, discriminate abnormal tampering traces, and improve the generalization ability of the model.

Description

Technical Field

[0001] This invention relates to the field of Deepfake video tampering detection technology, specifically to a Deepfake video detection method based on the difference in standard scores of multi-domain feature regions. Background Technology

[0002] With the development of artificial intelligence, video manipulation and forgery techniques are also evolving rapidly. Deepfake video manipulation technology, which has emerged in recent years, uses deep learning networks to replace faces in the original video with target faces while preserving the original scene and facial expressions. This technology has a low barrier to entry and generates realistic videos; however, if maliciously used and disseminated, it can negatively impact individual portrait rights and public opinion. Therefore, research on deepfake video manipulation detection technology is of great significance.

[0003] Existing Deepfake video tampering detection technologies mainly fall into two categories: those based on manual feature extraction and those based on neural network feature extraction. Manual feature extraction methods extract features such as frequency domain maps and optical flow maps from video frame images and input them into traditional pattern classifiers like SVM (Support Vector Machine) for classification. However, this approach suffers from insufficient feature extraction, leading to poor detection results. Neural network feature extraction methods input video frame images into neural networks for feature mining. This approach tends to overemphasize data content while overlooking tampering traces. While it achieves good detection results within a single database, its performance drops significantly in cross-database detection.

[0004] While the above methods improve the in-database detection performance of face-swapping video tampering detection to some extent, they suffer from insufficient cross-database detection performance, which reduces the practicality and application value of the methods. Summary of the Invention

[0005] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a Deepfake video detection method based on the difference in standard scores of multi-domain feature regions. This invention combines the spatial and frequency domain features of video data, utilizes a texture feature extraction module to enhance the response to spatial branch texture features, and calculates the standard scores of features in different regions through a special pooling layer in the anomaly feature discrimination module to determine the degree of feature anomaly. It employs binary classification labels and face-swapping region masks to supervise model training, effectively reducing the average error rate of the model in cross-database testing while ensuring good detection performance within the database, and improving the model's generalization ability.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for detecting deepfake videos based on the difference in standard scores of multi-domain feature regions includes the following steps:

[0008] Divide the data into different datasets, frame the video in each dataset, and extract the face region of each frame as the region to be detected.

[0009] Calculate the RGB features of the region to be detected in each frame of the image;

[0010] Calculate the NSCT sub-band image of the region to be detected in each frame of the image, fuse and stitch the NSCT sub-band images according to frequency bands to obtain frequency domain features;

[0011] A dual-branch convolutional neural network is constructed, including a spatial domain feature extraction branch and a frequency domain feature extraction branch. RGB features and frequency domain features are input into the spatial domain feature extraction branch and the frequency domain feature extraction branch, respectively.

[0012] Texture feature extraction modules are set up at different depths in the spatial feature extraction branch to extract local detail features in the shallow network and overall abstract features in the deep network, and output texture features respectively.

[0013] The output features of the spatial domain feature branch and the frequency domain feature branch are concatenated along the channel dimension to obtain the concatenated features;

[0014] The spliced ​​features are input into the anomaly feature discrimination module for anomaly feature discrimination, and the output is the tampered region prediction mask;

[0015] The tampering region prediction mask is input into the fully connected layer, and the output features and texture features are concatenated along the channel dimension. The binary classification prediction result is output through the fully connected layer. The training of the binary classification result and the tampering region prediction mask is supervised by the binary classification label and the face swapping region mask, respectively. The loss function is calculated and the network weight coefficients are updated by backpropagation. The network model and the optimal weights are saved.

[0016] Feature extraction and prediction classification are performed based on the trained dual-branch convolutional neural network, and the results of test set sample tampering detection are output.

[0017] As a preferred technical solution, the specific steps for calculating the NSCT sub-band image of the region to be detected in each frame of the image include:

[0018] Non-subsampled contourlet transform is performed on the detection region of each frame image to obtain NSCT sub-band images. Non-subsampled pyramid decomposition and non-subsampled directional filter bank are used to decompose and extract features of the image in different frequency bands according to different directions. After decomposition, low-frequency sub-band, horizontal mid-frequency sub-band, vertical mid-frequency sub-band, horizontal high-frequency sub-band and vertical high-frequency sub-band are obtained.

[0019] As a preferred technical solution, the step of fusing and stitching together the sub-band images of NSCT according to frequency bands to obtain frequency domain features includes the following specific steps:

[0020] Two directional subbands based on intermediate frequency subbands are weighted and fused according to a preset ratio to obtain an intermediate frequency fused subband;

[0021] Two directional subbands based on high-frequency subbands are weighted and fused according to a preset ratio to obtain a high-frequency fused subband.

[0022] The frequency domain features are obtained by splicing the low-frequency, mid-frequency, and high-frequency components along the channel dimension.

[0023] As a preferred technical solution, both the spatial domain feature extraction branch and the frequency domain feature extraction branch use EfficientNetB4 as the base network. EfficientNetB4 includes convolutional layers, MBConv layers, average pooling layers, and fully connected layers.

[0024] As a preferred technical solution, the texture feature extraction module includes three convolutional layers, a Gram matrix calculation layer, and a global average pooling layer. Batch normalization layers and ReLU activation functions are added after the second and third convolutional layers. The texture feature extraction module processes the input features of the spatial feature extraction branch and the output features of the first five MBConv layers of the EfficientNet-B4 backbone network, respectively. Then, the one-dimensional vector features output by each module are concatenated to obtain a one-dimensional vector output feature.

[0025] As a preferred technical solution, the Gram matrix calculation formula in the Gram matrix calculation layer is as follows:

[0026]

[0027] Among them, G ij This represents the value at coordinate (i,j) in the Gram matrix, c represents the number of channels in the input feature of this layer, and F represents the stretched feature of the input feature along each channel. ik Let F represent the value at coordinate (i,k). T This represents the transpose of F.

[0028] As a preferred technical solution, the step of inputting the spliced ​​features into the anomaly feature discrimination module for anomaly feature discrimination and outputting a tampered region prediction mask includes the following specific steps:

[0029] The anomaly feature discrimination module includes a 1×1 convolutional layer, a batch normalization layer, four special pooling layers of different scales, a ConvLSTM layer, and a 7×7 convolutional layer.

[0030] Each special pooling layer is used to calculate the standard score of the feature in a region of corresponding size. A one-dimensional pseudo-temporal dimension is added to the output feature of each special pooling layer, and the features are concatenated along this dimension. The concatenated features are then input into a ConvLSTM layer for spatiotemporal analysis, and the output is the tampered region prediction mask.

[0031] As a preferred technical solution, the formula for calculating the standard score is expressed as follows:

[0032]

[0033] Among them, Z l ij f represents the value at coordinates (i,j) in the output feature map of the l-th layer of the special pooling layer. l ij This represents the value at coordinates (i,j) in the input feature map of layer l. f l ij The characteristic mean of the region, s represents f l ij The characteristic standard deviation of the region.

[0034] This invention also provides a Deepfake video detection system based on the difference in standard scores of multi-domain feature regions, comprising: a dataset partitioning module, a video data preprocessing module, a feature image calculation module, a neural network construction module, a texture feature extraction module, a feature splicing module, an anomaly feature discrimination module, a network training module, and a detection module;

[0035] The dataset partitioning module is used to partition the datasets.

[0036] The video data preprocessing module is used to divide the video of each dataset into frames and extract the face region of each frame image as the region to be detected.

[0037] The feature image calculation module is used to calculate the RGB features of the region to be detected in each frame image, calculate the NSCT sub-band image of the region to be detected in each frame image, and fuse and stitch the NSCT sub-band images according to frequency bands to obtain frequency domain features.

[0038] The neural network construction module is used to construct a dual-branch convolutional neural network, including a spatial domain feature extraction branch and a frequency domain feature extraction branch, and inputs RGB features and frequency domain features into the spatial domain feature extraction branch and the frequency domain feature extraction branch respectively;

[0039] The texture feature extraction module is set at different depths of the spatial feature extraction branch to extract local detail features in the shallow network and overall abstract features in the deep network, and output texture features.

[0040] The feature splicing module is used to splice the output features of the spatial domain feature branch and the frequency domain feature branch along the channel dimension to obtain spliced ​​features;

[0041] The abnormal feature discrimination module is used to discriminate abnormal features in the spliced ​​features and output a tampered region prediction mask.

[0042] The network training module is used to input the tampering region prediction mask into the fully connected layer, concatenate the output features and texture features along the channel dimension, output the binary classification prediction result through the fully connected layer, supervise the training of the binary classification result and the tampering region prediction mask by using binary classification labels and face swapping region masks respectively, calculate the loss function and backpropagate to update the network weight coefficients, and save the network model and the optimal weights.

[0043] The detection module is used to perform feature extraction and prediction classification based on a trained dual-branch convolutional neural network, and output the test set sample tampering detection results.

[0044] As a preferred technical solution, the anomaly feature discrimination module includes a 1×1 convolutional layer, a batch normalization layer, four special pooling layers of different scales, a ConvLSTM layer, and a 7×7 convolutional layer.

[0045] Each special pooling layer is used to calculate the standard score of the feature in a region of corresponding size. A one-dimensional pseudo-temporal dimension is added to the output feature of each special pooling layer, and the features are concatenated along this dimension. The concatenated features are then input into a ConvLSTM layer for spatiotemporal analysis, and the output is the tampered region prediction mask.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] (1) This invention inputs the RGB features and NSCT (non-subsampled contour wave transform) subband fused image of the data to be tested into a dual-branch convolutional neural network to extract high-level semantic features. It uses a texture feature extraction module to enhance the response to spatial branch texture features. It achieves abnormal feature detection by calculating the standard scores of features in different regions. It adopts training of binary classification labels and face-swapping region mask supervision network, which provides an effective way for face-swapping video tampering detection. While maintaining good detection effect in the database, it effectively reduces the average error rate of the model in cross-database testing and improves the generalization ability of the model.

[0048] (2) The present invention uses a two-level bidirectional NSCT transform to extract low-frequency, mid-frequency and high-frequency sub-band images of samples, which effectively highlights the mid-frequency and high-frequency feature differences between real and fake samples, and provides strong assistance for subsequent feature extraction and abnormal feature discrimination of the network.

[0049] (3) This invention adopts a frequency band fusion method for each sub-band image of NSCT. On the one hand, this avoids the problem of training being difficult to fit due to the excessively sparse features of each sub-band image. On the other hand, the frequency band fusion method does not reduce the feature differences between real and fake samples, thus enhancing the robustness of the network to feature extraction.

[0050] (4) In the texture feature extraction module, the present invention captures the correlation between features by calculating the Gram matrix between features, which enhances the network’s response to texture features, strengthens the learning of spatial texture features of the entire branch, and thus effectively captures the texture feature differences between real and fake samples.

[0051] (5) In the abnormal feature discrimination module, this invention calculates the standard scores of features in different regions through special pooling layers, thereby realizing anomaly detection. On the one hand, when tampering traces appear, the standard scores of abnormal features will increase, which can be used to effectively detect tampering traces; on the other hand, large-sized special pooling layers have a larger feature receptive field, making it easier to discover most of the more obvious abnormal features, while small-sized special pooling layers are responsible for mining a small number of abnormal features that are not obvious in the global range but are more prominent in the local area. Combining large-sized and small-sized special pooling layers can improve the robustness of detection.

[0052] (6) This invention uses binary classification labels and face-swapping region masks to train and supervise the network, realizing multi-task learning. By splicing texture features and tampered region features, information exchange between multiple tasks can be realized, which can optimize the learning performance of each task, thereby alleviating the problem of overfitting in traditional single-task networks and achieving the technical effect of improving the generalization ability of the model. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the training process of the Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to the present invention.

[0054] Figure 2 This is a test flowchart of the Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to the present invention.

[0055] Figure 3 This is a schematic diagram of the overall architecture of the Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to the present invention;

[0056] Figure 4 This is a schematic diagram of the video frame image and RGB feature image of the present invention;

[0057] Figure 5 This is a schematic diagram of the non-subsampled contour wave transform of the present invention;

[0058] Figure 6 This is a schematic diagram of each subband image of the actual sample NSCT of the present invention;

[0059] Figure 7 This is a schematic diagram of each subband image of the face-swapping sample from the present invention using NSCT.

[0060] Figure 8 This is a schematic diagram of the NSCT subband fusion image of the real sample and the face-swapped sample of the present invention;

[0061] Figure 9 This is a schematic diagram of the MBConv layer structure of the present invention;

[0062] Figure 10 This is a schematic diagram of the structure of the backbone network EfficientNetB4 of this invention;

[0063] Figure 11 This is a schematic diagram of the texture feature extraction module of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0065] Example 1

[0066] This embodiment uses three face-swapping video databases—DeepFakeDetection (DFD), FaceForensics++ (FF++), and TIMIT—for training and testing. The DFD database contains 1089 real videos and 9204 face-swapping videos, categorized into three different compression levels: compression ratio 0 (C0), compression ratio 23 (C23), and compression ratio 40 (C40). The real video data is composed of footage shot by 28 actors in different scenes. The FF++ database contains 1000 real videos and 3000 face-swapping videos, including 1000 face-swapping videos synthesized using Deepfake manipulation. These FF++ videos are also categorized into three different compression levels: compression ratio 0 (C0), compression ratio 23 (C23), and compression ratio 40 (C40). The real video data is sourced from the video website YouTube. The TIMIT database contains 559 real videos and 640 face-swapped videos. The face-swapped videos include both low-quality (LQ) and high-quality (HQ) videos. The real video data consists of 43 objects, with 13 videos recorded for each object. Detailed information about the aforementioned publicly available face-swapped video database is shown in Table 1 below.

[0067] Table 1. Detailed information on the face-swapping video database.

[0068] DFD FF++ TIMIT Video source Camera shooting YouTube website Camera shooting resolution 1920×1080 1280×720 512×384 Number of people photographed 28 - 43 Number of real videos 1089 1000 559 Number of deepfake videos 9204 3000 640

[0069] This embodiment details the implementation process using training on the DFD(C23) database, intra-database testing on the DFD(C23) database, and cross-database testing on the FF++(C0) and TIMIT databases. The three databases are first divided into training, validation, and test sets in a 7:2:1 ratio. This embodiment is primarily based on the deep learning framework Keras. The graphics card used in the experiment is a TITAN XP, the operating system is Ubuntu 16.04, CUDA version is 9.0.176, cuDNN version is 7.4.1, and OpenCV version is 4.1.2.

[0070] like Figure 1 , Figure 2 and combined Figure 3 As shown, this embodiment provides a Deepfake video detection method based on the difference in standard scores of multi-domain feature regions, including the following steps:

[0071] S1. Divide the video of each dataset into frames and extract the face region of each frame as the region to be detected.

[0072] In this embodiment, the OpenCV tool library is used to segment the video samples of each dataset into frames, and one frame is extracted every k frames to obtain the video frame sequence {f0, f1, ..., f N-1}, where N is the total number of frames extracted, such as Figure 4 As shown, video frame images are obtained, and the face recognition algorithm of the Dlib tool library is used to extract the rectangular face regions of each frame in the video frame sequence as the regions to be detected. In this embodiment, the value of k is 5.

[0073] S2. Calculate the RGB features of the region to be detected in each frame of the image. RGB ;

[0074] In this embodiment, the detection region I of each frame image is... n The images were uniformly resized to 256×256×3 RGB images using bilinear interpolation and then normalized to obtain the spatial features I of the detection regions in each frame. RGB ,like Figure 4 As shown, the RGB feature image is obtained, where n is the video frame sequence number, n∈{0,1,…,N-1}, and N is the total number of frames extracted;

[0075] S3. Calculate the NSCT sub-band image of the region to be detected in each frame of the image;

[0076] In this embodiment, the detection region I of each frame image is... nPerforming non-subsampled contourlet transform (NSCT) yields NSCT sub-band images. For example... Figure 5 As shown, non-subsampled contourlet transform is an image transformation method that supports multi-band and multi-directional decomposition. Through non-subsampled pyramid decomposition and non-subsampled directional filter banks, it decomposes and extracts features from the image in different frequency bands according to different directions. The decomposition yields low-frequency sub-bands, horizontal mid-frequency sub-bands, vertical mid-frequency sub-bands, horizontal high-frequency sub-bands, and vertical high-frequency sub-bands. Each sub-band image has a size of 256×256×1. Figure 6 and Figure 7 As shown, the NSCT sub-band images of the real sample and the face-swapped sample are obtained;

[0077] S4. Fuse and stitch together the sub-band images of NSCT according to frequency band to obtain the frequency domain feature I. NSCT ;

[0078] like Figure 8 As shown, two directional subbands based on the intermediate frequency (IF) subband are weighted and fused in a 1:1 ratio to obtain an IF fused subband; two directional subbands based on the high frequency (HF) subband are weighted and fused in a 1:1 ratio to obtain a high frequency fused subband. Then, they are spliced ​​along the channel dimension in the order of low frequency, IF, and high frequency to obtain a frequency domain feature I with a size of 256×256×3. NSCT ;

[0079] S5. Construct a dual-branch convolutional neural network Z;

[0080] In this embodiment, the feature extraction module of the dual-branch convolutional neural network Z includes a spatial domain feature extraction branch and a frequency domain feature extraction branch. Both branches use EfficientNetB4 with an input size of 256×256×3 as the base network. Figure 9 and Figure 10 As shown, EfficientNetB4 consists of convolutional layers, MBConv layers, average pooling layers, and fully connected layers. The MBConv layer is the most important feature extraction module in the EfficientNetB4 network, which is composed of 1×1 convolutional layers, batch normalization (BN) layers, depthwise separable convolutional layers, SE attention modules, Swish activation functions, and Dropout layers.

[0081] S6. Employ a texture feature extraction module to enhance the response of the spatial feature branch to texture features, and output the texture feature F. α ;

[0082] like Figure 11As shown, the texture feature extraction module consists of three 3×3 convolutional layers, a Gram matrix calculation layer, and a global average pooling layer. The number of convolutional kernels in the three convolutional layers are 32, 16, and 32, respectively, and the convolutional strides are 1, 2, and 2, respectively. Batch normalization (BN) layers and ReLU activation functions are added after the second and third convolutional layers. In this embodiment, six texture feature extraction modules are inserted into layers of different depths in the spatial feature extraction branch to extract local detail features in the shallow network and overall abstract features in the deep network, respectively. These six modules process the input features of the spatial feature extraction branch and the output features of the first five MBConv layers of the EfficientNet-B4 backbone network, respectively. Then, the 32×1 one-dimensional vector features output by each module are concatenated to obtain a 192×1 one-dimensional vector output feature F. α .

[0083] In this embodiment, local detail features refer to features extracted from the spatial input and the output of the previous one or two MBConv layers, while overall abstract features refer to features extracted from the output of the subsequent MBConv layers. This is achieved by concatenating six 32×1 features into one 192×1 feature. The Gram matrix enhances the capture of correlations within features at the same level, thereby improving the response to texture features. The two are independent of each other.

[0084] The formula for calculating the Gram matrix in this embodiment is as follows:

[0085]

[0086] Among them, G ij This represents the value at coordinate (i,j) in the Gram matrix, c represents the number of channels in the input feature of this layer, and F represents the stretched feature of the input feature along each channel. ik Let F represent the value at coordinate (i,k). T Indicates the transpose of F;

[0087] S7. Concatenate the output features of the spatial domain feature branch and the frequency domain feature branch along the channel dimension to obtain the concatenated feature F. concat ;

[0088] S8, Splicing feature F concat Input the abnormal feature discrimination module and output the tampered region prediction mask;

[0089] like Figure 3As shown, the anomaly feature discrimination module consists of a 1×1 convolutional layer, a batch normalization layer, four special pooling layers of different sizes, a ConvLSTM layer, and a 7×7 convolutional layer. The number of kernels in the two convolutional layers and the ConvLSTM are 64, 2, and 8, respectively, and the stride of each convolution is 1. A Softmax activation function is added after the second convolutional layer. The sizes of the four special pooling layers are 7×7, 15×15, 31×31, and global, respectively. Each special pooling layer is used to calculate the standard score of the feature in the corresponding size region. The formula for calculating the standard score is as follows:

[0090]

[0091] Among them, Z l ij f represents the value at coordinates (i,j) in the output feature map of the l-th layer of the special pooling layer. l ij This represents the value at coordinates (i,j) in the input feature map of layer l. f l ij The characteristic mean of the region, s represents f l ij The standard deviation of the features in the region; when tampering is found, the feature map of the tampered face is relatively flat, so the standard deviation of the denominator is small, which makes the standard score of the abnormal features larger.

[0092] Add a one-dimensional pseudo-temporal dimension to the output features of each special pooling layer, concatenate along this dimension, and input the concatenated features into ConvLSTM for spatiotemporal analysis. The output is a tampering region prediction mask.

[0093] In this embodiment, the one-dimensional pseudo-temporal dimension refers to transforming the original 3-dimensional matrix into a 4-dimensional matrix. For example, if the output size of a special pooling layer is H×W×C, the size becomes H×W×C×1 after adding the one-dimensional pseudo-temporal dimension, and the feature size of the input ConvLSTM is H×W×C×4.

[0094] S9. Input the tampered region prediction mask into the fully connected layer, and output the features and texture features F. α The data is stitched together along the channel dimension, and finally the binary classification prediction result is output through a fully connected layer.

[0095] S10. The training of the binary classification results and the tampering region prediction mask is supervised by using binary classification labels and face-swapping region masks respectively.

[0096] S11. Train the above model, calculate the loss function and backpropagate to update the network weight coefficients, save the network model and optimal weights, including the dual-branch convolutional neural network, 6 texture feature extraction modules, anomaly feature discrimination module and the final fully connected layer, and perform end-to-end training to optimize all learnable parameters.

[0097] In this embodiment, the calculated RGB features and NSCT subbands are fused into an image [I]. RGB ,I NSCT The input is a dual-branch convolutional neural network Z, which is trained end-to-end using the Adam optimizer with a learning rate of 1×10⁻⁶. -4 The exponential decay rate beta_1 estimated by the first moment is 0.9, and the exponential decay rate beta_2 estimated by the second moment is 0.999. The ambiguity factor epsilon, used to prevent division by zero in the implementation, is 1 × 10^- ... -8 The learning rate decreases to 0.0. The cross-entropy loss function is used as the training loss function, and the network weight coefficients are iteratively updated with the goal of minimizing the loss function.

[0098] To avoid overfitting, this embodiment employs an early stopping strategy, dividing the training into two phases. Based on experience, the threshold for the maximum sustained increase of the validation set loss value is set as e1 = 6 in the first phase and e2 = 10 in the second phase. When the number of sustained increases of the validation set loss value in the first phase reaches the set threshold e1, the learning rate is reduced to 1 × 10. -6 The second stage of training is carried out. When the number of times the loss value of the validation set in the second stage continues to rise reaches the set threshold e2, the model training is terminated, and the weight with the minimum loss value is saved as the optimal weight of the network.

[0099] S12. During the testing phase, the trained dual-branch convolutional neural network Z is used for feature extraction and prediction classification, and the test set sample tampering detection results are output.

[0100] In this embodiment, the model and weights of the multi-path convolutional neural network Z trained using the training set of the DFD(C23) database are loaded. The decision threshold T is calculated using the validation set of the DFD(C23) database. The test set data of the three databases DFD(C23), FF++(C0) and TIMIT are predicted respectively. The average error rate HTER within and across databases is calculated based on the decision threshold T. The ROC curve is plotted based on the sample prediction results of each test set, and the AUC value is calculated.

[0101] The decision threshold T is determined using the equal error rate (EER) criterion. The model predicts probabilities and labels using validation set data, and calculates the false alarm rate (FAR) and false negative rate (FNR) at different thresholds. When the false alarm rate equals the false negative rate, an equal error rate state is achieved, and the current threshold is recorded as the decision threshold T. In this embodiment, the decision threshold T = 0.2505. The formulas for calculating FAR and FNR are as follows:

[0102]

[0103]

[0104] Among them, V FP V represents the number of images in the validation set where real faces were misidentified as altered faces. TN V represents the number of images in the validation set that were identified as real faces. FN V represents the number of images in the validation set whose faces were altered but were mistakenly identified as real faces. TP This represents the number of images in the validation set that were judged as having been manipulated due to facial alteration.

[0105] The Half Total Error Rate (HTER) and the Area Under the Curve (AUC) of the Receiver Operating Characteristic Curve (ROC) are used as algorithm evaluation metrics. HTER is the average of the false alarm rate and the false negative rate at the decision threshold T. The smaller the HTER value, the better the detection performance of the model. The ROC curve is a curve plotted with FPR as the x-axis and TPR as the y-axis, calculated using the sample prediction probability as the threshold. The larger the AUC value, the better the classifier performance. The specific calculation formulas for HTER, FPR, and TPR are as follows:

[0106]

[0107]

[0108]

[0109] Among them, T FP T represents the number of images in the test set where real faces were misidentified as manipulated faces. FNT represents the number of images in the test set where manipulated faces were mistakenly identified as real faces. TN T represents the number of images in the test set where real faces were identified as real faces. TP This represents the number of images in the test set that were judged as having had their faces altered.

[0110] The in-database and cross-database test results of the model trained on the DFD(C23) database in this embodiment are shown in Table 2.

[0111] Table 2 Test Results of DFD(C23) Database Training Model

[0112]

[0113] As shown in Table 2, the average error rate of the model trained on the DFD(C23) database in this embodiment is 2.3%, and the AUC area is 99.6%. The average error rate and AUC value are relatively low, indicating good results in the in-database test. The average error rate on the FF++(C0) database is 7.1%, and the AUC area is 97.8%. The average error rate on the TIMIT database is 14.8%, and the AUC area is 92.6%. The average error rate and AUC value are relatively low in the cross-database test, indicating good results in the cross-database test, thus verifying the effectiveness of the method in this embodiment.

[0114] This embodiment combines traditional hand-crafted features with deep neural networks to extract tampering traces in the spatial and frequency domains of face-swapping video data. A texture feature extraction module calculates the Gram matrix between features, enhancing the network's response to texture features. In the anomaly feature discrimination module, a special pooling layer calculates standard scores for features in different regions to determine the degree of anomaly. Then, ConvLSTM is used for spatiotemporal analysis, outputting a predicted mask for the tampered region. Binary classification labels and the face-swapping region mask are used to supervise model training, effectively reducing the average error rate in cross-database tests while maintaining good detection performance within the database. The experimental results demonstrate that this embodiment performs well in both intra-database and cross-database tests on the DFD(C23) database, effectively improving the model's generalization ability.

[0115] Example 2

[0116] This embodiment provides a Deepfake video detection system based on the difference in standard scores of multi-domain feature regions, including: a dataset partitioning module, a video data preprocessing module, a feature image calculation module, a neural network construction module, a texture feature extraction module, a feature splicing module, an anomaly feature discrimination module, a network training module, and a detection module;

[0117] In this embodiment, the dataset partitioning module is used to partition the dataset into training set, validation set and test set;

[0118] In this embodiment, the video data preprocessing module is used to divide the video of each dataset into frames and extract the face region of each frame image as the region to be detected.

[0119] In this embodiment, the feature image calculation module is used to calculate the RGB features of the region to be detected in each frame of the image, calculate the NSCT sub-band image of the region to be detected in each frame of the image, and fuse and stitch the NSCT sub-band images according to frequency bands to obtain frequency domain features.

[0120] In this embodiment, the neural network construction module is used to construct a dual-branch convolutional neural network, including a spatial domain feature extraction branch and a frequency domain feature extraction branch, and inputs RGB features and frequency domain features into the spatial domain feature extraction branch and the frequency domain feature extraction branch, respectively;

[0121] In this embodiment, the texture feature extraction module is set at different depths of the spatial feature extraction branch to extract local detail features in the shallow network and overall abstract features in the deep network, and output texture features.

[0122] In this embodiment, the feature splicing module is used to splice the output features of the spatial domain feature branch and the frequency domain feature branch along the channel dimension to obtain spliced ​​features;

[0123] In this embodiment, the abnormal feature discrimination module is used to discriminate abnormal features on the spliced ​​features and output a tampered region prediction mask;

[0124] In this embodiment, the network training module is used to input the tampering region prediction mask into the fully connected layer, concatenate the output features and texture features along the channel dimension, output the binary classification prediction result through the fully connected layer, supervise the training of the binary classification result and the tampering region prediction mask by using binary classification labels and face swapping region mask respectively, calculate the loss function and backpropagate to update the network weight coefficients, and save the network model and optimal weights.

[0125] In this embodiment, the detection module is used to perform feature extraction and prediction classification based on the trained dual-branch convolutional neural network, and output the test set sample tampering detection results.

[0126] In this embodiment, the anomaly feature discrimination module includes a 1×1 convolutional layer, a batch normalization layer, four special pooling layers of different scales, a ConvLSTM layer, and a 7×7 convolutional layer.

[0127] Each special pooling layer is used to calculate the standard score of the feature in a region of corresponding size. A one-dimensional pseudo-temporal dimension is added to the output feature of each special pooling layer, and the features are concatenated along this dimension. The concatenated features are then input into a ConvLSTM layer for spatiotemporal analysis, and the output is the tampered region prediction mask.

[0128] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for detecting Deepfake videos based on the difference in standard scores of multi-domain feature regions, characterized in that, Includes the following steps: Divide the data into different datasets, frame the video in each dataset, and extract the face region of each frame as the region to be detected. Calculate the RGB features of the region to be detected in each frame of the image; Calculate the NSCT sub-band image of the region to be detected in each frame of the image, fuse and stitch the NSCT sub-band images according to frequency bands to obtain frequency domain features; A dual-branch convolutional neural network is constructed, including a spatial domain feature extraction branch and a frequency domain feature extraction branch. RGB features and frequency domain features are input into the spatial domain feature extraction branch and the frequency domain feature extraction branch, respectively. Texture feature extraction modules are set up at different depths in the spatial feature extraction branch to extract local detail features in the shallow network and overall abstract features in the deep network, and output texture features respectively. The texture feature extraction module includes three convolutional layers, a Gram matrix calculation layer, and a global average pooling layer. Batch normalization layers and ReLU activation functions are added after the second and third convolutional layers. The texture feature extraction module processes the input features of the spatial feature extraction branch and the output features of the first five MBConv layers of the EfficientNet-B4 backbone network, respectively. Then, the one-dimensional vector features output by each module are concatenated to obtain a one-dimensional vector output feature. The output features of the spatial domain feature branch and the frequency domain feature branch are concatenated along the channel dimension to obtain the concatenated features; The spliced ​​features are input into the anomaly feature discrimination module for anomaly feature discrimination, and the output is the tampered region prediction mask; The anomaly feature discrimination module includes a 1×1 convolutional layer, a batch normalization layer, four special pooling layers of different scales, a ConvLSTM layer, and a 7×7 convolutional layer. Each special pooling layer is used to calculate the standard score of the feature in the region of the corresponding size. A one-dimensional pseudo-temporal dimension is added to the output feature of each special pooling layer, and the feature is concatenated along this dimension. The concatenated feature is then input into the ConvLSTM layer for spatiotemporal analysis, and the output is the tampered region prediction mask. The tampering region prediction mask is input into the fully connected layer, and the output features and texture features are concatenated along the channel dimension. The binary classification prediction result is output through the fully connected layer. The training of the binary classification result and the tampering region prediction mask is supervised by the binary classification label and the face swapping region mask, respectively. The loss function is calculated and the network weight coefficients are updated by backpropagation. The network model and the optimal weights are saved. Feature extraction and prediction classification are performed based on the trained dual-branch convolutional neural network, and the results of test set sample tampering detection are output.

2. The Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to claim 1, characterized in that, The specific steps for calculating the NSCT sub-band image of the region to be detected in each frame of the image include: Non-subsampled contourlet transform is performed on the detection region of each frame image to obtain NSCT sub-band images. Non-subsampled pyramid decomposition and non-subsampled directional filter bank are used to decompose and extract features of the image in different frequency bands according to different directions. After decomposition, low-frequency sub-band, horizontal mid-frequency sub-band, vertical mid-frequency sub-band, horizontal high-frequency sub-band and vertical high-frequency sub-band are obtained.

3. The Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to claim 2, characterized in that, The specific steps for fusing and stitching together the various sub-band images of NSCT according to frequency bands to obtain frequency domain features include: Two directional subbands based on intermediate frequency subbands are weighted and fused according to a preset ratio to obtain an intermediate frequency fused subband; Two directional subbands based on high-frequency subbands are weighted and fused according to a preset ratio to obtain a high-frequency fused subband. The frequency domain features are obtained by splicing the low-frequency, mid-frequency, and high-frequency components along the channel dimension.

4. The Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to claim 1, characterized in that, Both the spatial domain feature extraction branch and the frequency domain feature extraction branch use EfficientNetB4 as the base network. EfficientNetB4 includes convolutional layers, MBConv layers, average pooling layers, and fully connected layers.

5. The Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to claim 1, characterized in that, The formula for calculating the Gram matrix in the Gram matrix calculation layer is as follows: Among them, G ij This represents the value at coordinate (i,j) in the Gram matrix, c represents the number of channels in the input feature of this layer, and F represents the stretched feature of the input feature along each channel. ik Let F represent the value at coordinate (i,k). T This represents the transpose of F.

6. The Deepfake video detection method based on the difference in standard scores of multi-domain feature regions according to claim 1, characterized in that, The formula for calculating the standard score is as follows: Among them, Z l ij f represents the value at coordinates (i,j) in the output feature map of the l-th layer of the special pooling layer. l ij This represents the value at coordinates (i,j) in the input feature map of layer l. f l ij The characteristic mean of the region, s represents f l ij The characteristic standard deviation of the region.

7. A Deepfake video detection system based on the difference in standard scores of multi-domain feature regions, characterized in that, include: The system includes a dataset partitioning module, a video data preprocessing module, a feature image calculation module, a neural network construction module, a texture feature extraction module, a feature stitching module, an anomaly feature discrimination module, a network training module, and a detection module. The dataset partitioning module is used to partition the datasets. The video data preprocessing module is used to divide the video of each dataset into frames and extract the face region of each frame image as the region to be detected. The feature image calculation module is used to calculate the RGB features of the region to be detected in each frame image, calculate the NSCT sub-band image of the region to be detected in each frame image, and fuse and stitch the NSCT sub-band images according to frequency bands to obtain frequency domain features. The neural network construction module is used to construct a dual-branch convolutional neural network, including a spatial domain feature extraction branch and a frequency domain feature extraction branch, and inputs RGB features and frequency domain features into the spatial domain feature extraction branch and the frequency domain feature extraction branch respectively; The texture feature extraction module is set at different depths of the spatial feature extraction branch to extract local detail features in the shallow network and overall abstract features in the deep network, and output texture features. The texture feature extraction module includes three convolutional layers, a Gram matrix calculation layer, and a global average pooling layer. Batch normalization layers and ReLU activation functions are added after the second and third convolutional layers. The texture feature extraction module processes the input features of the spatial feature extraction branch and the output features of the first five MBConv layers of the EfficientNet-B4 backbone network, respectively. Then, the one-dimensional vector features output by each module are concatenated to obtain a one-dimensional vector output feature. The feature splicing module is used to splice the output features of the spatial domain feature branch and the frequency domain feature branch along the channel dimension to obtain spliced ​​features; The abnormal feature discrimination module is used to discriminate abnormal features in the spliced ​​features and output a tampered region prediction mask. The anomaly feature discrimination module includes a 1×1 convolutional layer, a batch normalization layer, four special pooling layers of different scales, a ConvLSTM layer, and a 7×7 convolutional layer. Each special pooling layer is used to calculate the standard score of the feature in the region of the corresponding size. A one-dimensional pseudo-temporal dimension is added to the output feature of each special pooling layer, and the feature is concatenated along this dimension. The concatenated feature is then input into the ConvLSTM layer for spatiotemporal analysis, and the output is the tampered region prediction mask. The network training module is used to input the tampering region prediction mask into the fully connected layer, concatenate the output features and texture features along the channel dimension, output the binary classification prediction result through the fully connected layer, supervise the training of the binary classification result and the tampering region prediction mask by using binary classification labels and face swapping region masks respectively, calculate the loss function and backpropagate to update the network weight coefficients, and save the network model and the optimal weights. The detection module is used to perform feature extraction and prediction classification based on a trained dual-branch convolutional neural network, and output the test set sample tampering detection results.

Images