A method and apparatus for detecting deepfake videos

Abstract

This application discloses a method and apparatus for detecting deepfake videos. First, face extraction and label generation are performed on video frames from a video dataset to obtain training sample video frames. Image enhancement processing is then performed on these frames to generate multiple enhanced sample images. Next, a three-channel co-occurrence matrix is ​​calculated from the enhanced sample images to obtain the co-occurrence matrix image, which is then input into an attribution network. An N-dimensional weight vector is obtained based on the forgery algorithm labels. A deepfake detection network is constructed and trained on the video dataset to obtain the model output. Finally, the model output and the N-dimensional weight vector are weighted and averaged to obtain the video frame forgery detection result. By employing data augmentation and algorithm attribution processing operations specifically for deepfake faces, the method can guide the effective fusion of detection models from different types of generative algorithms, thereby improving the accuracy and generalization of deepfake video detection using unknown generative algorithms in real-world detection scenarios.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method and apparatus for detecting video depth forgery. Background Technology

[0002] Deepfake technology is a technique that automatically edits or synthesizes fake content based on artificial intelligence methods such as deep learning. In recent years, with the development of deep learning information technology in the field of computer vision, deepfake technology has become more intelligent and streamlined, significantly reducing the cost and barrier to entry for forgery. Furthermore, relying on powerful intelligent algorithms and continuously improving deepfake models, the generated forged images can achieve a level of realism that is difficult to distinguish from the real thing. However, malicious deepfake images, especially deepfake facial video data targeting public figures, have spread rapidly on social media and content sharing platforms in recent years, attracting widespread public attention. The generation and dissemination of this audio and video data seriously erodes social trust and disrupts work and life.

[0003] Currently, deepfake video technology mainly focuses on facial video forgery, including face swapping, attribute modification, and full-face synthesis. Therefore, current research on deepfake detection technology is primarily based on experiments using datasets constructed from typical forgery methods. Related patents mainly protect rights by incorporating current academic research trends, focusing on innovative detection methods for single data points and single algorithms on typical academic datasets.

[0004] Although advanced deepfake detection algorithms demonstrate good accuracy on mainstream academic datasets such as FF++, DFDC, and Celeb-DF, their performance drops sharply when the detection model is mismatched with the generation algorithm of the training data (e.g., a model trained on FF++ is applied to the Celeb-DF dataset for detection). This is because different datasets are often composed of different generation algorithms, and the data sample compression methods, resolutions, and noise levels are inconsistent, which greatly affects the robustness of network feature extraction. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of existing technologies and provide a video deepfake detection method and apparatus. By applying data augmentation and algorithm attribution processing operations for deepfake videos targeting face deepfakes, the method guides the effective fusion of detection models of different generation algorithms, thereby improving the accuracy and generalization of deepfake video detection using unknown generation algorithms in actual detection scenarios.

[0006] The objective of this application is achieved through the following technical solution:

[0007] Firstly, this application proposes a method for detecting deepfake videos, including:

[0008] The collected video dataset is subjected to face extraction and label generation from video frames to obtain training sample video frames, which are labeled with fake algorithm tags.

[0009] The training sample video frames are subjected to image enhancement processing to form multiple enhanced sample images;

[0010] The three-channel co-occurrence matrix of the enhanced sample image is obtained to obtain the co-occurrence matrix image. The co-occurrence matrix image is input into the attribution network, and an N-dimensional weight vector is obtained according to the forgery algorithm label.

[0011] Construct a deepfake detection network, and use the deepfake detection network to train the video dataset to obtain the model output results;

[0012] The video frame forgery detection result is obtained by using the weighted fusion averaging method with the output of the model and the N-dimensional weight vector.

[0013] Optionally, the steps of extracting faces from video frames and generating labels on the collected video dataset to obtain training sample video frames include:

[0014] The video dataset is sampled at preset intervals to obtain multiple video frames with forged algorithm tags;

[0015] The key point matrix of each face image in the video frame is obtained using the MTCNN model;

[0016] The face region containing the key point matrix is ​​enlarged and cropped to obtain training sample video frames.

[0017] Optionally, the step of performing image enhancement processing on the training sample video frames to form multiple enhanced sample images includes:

[0018] According to a preset probability, the training sample video frames are subjected to image enhancement processing based on a first cropping scheme, such as compression, Gaussian noise, horizontal flipping, brightness adjustment, contrast adjustment, chroma adjustment, and affine transformation; or a second cropping scheme based on contour feature points in the keypoint matrix; or a third cropping scheme based on nose feature points in the keypoint matrix; or a fourth cropping scheme based on eye feature points in the keypoint matrix; or a fifth cropping scheme based on lip feature points in the keypoint matrix, thereby forming multiple enhanced sample images.

[0019] Optionally, the step of obtaining a co-occurrence matrix image by calculating a three-channel co-occurrence matrix for the enhanced sample image, and inputting the co-occurrence matrix image into the attribution network to obtain an N-dimensional weight vector based on the forgery algorithm label includes:

[0020] The enhanced sample image is separated into RGB three channels, and the co-occurrence pair values ​​of adjacent pixels of each pixel in a single image in each channel are calculated to obtain the co-occurrence matrix;

[0021] The co-occurrence matrix is ​​normalized and then superimposed with depth dimensions to obtain a co-occurrence matrix image;

[0022] The co-occurrence matrix image is input into the attribution network, so that the attribution network obtains an N-dimensional weight vector based on the forgery algorithm label.

[0023] Optionally, the attribution network includes an input layer, a fully connected layer, and an output layer. The step of obtaining an N-dimensional weight vector based on the forged algorithm labels includes:

[0024] The input layer fine-tunes the co-occurrence matrix image based on the co-occurrence matrix image, transforms the co-occurrence matrix image into a 2048-dimensional tensor image, and sends it to the fully connected layer;

[0025] The fully connected layer is fine-tuned according to the forgery algorithm labels, transforming the image of the 2048-dimensional tensor into an image whose tensor is the total number of categories of the forgery algorithm labels, and then sending it to the output layer;

[0026] The output layer transforms the image, whose tensor represents the total number of categories of the forgery algorithm labels, into an N-dimensional weight vector using a multi-class activation function.

[0027] Optionally, the attribution network employs a multi-class cross-entropy loss function, wherein the multi-class cross-entropy loss function is logloss. Attri for:

[0028]

[0029] Where N is the number of generative algorithm categories, y ic It is a sign function, where i is a sample and p is a symbolic function. ic Let be the predicted probability that sample i belongs to category c.

[0030] Optionally, the step of constructing a deepfake detection network and using the deepfake detection network to train the video dataset to obtain the model output includes:

[0031] The EfficientNet model trained on the ImagNet dataset was used as the initial model to build a deep forgery detection network;

[0032] The video dataset is divided into multiple subsets using the deepfake detection network.

[0033] Select two subsets and label one of them to obtain the labeled subset and the unlabeled subset;

[0034] A teacher classification model is trained on the labeled subset, and the unlabeled subset is classified using the teacher classification model to obtain the classification result;

[0035] The classification results are filtered to obtain soft-label data pairs, and the labeled subset and the soft-label data pairs are used to train the student model;

[0036] The student model is used to train additional subsets selected from multiple subsets until the expected number of training iterations is reached or the network converges and training stops, thus obtaining the model output.

[0037] Secondly, this application proposes a video deepfake detection device, the device comprising:

[0038] The extraction module is used to extract faces from video frames and generate labels on the collected video dataset to obtain training sample video frames, which are labeled with fake algorithm tags.

[0039] The image processing module is used to perform image enhancement processing on the training sample video frames to form multiple enhanced sample images;

[0040] The vector generation module is used to obtain a co-occurrence matrix image by calculating the three-channel co-occurrence matrix of the enhanced sample image, and input the co-occurrence matrix image into the attribution network to obtain an N-dimensional weight vector based on the forgery algorithm label;

[0041] A network construction module is used to construct a deepfake detection network and train the video dataset using the deepfake detection network to obtain the model output results;

[0042] The result generation module is used to obtain the video frame forgery detection result by using the weighted fusion averaging method to combine the model output result and the N-dimensional weight vector.

[0043] Thirdly, this application also proposes a computer device comprising a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the video depth forgery detection method as described in any of the first aspects.

[0044] Fourthly, this application also proposes a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the video depth forgery detection method as described in any of the first aspects.

[0045] The main solution and its various further alternatives described above can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application; furthermore, the (non-conflicting alternatives) can also be freely combined with each other and with other alternatives. Those skilled in the art, after understanding the solution of this application, will realize from the prior art and common general knowledge that there are many combinations, all of which are technical solutions to be protected by this application, and will not be exhaustively listed here.

[0046] The beneficial effects of this application are as follows:

[0047] First, this application combines multiple methods to enhance image processing, which can effectively improve the accuracy of a single detection algorithm.

[0048] Secondly, classifying N types of forgery generation methods according to algorithm similarity can reduce the number of categories in the detection model and lower the complexity of model fusion. Using attribution algorithms to attribute the generation algorithms of forged videos generated by unknown methods can guide the fusion of different models trained on multiple types of data, effectively improving the generalization and accuracy of detection algorithms in complex environments. Attached Figure Description

[0049] Figure 1 A schematic flowchart of a video depth forgery detection method proposed in an embodiment of this application is shown.

[0050] Figure 2 The flowchart illustrating the one-dimensional classification results of the EfficientNet network model provided in this application is shown. Detailed Implementation

[0051] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0052] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0053] In existing technologies, although advanced deepfake detection algorithms can demonstrate good accuracy on mainstream academic datasets such as FF++, DFDC, and Celeb-DF, the performance of the algorithm will drop sharply when the detection model is mismatched with the generation algorithm of the training data (e.g., a model trained on FF++ is applied to the Celeb-DF dataset for detection). This is because different datasets are often composed of different generation algorithms, and the data sample compression methods, resolutions, and noise levels are inconsistent, which will greatly affect the robustness of network feature extraction.

[0054] To address the aforementioned technical issues, this application proposes a video deepfake detection method. This method not only improves the interpretability of detection results but also guides the fusion of multiple detection models based on attribution results, thereby enhancing the accuracy and generalization of fake detection. The following is a detailed explanation of this method.

[0055] Please refer to Figure 1 , Figure 1 This paper presents a schematic flowchart of a video depth fake detection method according to an embodiment of the present application, which includes the following steps:

[0056] S100. Perform face extraction and label generation on the collected video dataset to obtain training sample video frames.

[0057] Video forgery detection mainly targets facial videos. There are two detection methods: one is to perform consistency detection between frames, and the other is to perform frame detection on video images within a frame.

[0058] The training sample video frames are labeled with fake algorithms. The collected video datasets mainly include real and fake datasets constructed with features such as deepfakes, faceswap, face2face, neuraltextures, deepfacelab, ALAE, FSGAN, and FOMM, as well as real and fake datasets collected by networks such as WildDeepfake. Through the above fake methods, typical academic research datasets such as FF++ (FaceForebsics++), Celeb-DF, WildDeepfake, and DFDC are constructed. The video datasets collected in this application are mainly used for training and testing the model.

[0059] Step S100 further includes the following steps:

[0060] S110. Frame sampling is performed on the video dataset at preset intervals to obtain multiple video frames with fake algorithm labels.

[0061] S120. Use the MTCNN model to obtain the key point matrix of each face image in the video frame.

[0062] S130. Expand and crop the face region where the key point matrix is ​​located to obtain training sample video frames.

[0063] The MTCNN (Multi-task convolutional neural network) model is applied to the face alignment stage in the field of face recognition, and has the advantages of high speed, low false detection rate and strong generalization ability.

[0064] In one possible embodiment, a video is generated by a fake algorithm A. Frame sampling is performed at preset intervals of 10 frames to obtain 30 video frames. The MTCNN model is used to perform face clipping detection on the faces in each video frame to obtain the 68 key point matrix of the faces in each video frame. The extracted face regions are then expanded by 30% around the edges to crop the face image and form a new image. These 30 cropped images are then labeled with a dual label of "fake" and "algorithm A" as a fake algorithm label. If it is a real video frame, it is labeled "real" + "no algorithm". The "fake" and "algorithm A" labels are used for classification by the fake detection module, while the "real" + "no algorithm" labels are used for classification by the attribution module.

[0065] S200. Perform image enhancement processing on the training sample video frames to form multiple enhanced sample images.

[0066] To improve the generalization performance of the network, data augmentation is necessary during training on sample video frames to enhance its robustness during actual prediction. Since the most widely used datasets in academic research, such as FF++ and Celeb-DF, are derived from a limited number of actors and celebrities with swapped faces, their data distribution is relatively homogeneous. Therefore, detection networks may experience overfitting when multiple videos use the same face. In such cases, the network tends to overfit the facial features of the actual person rather than the features of the deepfake, resulting in low generalization. The image augmentation process proposed in this application effectively addresses these issues.

[0067] For known genuine and fake video pairs, in order to avoid the network overfitting to a single obvious feature, the SSIM parameters of the pair of genuine and fake video frames are calculated to obtain the grayscale mask of the image's authenticity difference. If the sum of the pixel values ​​of the mask grayscale image is greater than a preset difference threshold, then the image is cropped based on a random rectangular region.

[0068] The image enhancement process involves performing image enhancement on training sample video frames according to a preset probability. This is done using either a first cropping scheme (compression, Gaussian noise, horizontal flipping, brightness adjustment, contrast adjustment, chroma adjustment, affine transformation, etc.), a second cropping scheme (cropping based on contour feature points in the keypoint matrix), a third cropping scheme (cropping based on nose feature points in the keypoint matrix), a fourth cropping scheme (cropping based on eye feature points in the keypoint matrix), or a fifth cropping scheme (cropping based on lip feature points in the keypoint matrix), thereby generating multiple enhanced sample images.

[0069] The first cropping scheme A1 is a traditional data augmentation scheme, which enhances the diversity of data samples through processing such as compression, Gaussian noise, horizontal flipping, brightness adjustment, contrast adjustment, chroma adjustment, and affine transformation.

[0070] The second cropping scheme A2 is based on the contour feature points 0 to 26 in the 68 key point matrix of the face. It randomly selects 3 to 6 feature points to form a polygonal convex hull, and sets the pixel values ​​inside the convex hull to 0, while keeping the pixel values ​​outside the convex hull unchanged from the original image.

[0071] The third cropping scheme A3 is based on the nose feature points 27 to 35 in the 68 key point matrix of the face. It constructs a polygonal convex hull based on the corresponding feature points, sets the pixel values ​​inside the convex hull to 0, and keeps the pixel values ​​outside the convex hull unchanged from the original image.

[0072] The fourth cropping scheme A4 is based on the left and right eye feature points 36 to 47 in the 68 key point matrix of the face. It constructs a polygonal convex hull based on the corresponding feature points, sets the pixel values ​​inside the convex hull to 0, and keeps the pixel values ​​outside the convex hull unchanged from the original image.

[0073] The fifth cropping scheme, A5, is based on the lip feature points 48 to 67 in the 68 key point matrix of the face. It constructs a polygonal convex hull based on the corresponding feature points, sets the pixel values ​​inside the convex hull to 0, and keeps the pixel values ​​outside the convex hull unchanged from the original image.

[0074] Image enhancement processing is performed by using image databases such as Albumentations and executing different cropping schemes with different preset probabilities. For example, the first cropping scheme A1 is implemented with probability R1, the second cropping scheme A2 is implemented with probability R2, the third cropping scheme A3 is implemented with probability (1-R2)R3, the fourth cropping scheme A4 is implemented with probability (1-R2)R4, and the fifth cropping scheme A5 is implemented with probability (1-R2)R5, resulting in multiple enhanced sample images.

[0075] The image enhancement processing proposed in this application, through a face-based deepfake data enhancement scheme, can improve the data's resistance to compression, noise, and other interference, as well as the robustness of the training model. Simultaneously, by randomly reducing key features of the person's facial features, the dependence of the deep network on features in the training data can be reduced, preventing overfitting of the deep network to a single fake feature. This allows the deep network to better learn the fake characteristics of the algorithm, improving the generalization performance of the detection algorithm.

[0076] S300. Obtain the co-occurrence matrix image by calculating the three-channel co-occurrence matrix of the enhanced sample image. Input the co-occurrence matrix image into the attribution network and obtain the N-dimensional weight vector based on the forgery algorithm label.

[0077] The attribution network employs a multi-class cross-entropy loss function, specifically logloss. Attri for:

[0078] Where N is the number of generative algorithm categories, y ic It is a sign function, where i is a sample and p is a symbolic function. ic Let be the predicted probability that sample i belongs to category c.

[0079] The specific steps for obtaining the N-dimensional weight vector in step S300 are as follows:

[0080] S310. Separate the enhanced sample image according to the RGB three channels, and calculate the co-occurrence pair value of each pixel in each channel to obtain the co-occurrence matrix.

[0081] S320. Normalize the co-occurrence matrix and superimpose the depth dimensions to obtain a co-occurrence matrix image.

[0082] S330. Input the co-occurrence matrix image into the attribution network so that the attribution network obtains an N-dimensional weight vector based on the forgery algorithm label.

[0083] Since video frames are typically three-channel color images, the co-occurrence matrix is ​​calculated to obtain the similarity features of adjacent pixels. First, the enhanced sample image obtained in step S200 is separated into RGB three channels and processed separately. The co-occurrence pair values ​​[i, j] of adjacent pixels in each single image of each channel are calculated. To avoid redundancy, taking a single pixel as an example, the adjacent pixels can be the right, lower right, lower, and lower left of its eight adjacent pixels. For any 8-bit depth image, a co-occurrence matrix of size 256×256 can be obtained. If the pixels below a single pixel are taken, the following calculation formula is used:

[0084]

[0085] Where I is the input single-channel image, [m, n] are pixel coordinates, and [i, j] are pixel co-occurrence pairs.

[0086] By processing the enhanced sample image through the RGB three channels, and selecting four points (right, bottom right, bottom, and bottom left) as adjacent pixels for each channel image to obtain the co-occurrence matrix, 12 (3*4) co-occurrence matrices of size 256*256 can be obtained. The co-occurrence matrix is ​​then normalized and the depth dimension is superimposed to obtain a 256×256×12 co-occurrence matrix image.

[0087] Attribution networks are fine-tuned CNN classification networks. Typical classification networks include ResNet, Xception, and EfficientNet. Attribution networks consist of an input layer, a fully connected layer, and an output layer. When the co-occurrence matrix image is input into the attribution network, the attribution network obtains an N-dimensional weight vector based on the fake algorithm labels.

[0088] The input layer is fine-tuned based on the co-occurrence matrix image, and the co-occurrence matrix image is transformed into a 2048-dimensional tensor image and sent to the fully connected layer.

[0089] The fully connected layer is fine-tuned according to the forgery algorithm labels, transforming the 2048-dimensional tensor image into an image whose tensor is the total number of categories of the forgery algorithm labels, and then sending it to the output layer;

[0090] The output layer transforms the image, whose tensor is the total number of categories of the forgery algorithm labels, into an N-dimensional weight vector using a multi-class activation function.

[0091] The input layer tensor is 299*299*3 in size. To match the size of the co-occurrence matrix image, it is fine-tuned to a tensor of 256*256*12, resulting in a 1*2048 dimensional tensor image output to the fully connected layer. The fully connected layer is fine-tuned to a tensor of 1*N according to the forgery algorithm labels, where N represents the total number of categories of the forgery algorithm labels. This image is then sent to the output layer. The output layer processes the 1*N tensor image using a multi-class activation function (Softmax function) and finally outputs an N-dimensional weight vector consisting of the probabilities of different category generation algorithms, with a sum of 1.

[0092] S400. Construct a deepfake detection network and use the deepfake detection network to train the video dataset to obtain the model output results.

[0093] The process of obtaining the model output result in step S400 includes the following steps:

[0094] S410. Use the EfficientNet model trained in the ImagNet dataset as the initial model to build a deep forgery detection network;

[0095] S420: The video dataset is divided into multiple subsets using a deepfake detection network.

[0096] S430. Select two subsets and label one of them to obtain the labeled subset and the unlabeled subset.

[0097] S440. Train a teacher classification model on the labeled subset, and use the teacher classification model to classify the unlabeled subset to obtain the classification result.

[0098] S450. Filter the classification results to obtain soft-label data pairs, and use the labeled subset and soft-label data pairs to train the student model.

[0099] S460. Use the student model to train additional subsets selected from multiple subsets until the expected number of training iterations is reached or the network converges and training stops to obtain the model output results.

[0100] The ImagNet dataset is a massive image classification dataset. Image classification networks all need to be pre-trained on the ImagNet dataset, and the pre-trained models of typical classification networks on this dataset are already very mature and can be directly downloaded and used as initial models. The backbone network of the deep forgery detection network is the EfficientNet model. It utilizes the image attention mechanism of the EfficientNet model to guide the network to learn forgery features in key regions, and at the same time, it adopts a semi-supervised self-training method to improve the network's classification performance.

[0101] In addition, shallow texture feature maps can be used as attention maps to guide the network's deep feature classification learning. Visualizing the attention map can locate the position in deepfake artifacts that has been tampered with by deepfake technology, thereby improving the reliability of the detection results.

[0102] The EfficientNetB4 network model will be explained next; please refer to [link / reference]. Figure 2 , Figure 2The flowchart illustrating the one-dimensional classification result of the EfficientNet network model provided in this application is shown. The input is a co-occurrence matrix image, which is fine-tuned using an attention mechanism. This network model contains multiple MBConv (mobile inverted bottleneck convolution) modules and one attention module. When the attention module is added after the third MBConv3 module, since the output of the MBConv3 module is an f1 feature map, after 1×1 convolution, normalization, and ReLU activation by the attention module, the resulting f2 feature map is used as the attention map. The f2 feature map is then used to guide the f1 feature map through element-wise multiplication to obtain the output feature map. This output feature map is used as the input of the next MBConv4 module. Finally, the EfficientNetB4 network model obtains the one-dimensional classification result after Sigmoid activation.

[0103] After constructing the deepfake detection network, it is necessary to perform semi-supervised training on the self-trained semi-supervised learning method. The semi-supervised training process is as follows:

[0104] The first step is to divide the video dataset into multiple subsets;

[0105] The second step is to randomly select two subsets from the multiple subsets, and mark one of the subsets to obtain the marked subset and the unmarked subset.

[0106] The third step is to train the teacher classification model on the labeled subset. Since the teacher classification model is a binary classification model, its cross-entropy loss function is minimized.

[0107] The fourth step is to use the teacher classification model to classify the unlabeled subsets and obtain the classification results of the unlabeled subsets.

[0108] Step 5: Filter the data with classification results between [0.3, 0.7] to obtain soft-label data pairs composed of multiple sample pairs.

[0109] Step 6: Use the labeled subset and soft-labeled data pairs as new mixed training data to train the student model and minimize its cross-entropy loss function on the mixed training data.

[0110] Step 7: Use the trained student model as the teacher model and return to step 2 to continue training until the expected number of training iterations is reached or the network converges. Train on M major classes of data to obtain the model output results.

[0111] Furthermore, training with a semi-supervised teacher-student model can improve the classification performance of pre-trained networks on deep fake datasets.

[0112] S500: The model output and the N-dimensional weight vector are combined using a weighted fusion averaging method to obtain the video frame forgery detection result.

[0113] First, the weight vectors of N generation algorithms are divided into M types of data according to the principle of adding the weights of the same type of data. Then, the M-dimensional vectors are converted into M-dimensional vectors and the model output results are integrated into M cross-dataset classification models using a weighted fusion averaging method. Finally, the video frame forgery detection results are output after the model integration and fusion is guided by attribution weights. This realizes the fusion of forgery detection models with multiple algorithms and multiple data. Combined with the attribution results and attention map visualization results, the interpretability of forgery detection is improved.

[0114] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0115] First, this application combines multiple methods to enhance image processing, which can effectively improve the accuracy of a single detection algorithm.

[0116] Secondly, classifying N types of forgery generation methods according to algorithm similarity can reduce the number of categories in the detection model and lower the complexity of model fusion. Using attribution algorithms to attribute the generation algorithms of forged videos generated by unknown methods can guide the fusion of different models trained on multiple types of data, effectively improving the generalization and accuracy of detection algorithms in complex environments.

[0117] The following describes a possible implementation of video depth-spoofing detection, which performs the various execution steps and corresponding technical effects of the video depth-spoofing detection method shown in the above embodiments and possible implementations. The video depth-spoofing detection apparatus includes:

[0118] The extraction module is used to extract faces from video frames and generate labels on the collected video dataset to obtain training sample video frames, which are labeled with fake algorithm tags.

[0119] The image processing module is used to perform image enhancement processing on the training sample video frames to form multiple enhanced sample images;

[0120] The vector generation module is used to obtain a co-occurrence matrix image by calculating the three-channel co-occurrence matrix of the enhanced sample image. The co-occurrence matrix image is then input into the attribution network to obtain an N-dimensional weight vector based on the forgery algorithm label.

[0121] The network building module is used to build a deepfake detection network and use the deepfake detection network to train the video dataset to obtain the model output results;

[0122] The result generation module is used to obtain the video frame forgery detection result by weighted fusion averaging the model output and the N-dimensional weight vector.

[0123] Furthermore, this preferred embodiment provides a computer device that can implement the steps in any embodiment of the video depth forgery detection method provided in this application. Therefore, it can achieve the beneficial effects of the video depth forgery detection method provided in this application, as detailed in the preceding embodiments, which will not be repeated here.

[0124] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of this application provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the video depth-spoofing detection method provided in this application.

[0125] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0126] Since the instructions stored in the storage medium can execute the steps in any of the video depth forgery detection method embodiments provided in this application, the beneficial effects that any of the video depth forgery detection methods provided in this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0127] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting deepfake videos, characterized in that, include: The collected video dataset is subjected to face extraction and label generation from video frames to obtain training sample video frames, which are labeled with fake algorithm tags. The training sample video frames are subjected to image enhancement processing to form multiple enhanced sample images; The enhanced sample image is separated into RGB three channels, and the co-occurrence pair values ​​of adjacent pixels of each pixel in a single image in each channel are calculated to obtain the co-occurrence matrix; The co-occurrence matrix is ​​normalized and then superimposed with depth dimensions to obtain a co-occurrence matrix image; The co-occurrence matrix image is input into the attribution network, so that the attribution network obtains an N-dimensional weight vector based on the forgery algorithm label; The attribution network includes an input layer, a fully connected layer, and an output layer. The step of obtaining an N-dimensional weight vector based on the forged algorithm labels includes: The input layer fine-tunes the co-occurrence matrix image based on the co-occurrence matrix image, transforms the co-occurrence matrix image into a 2048-dimensional tensor image, and sends it to the fully connected layer; The fully connected layer is fine-tuned according to the forgery algorithm labels, transforming the image of the 2048-dimensional tensor into an image whose tensor is the total number of categories of the forgery algorithm labels, and then sending it to the output layer; The output layer transforms the image, in which the tensor represents the total number of categories of the forgery algorithm labels, into an N-dimensional weight vector using a multi-class activation function. The attribution network employs a multi-class cross-entropy loss function, specifically the logloss function. Attri for: Where N is the number of generation algorithm categories, y ic It is a sign function, where i is a sample and p is a symbolic function. ic Let i be the predicted probability that sample i belongs to category c; Construct a deepfake detection network, and use the deepfake detection network to train the video dataset to obtain the model output results; The video frame forgery detection result is obtained by using the weighted fusion averaging method with the output of the model and the N-dimensional weight vector.

2. The video deepfake detection method as described in claim 1, characterized in that, The steps for extracting faces from video frames and generating labels on the collected video dataset to obtain training sample video frames include: The video dataset is sampled at preset intervals to obtain multiple video frames with forged algorithm tags; The key point matrix of each face image in the video frame is obtained using the MTCNN model; The face region containing the key point matrix is ​​enlarged and cropped to obtain training sample video frames.

3. The video deepfake detection method as described in claim 2, characterized in that, The step of performing image enhancement processing on the training sample video frames to form multiple enhanced sample images includes: According to a preset probability, the training sample video frames are subjected to image enhancement processing based on a first cropping scheme, such as compression, Gaussian noise, horizontal flipping, brightness adjustment, contrast adjustment, chroma adjustment, and affine transformation; or a second cropping scheme based on contour feature points in the keypoint matrix; or a third cropping scheme based on nose feature points in the keypoint matrix; or a fourth cropping scheme based on eye feature points in the keypoint matrix; or a fifth cropping scheme based on lip feature points in the keypoint matrix, thereby forming multiple enhanced sample images.

4. The video deepfake detection method as described in claim 1, characterized in that, The steps of constructing a deepfake detection network and training the video dataset using the deepfake detection network to obtain the model output include: The EfficientNet model trained on the ImagNet dataset was used as the initial model to build a deep forgery detection network; The video dataset is divided into multiple subsets using the deepfake detection network. Select two subsets and label one of them to obtain the labeled subset and the unlabeled subset; A teacher classification model is trained on the labeled subset, and the unlabeled subset is classified using the teacher classification model to obtain the classification result; The classification results are filtered to obtain soft-label data pairs, and the labeled subset and the soft-label data pairs are used to train the student model; The student model is used to train additional subsets selected from multiple subsets until the expected number of training iterations is reached or the network converges and training stops, thus obtaining the model output.

5. A video deepfake detection device, characterized in that, The device includes: The extraction module is used to extract faces from video frames and generate labels on the collected video dataset to obtain training sample video frames, which are labeled with fake algorithm tags. The image processing module is used to perform image enhancement processing on the training sample video frames to form multiple enhanced sample images; The vector generation module is used to separate the enhanced sample image according to the RGB three channels, calculate the co-occurrence pair value of each pixel in a single image in each channel, and obtain the co-occurrence matrix; The co-occurrence matrix is ​​normalized and then superimposed with depth dimensions to obtain a co-occurrence matrix image; The co-occurrence matrix image is input into the attribution network, so that the attribution network obtains an N-dimensional weight vector based on the forgery algorithm label; The attribution network includes an input layer, a fully connected layer, and an output layer. The step of obtaining an N-dimensional weight vector based on the forged algorithm labels includes: The input layer fine-tunes the co-occurrence matrix image based on the co-occurrence matrix image, transforms the co-occurrence matrix image into a 2048-dimensional tensor image, and sends it to the fully connected layer; The fully connected layer is fine-tuned according to the forgery algorithm labels, transforming the image of the 2048-dimensional tensor into an image whose tensor is the total number of categories of the forgery algorithm labels, and then sending it to the output layer; The output layer transforms the image, in which the tensor represents the total number of categories of the forgery algorithm labels, into an N-dimensional weight vector using a multi-class activation function. The attribution network employs a multi-class cross-entropy loss function, specifically the logloss function. Attri for: Where N is the number of generation algorithm categories, y ic It is a sign function, where i is a sample and p is a symbolic function. ic Let i be the predicted probability that sample i belongs to category c; A network construction module is used to construct a deepfake detection network and train the video dataset using the deepfake detection network to obtain the model output results; The result generation module is used to obtain the video frame forgery detection result by using the weighted fusion averaging method to combine the model output result and the N-dimensional weight vector.

6. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, which is loaded and executed by the processor to implement the video depth forgery detection method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is loaded and executed by a processor to implement the video depth forgery detection method as described in any one of claims 1-4.

Images