Deep counterfeit video detection method based on local pixel relation and face-background multi-proportion branch

Abstract

The invention relates to a deep counterfeit video detection method based on a local pixel relation and a face-background multi-proportion branch. The method mainly comprises the following steps: providing a deep counterfeit video detection method using a spatial domain local pixel relation of a video frame and a face-background multi-proportion branch, and learning the difference between a real video and a deep counterfeit video in the aspect of spatial domain local pixel relation characteristics. And integration and fusion of detection results are carried out based on branches of different face-background cutting proportions, so that good detection capability of a deep counterfeit video is realized. According to the method, the detection model is trained from the perspective of the local pixel relation, the deep counterfeit mode feature synthesis method is combined, the multi-proportion branch fusion strategy is designed, various types of deep counterfeit videos can be effectively detected, and the method is suitable for generalization detection of the deep counterfeit videos in an actual scene and has actual application value.

Description

Technical Field

[0001] The present invention relates to the field of video forensics, and in particular to a deep fake video detection method based on local pixel relationship and face-background multi-scale branches. Background Art

[0002] With the rapid development of deep learning technology, real pictures and videos can be tampered with and forged by deep neural networks, and the tampered and forged pictures and videos can be indistinguishable from the real thing, and the traces of forgery cannot be seen with the naked eye. Without appropriate detection and identification methods, fake pictures and videos will mislead the public, influence public opinion, cause social division and confrontation, and even more seriously endanger national security. Therefore, it is urgent to develop detection technology for deep forged content, especially deep forged videos that are more harmful.

[0003] Existing deepfake video detection methods can be divided into two categories: detection methods based on manual features and detection methods based on learning. The former is a feature designed by researchers based on the differences between real videos and fake videos, allowing the model to focus on this feature to classify the video, while the latter allows the model to mine features that can be used for classification during training. The literature "Sun Z, Han Y, Hua Z, et al. Improving the efficiency and robustness of deepfakes detection through precise geometric features [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 3609-3618." Based on the fact that deepfake videos usually process faces frame by frame, destroying the temporal continuity, the geometric features are modeled by the motion changes of facial landmarks in the video to detect real videos and fake videos. The paper "Xu Y, Liang J, Jia G, et al. Tall: Thumbnail layout for deepfake video detection [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2023: 22658-22668." proposed a novel data combination method, which arranges four consecutive frames of face images in the video into thumbnails for network training. During training, the network can not only learn the classification features of the spatial domain, but also learn the relevant classification features of the temporal continuity between images. After continuous research and development, the performance of deep fake video detection methods has tended to saturation when tested in the dataset used during training. However, the performance of deep fake video detection methods usually drops sharply when testing for fake data that has not been seen during training. This is because the model has already fitted the specific fake pattern features in the dataset during training, and once it detects other fake data, it loses the detection ability. Therefore, the current research focus is shifted to the performance of the detection method across datasets and tampering methods, that is, the generalization and versatility of the detection method.The paper "Shiohara K, Yamasaki T. Detecting deepfakes with self-blended images [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 18720-18729." uses image processing methods to artificially synthesize deep fake data using real data, thereby reducing the risk of overfitting specific types of fake data in the dataset and enhancing the generalization ability of the detection model.

[0004] The present invention detects fake videos based on the spatial domain local pixel relationship of video frames, which reduces the risk of overfitting the model to the semantic content of the image and the data set, and is conducive to enhancing its generalization. The designed face-background multi-scale branch fusion strategy further improves the detection performance. Taking into account practical application problems, the present invention aims to improve the generalization of deep fake video detection methods and proposes a deep fake video detection method based on local pixel relationship and face-background multi-scale branch. Summary of the invention

[0005] The present invention proposes a deep fake video detection method based on local pixel relationship and face-background multi-scale branch to achieve generalized detection of deep fake videos. The method mainly includes: proposing a detection method based on local pixel relationship in spatial domain, and on this basis, designing a face-background multi-scale branch fusion strategy to further improve the detection performance, and achieve good generalization detection of known type of fake videos and unknown type of fake videos.

[0006] The specific contents are as follows:

[0007] (1) A deep fake video detection method based on the spatial domain local pixel relationship of video frames is proposed: frames containing faces are extracted from real and fake videos, the positions of faces are detected and the face regions are cropped to a specified size, and the saved face images I are o The transformed image I is obtained by downsampling to reduce the size and upsampling to restore the original size. r , the difference between the original image and the transformed image I o -I r That is, the local pixel relationship is extracted, and then the local pixel relationship is input into the classification network to obtain the classification result.

[0008] The overall detection method framework mainly includes two stages: training and testing. Figure 1As shown in the figure, in the training stage, the video is first preprocessed, and video frames containing faces are extracted from real and forged videos. The RetinaFace algorithm is used to detect faces and return the face position. The face area is cropped to a specified size and saved as an image. In order to improve the generalization performance of the detection method, a series of transformations and synthesis are performed on the face images of real videos to obtain additional forged image data. The local pixel relationship features of all image data are extracted and input into the Swin-Transformer network for learning, and the cross entropy loss is used to guide the update of network parameters. The Swin-Transformer network first segments the input image, and the features obtained in the linear embedding layer are input into the sub-module. The self-attention of the window and the shifted window is calculated in the sub-module. Subsequently, the blocks of different depths are merged and sub-modules are passed through. Finally, the features are passed through the fully connected layer to obtain the probability of real and forged. During the testing phase, the input video is also preprocessed to extract frames, detect face positions, crop face areas, and extract local pixel relationships. Different from the single ratio used for cropping face areas during training, the face-background multi-ratio branch fusion strategy designed during testing uses three face-background cropping ratios. Each branch of the cropping ratio follows the process of video preprocessing - extracting local pixel relationship features - network prediction to obtain a classification result. The results of the three branches are fused to obtain the final classification result.

[0009] The steps of extracting local pixel relationships from images are as follows: Figure 2 As shown, for picture I o First, downsample to half the original size, and then upsample to restore the original size. o and restore image I r The difference is the local pixel relationship feature. Two combinations of upsampling and downsampling are used randomly. The first is to use the maximum pooling with a convolution kernel of 2×2 and a step size of 2 to find the maximum pixel value in the 2×2 local pixel block. After the maximum pooling, the size of the image becomes half of the original size, and the pixel value is the maximum value of all local pixel blocks. The pooled image is upsampled by nearest neighbor interpolation to restore the image to its original size, and the four pixel values ​​in the original corresponding 2×2 local pixel block are all converted to the maximum pixel value. The second is to first perform average pooling on the image with a 2×2 convolution kernel and a step size of 2. Each 2×2 local pixel block is replaced with the pixel average value, and the pooled image is restored to its original size using bilinear interpolation. Existing AI-generated image detection methods take the upsampling characteristics of the image generation model as the starting point, considering that there is a certain connection between adjacent pixels in the local pixel block caused by upsampling. However, this method is not aimed at the case of complete generation, but at the case of fake face-swapping, where the background remains unchanged but the face area is manipulated. The local pixel relationship is not caused by upsampling of the generation model, but by the local difference between the unchanged background and the manipulated face area.

[0010] (2) A face-background multi-ratio branch fusion strategy is proposed: the face area is cropped from the video and input into the trained network for prediction. Instead of using the single ratio used in the preprocessing in the training phase, three face-background cropping ratios are used. Each branch of the cropping ratio follows the process of video preprocessing-extracting local pixel relationship features-network prediction to obtain a classification result. The results of the three branches are p i ,i∈{1,2,3} are fused to obtain the final classification result. The fusion method is average, and the final classification result of the video is p=avg(p1,p2,p3). Since the resolution of the video varies, the proportion of the face in the video is also different. The proposed face-background multi-ratio branch is to crop the same video with multiple face-background ratios. The resulting images have the same size after scaling, but the proportions of the face and background in the image are different, such as Figure 3 As shown in the figure. The detection method is based on the local pixel relationship. A slight change in the ratio of the face to the background can have a greater impact at the local pixel level. Selecting multiple ratios can cover a more diverse range of face data types, and the fusion results can further improve the detection performance, which is suitable for detection task scenarios in real life where the resolution of the target video data is unknown and the ratio of the face to the video frame space is unknown.

[0011] When the face detection algorithm successfully detects a face, it will get a face frame. The position of the rectangular face frame in the video frame is where the face appears. The face area can be cropped out according to its coordinates. When cropping, this method does not directly use the face frame coordinates, but first finds the longest side of the length and width of the rectangular frame, keeps the center unchanged, and adjusts the rectangular frame to a square with the longest side length as the side length. The cropping ratio is the magnification ratio relative to the adjusted side length. The cropping ratio is multiplied by the side length, and the coverage of the square face frame is adjusted again. The cropping, scaling and saving are performed as images. The fixed ratio of 1.2 is used for cropping in the training data set, and the three ratios selected in the test are 1.2, 1.1 and 1.0 respectively. The images cropped at each ratio are scaled to a fixed size and then input into the trained network to obtain a classification result. The results obtained by different branches of the three ratios are integrated and fused to obtain the final video classification result.

[0012] Compared with the existing technology, the above scheme has at least the following significant effects:

[0013] 1. The present invention provides a deep fake video detection method based on local pixel relationship, which uses cropped original data and synthesized additional data to extract local pixel relationship features, and guides the classification network to focus on local and generalized features in the learning process. Compared with existing methods, this scheme synthesizes additional fake data and utilizes local pixel relationship features, so the generalization performance of the trained network is improved when detecting across data sets.

[0014] 2. The present invention provides a face-background multi-ratio branch fusion strategy to further improve detection performance. Previous methods use the same preprocessing steps during training and testing, but the cropping ratio used during training may not be suitable for other data sets or data encountered in real life. In comparison, cropping using multiple face-background ratios can cover more diverse data types and is more suitable for actual detection needs. Finally, the integrated fusion of multi-ratio branch detection results also further improves detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the overall framework diagram of deep fake video detection of the present invention;

[0016] Figure 2 A schematic diagram of extracting local pixel relationships according to the present invention;

[0017] Figure 3 A schematic diagram of the face-background multi-scale branch used in the present invention; DETAILED DESCRIPTION

[0018] The present invention is a deep fake video detection method based on local pixel relationship and face-background multi-scale branch. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of application of the present invention.

[0019] The development language is Python, and the deep learning framework is PyTorch. The specific steps are as follows:

[0020] Step 1: Preparation of training dataset. The widely used FaceForensics++ (FF++) is used as the training dataset, which contains 1,000 real videos and 4,000 fake videos generated by four fake methods (DeepFakes, FaceSwap, Face2Face, and NeuralTextures). FF++ has three versions: lossless, lightly compressed, and heavily compressed. The lightly compressed version of the video data is used in this solution. The division of the dataset is based on official recommendations.

[0021] Step 2: Process the dataset video and create an image dataset. Take the video from the divided training set, validation set, and test set, open the video with the OpenCV library, obtain the video frame, use the face detection algorithm RetinaFace to detect the face position in the video frame, set 1.2 as the cropping ratio, crop and scale it to 224×224 and save it in png image format. When cropping the face image in the real video, a series of random transformation operations such as color shift, contrast adjustment, saturation adjustment, sharpening, maximum pooling downsampling and nearest neighbor upsampling are performed on the image to obtain the transformed image. There is a certain difference between the cropped real image and the transformed image. At this time, the face area of ​​one image is replaced with the face area of ​​another image and Gaussian blur is performed on the edge of the replacement area to obtain additional synthetic forged image data. The cropped images of the real video and the four forged videos and the synthesized forged images all use a ratio of 1.2 and a size of 224×224. Each forged video crops 40 images, and the real video crops 40 images and synthesizes an additional 40 forged images.

[0022] Step 3: Extract local pixel relationship features of the image and perform network training. During training, the learning rate is set to 0.0004, and the batch size is set to 16. When loading images from the image dataset in each batch, the images are randomly restored to their original size by maximum pooling downsampling, nearest neighbor interpolation upsampling or average pooling downsampling, and bilinear interpolation upsampling. The original image is then subtracted from the restored image to obtain the local pixel relationship, which is then input into the network for training. The network uses a pre-trained model adapted to the 224×224 size using Swin-Transformer, and the network weight parameters are updated using SGD and SAM optimizers. Since the number of real and fake images is unbalanced, real images are densely sampled so that the number of real and fake images during training is roughly balanced. The network is trained for 40 cycles, and the model is saved once each round to achieve higher performance on the validation set.

[0023] Step 4: Test the algorithm. After the network training is completed, the best performing model on the validation set is used as the test model. During the test, 32 frames of video are input, and the local pixel relationship is extracted from each frame. After merging, they are input into the network to obtain the output. After softmax normalization, the true and false probability of each frame is obtained. The probabilities of the 32 frames are averaged to obtain the video-level classification probability. Based on the 32 frames, the video frames are cropped at multiple face-background ratios. Each branch of the ratio obtains a classification result according to the normal process, and finally the results of each branch are fused to obtain the final classification result. The three ratios are more common choices. Specifically, during the test, the selection of the ratio and the adjustment of the number of ratios are expected to obtain better detection performance.

[0024] In summary, in order to achieve generalized detection of deep fake videos, the present invention designs a deep fake video detection method based on local pixel relationship and face-background multi-scale branch. The method has good generalized detection performance for various types of fake videos, which meets the requirements of detection tasks in real life.

[0025] Those skilled in the art will appreciate that the protection scope of the present invention is not limited to the specific embodiments described. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features. It should be noted that the technical solutions after the changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A deep fake video detection method based on local pixel relationship and face-background multi-scale branch, characterized in that: The spatial domain local pixel relationship features of the video frame are used to detect forged videos. The face area is cropped from the video and saved as an image. Based on the image, the local pixel relationship features are extracted to suppress the semantic content of the image and reduce the risk of overfitting the network to the training data. The face-background multi-proportional branch fusion strategy is combined to crop the face area at multiple proportions according to the actual situation where the resolution of the forged video data and the proportion of the face in the video are different. Specifically, the face area is cropped at multiple proportions, including: Deep fake video detection method based on local pixel relationship, face-background multi-scale branch fusion strategy; preprocess the videos in the dataset to obtain face images, extract local pixel relationships from them and input them into the classification network for training; when testing with the trained model, combine the face-background multi-scale branch fusion strategy to cover more diverse data types; The method for detecting deep fake videos based on local pixel relationships includes video-to-image preprocessing, image extraction of local pixel relationship features, and a Swin-Transformer network. In the training phase, the video-to-image preprocessing includes cropping the face areas in real and fake videos and synthesizing real face images to obtain additional fake image data. During cropping, the face frame obtained by face detection is adjusted to a square, and the face frame is adjusted again with a cropping ratio of 1.2, that is, the magnification ratio of the side length of the square, and then cropped, scaled, and saved as an image. In the training phase, the cropping is uniformly performed with a ratio of 1.

2. Images are loaded from the image data set and input to the network for training. Each image The image is first downsampled to reduce its size, and then upsampled to restore its original size. The difference between the original image and the restored image is the local pixel relationship feature. After the local pixel relationship features of a batch of images are extracted, they are input into the network for training; the classification network adopts the Swin-Transformer pre-training model, and the cross entropy loss is used to guide the update of network parameters during the training process; in the testing phase, three face-background cropping ratios are used in the preprocessing of the video. The branches of each ratio follow the process of video preprocessing-extraction of local pixel relationship features-network prediction to obtain the classification result, and the results of the three branches are averaged to obtain the final classification result of the video.

2. The deep fake video detection method based on local pixel relationship and face-background multi-scale branch according to claim 1 is characterized in that: The extraction of the local pixel relationship is first down-sampled to half the original size, and then up-sampled to restore the original size; two up-sampling and down-sampling combinations are randomly used in training and testing: one is to use a 2×2 convolution kernel with a step size of 2 for maximum pooling to obtain the maximum pixel value in each 2×2 pixel block of the image, and then use the nearest neighbor interpolation up-sampling to restore to the original size; the other is to use a 2×2 convolution kernel with a step size of 2 for average pooling to obtain the average pixel value in each 2×2 pixel block of the image, and then use bilinear interpolation to restore to the original size.

3. The deep fake video detection method based on local pixel relationship and face-background multi-scale branching according to claim 1 is characterized in that: The face-background multi-ratio branch is a strategy adopted when detecting videos after training. Three different ratios are used when cropping the face area: 1.2, 1.1 and 1.0, so that the proportion of the face and the background in the cropped image changes; the cropped image of each ratio is scaled to a fixed size and then input into the network to obtain the prediction result, and finally the three branches are integrated, and the results are averaged to obtain the final prediction result of the video.

Images