A method and apparatus for detecting face forgery based on multi-scale VIT neural networks

Abstract

This invention provides a method and apparatus for face forgery detection based on a multi-scale VIT neural network. The method includes: S1, acquiring the FaceForensics++ dataset and dividing it into training and testing sets, as well as performing data preprocessing and augmentation; S2, constructing a VIT neural network, extracting features using a residual network, obtaining multiple feature images at different scales using a feature pyramid, fusing the feature images at different scales to obtain a fused feature map, and feeding the fused feature map into the VIT neural network to model inconsistencies in face forgery regions; S3, feeding the divided FaceForensics++ dataset into the VIT neural network for training until the VIT neural network converges, obtaining a weight file; S4, using the trained VIT neural network and the weight file to detect test images and outputting classification results and classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of video detection, and more specifically, to a method and apparatus for detecting face forgery based on a multi-scale VIT neural network. Background Technology

[0002] Deepfake is a type of fake video created using deep learning. After repeated adjustments and human intervention, the resulting videos become so sophisticated that they are virtually indistinguishable from human videos. This field is a branch of machine learning and falls under the umbrella of cybersecurity and computer vision.

[0003] Deepfake technology can replace the face of a target video with the corresponding face in the source video, allowing the person in the source video to say things they never said and do things they never did, thus misleading the viewer. This technology has creative and productive applications. For example, it can be used to realistically dub foreign films, to "revive" historical figures for educational purposes, and for virtual try-on while shopping. Many online communities also dedicate themselves to creating Deepfake emojis for entertainment. However, despite its positive applications, Deepfake technology has also been used for unethical and malicious purposes, raising concerns about identity theft, impersonation, and the spread of misinformation on social media. Because of the low barrier to entry for video creation, high fidelity, and strong deceptiveness, this technology can be abused. Since 2017, the number of papers published in this field has increased from 3 to several hundred (2018-2020). To understand where the threats lie and how to mitigate them, we need a clear understanding of the technology's current state, challenges, limitations, capabilities, and development trajectory. Although my country has relevant technologies for identifying illegal and irregular audio and video, research on deepfake video detection technology is still lacking, and there is no mature detection technology available for use. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for detecting face forgery based on a multi-scale VIT neural network.

[0005] The present invention aims to address the common forgery traces in Deepfake videos in existing technologies, and to identify significant inconsistencies between facial areas and the background.

[0006] Compared with the prior art, the technical solution of the present invention and its beneficial effects are as follows:

[0007] The first aspect of this invention discloses a method for detecting face forgery based on a multi-scale VIT neural network, comprising: S1, acquiring the FaceForensics++ dataset, dividing the FaceForensics++ dataset into a training set and a test set, and performing data preprocessing and data augmentation; S2, constructing a VIT neural network, extracting features using a residual network, obtaining multiple feature images of different scales using a feature pyramid, fusing the feature images of different scales to obtain a fused feature map, and feeding the fused feature map into the VIT neural network to model inconsistencies in face forgery regions; S3, feeding the divided FaceForensics++ dataset into the VIT neural network for training until the VIT neural network converges, and obtaining a weight file; S4, using the trained VIT neural network and the weight file to detect test images, and outputting classification results and classification accuracy.

[0008] As a further improvement, step S1 includes: S11, dividing the video dataset into a training set and a test set using a dataset partitioning file; S12, extracting frames from each video in the training set and the test set, saving the data frames every 4 frames; S13, using a tool to perform face detection and cropping on the saved data frames to obtain the final training data and test data.

[0009] As a further improvement, step S2 includes: S21, constructing a VIT neural network and using a residual network to extract features from training samples; S22, using a feature pyramid to add features of the same scale to obtain four feature maps of different scales; S23, fusing the feature maps of different scales by channel stacking to obtain a fused feature map; S24, feeding the fused feature map into the VIT neural network to model the inconsistency problem of face forgery regions and capture contextual information.

[0010] As a further improvement, step S3 includes: S31, processing the FaceForensics++ dataset using random cropping and random horizontal flipping data augmentation methods based on the size of the targets in the FaceForensics++ dataset; S32, optimizing using stochastic gradient descent, with the learning rate decaying twice from the initial value to enable the VIT neural network to achieve better detection results; S33, trying different training hyperparameters on the VIT neural network to augment the input image and train it, stopping training when the loss function converges or reaches the maximum number of iterations to obtain the trained VIT neural network and weight file.

[0011] As a further improvement, step S31 includes: before training the VIT neural network, assuming the FaceForensics++ dataset contains n sample data points, denoted as x1, x2, ..., xn, the mean of the FaceForensics++ dataset is expressed as the sum of the values ​​of all sample data points divided by the number of samples, i.e., mean = (x1 + x2 + ... + xn) / n; calculating the difference between each data point and the mean: (x1 - mean), (x... + xn) / n. 2 - mean),...,(xn - mean); Calculate the square of the difference: (x1 - mean)^2,(x2 - mean)^2,...,(xn - mean)^2; Calculate the mean of the squared differences: [(x1 - mean)^2 + (x2 - mean)^2 +... + (xn - mean)^2] / n; Normalize the data. Standardization is performed by transforming the data into a distribution with a mean of 0 and a standard deviation of 1. That is, the standardized value = (original value - mean) / standard deviation.

[0012] As a further improvement, step S32 includes: using random weights as initial weights to set the learning rate, number of iterations, batch size, etc.; and decaying the learning rate from the initial value at 100 and 150 rounds, so that the VIT neural network can achieve better detection results.

[0013] As a further improvement, step S4 includes: S41, sending the test image corresponding to the video into a multi-scale VIT neural network to detect and obtain the image prediction result; S42, averaging the image prediction result to obtain the final video prediction result; S43, if the result of averaging the image prediction result is greater than 0.5, then the corresponding video is a fake video.

[0014] The second aspect of this invention discloses a face forgery detection device based on a multi-scale VIT neural network, comprising: a segmentation module configured to acquire a FaceForensics++ dataset, and to divide the FaceForensics++ dataset into a training set and a test set, as well as perform data preprocessing and data augmentation; a feature extraction module configured to construct a VIT neural network, extract features using a residual network, obtain multiple feature images of different scales using a feature pyramid, fuse the feature images of different scales to obtain a fused feature map, and feed the fused feature map into the VIT neural network to model inconsistencies in face forgery regions; a training module configured to feed the segmented FaceForensics++ dataset into the VIT neural network for training until the VIT neural network converges and obtains a weight file; and a detection module configured to use the trained VIT neural network and the weight file to detect test images and output classification results and classification accuracy.

[0015] The beneficial effects of this invention are as follows:

[0016] Based on the VIT neural network, this invention extracts features from images at different stages through a residual network and fuses feature maps of different scales to better capture forgery traces at different scales. By combining deep and shallow semantic information, the classification accuracy of the model is enhanced.

[0017] This invention utilizes a self-attention mechanism to perform contextual modeling of forgery traces. By capturing inconsistencies between the foreground and background, it detects forged images, more effectively locating forgery traces and improving classification accuracy. This invention effectively solves the problem of low detection accuracy in current face forgery detection models, significantly improving the overall detection accuracy. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a face forgery detection method based on a multi-scale VIT neural network provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of a face forgery detection device based on a multi-scale VIT neural network provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram comparing the detection results of an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0023] Reference Figure 1 As shown, a face forgery detection method based on a multi-scale VIT neural network includes: S1, acquiring the FaceForensics++ dataset, dividing the FaceForensics++ dataset into training and testing sets, and performing data preprocessing and data augmentation; S2, constructing a VIT neural network, extracting features using a residual network, obtaining multiple feature images of different scales using a feature pyramid, fusing the feature images of different scales to obtain a fused feature map, and feeding the fused feature map into the VIT neural network to model inconsistencies in face forgery regions; S3, feeding the divided FaceForensics++ dataset into the VIT neural network for training until the VIT neural network converges, and obtaining a weight file; S4, using the trained VIT neural network and the weight file to detect test images, and outputting classification results and classification accuracy.

[0024] Step S1 includes: S11, dividing the video dataset into a training set and a test set using a dataset partitioning file; S12, extracting frames from each video in the training set and the test set, saving the data frames every 4 frames; S13, using the RetinaFace tool to perform face detection and cropping on the saved data frames to obtain the final training data and test data.

[0025] Step S2 includes: S21, constructing a VIT neural network and using a residual network to extract features from training samples; S22, using a feature pyramid to add features of the same scale to obtain four feature maps of different scales; S23, fusing the feature maps of different scales by channel stacking to obtain a fused feature map; S24, feeding the fused feature map into the VIT neural network to model the inconsistency problem of face forgery regions and capture contextual information.

[0026] In step S21, the features from the first layer to the third layer of the residual network are extracted, allowing the network to learn important features.

[0027] In step S24, the fused feature map is fed into the VIT neural network. The VIT neural network captures forgery traces in the fused feature map, models the inconsistency problem of face forgery regions (foreground and background), and captures contextual information.

[0028] Step S3 includes: S31, processing the FaceForensics++ dataset using random cropping and random horizontal flipping data augmentation methods based on the size of the targets in the FaceForensics++ dataset; S32, optimizing the dataset using stochastic gradient descent, with the learning rate decaying twice from the initial value to enable the VIT neural network to achieve better detection results; S33, trying different training hyperparameters on the VIT neural network to augment the input image and train it, stopping training when the loss function converges or reaches the maximum number of iterations to obtain the trained VIT neural network and weight file.

[0029] In step S31, the original image is randomly cropped with a cropping padding size of 8.

[0030] Step S31 includes:

[0031] Before training the VIT neural network, assume that the FaceForensics++ dataset contains n data points, denoted as x1, x2, ..., xn. The mean of the FaceForensics++ dataset is expressed as the sum of the values ​​of all data points divided by the number of samples, i.e.,

[0032] Mean = (x1 + x2 + ... + xn) / n;

[0033] Calculate the difference between each data point and the mean: (x1 - mean), (x2 - mean), ..., (xn - mean);

[0034] Calculate the square of the difference: (x1 - mean)^2, (x2 - mean)^2, ..., (xn - mean)^2;

[0035] Calculate the mean of the squared differences: [(x1 - mean)^2 + (x2 - mean)^2 + ... + (xn - mean)^2] / n;

[0036] Normalization is performed on the data by transforming it into a distribution with a mean of 0 and a standard deviation of 1. That is, the standardized value = (original value - mean) / standard deviation.

[0037] Step S32 includes: using random weights as initial weights to set the learning rate, number of iterations, batch size, etc.; and at rounds 100 and 150, decaying the learning rate from the initial value so that the VIT neural network can achieve better detection results.

[0038] Step S4 includes: S41, sending the test image corresponding to the video into a multi-scale VIT neural network to detect and obtain the image prediction result; S42, averaging the image prediction result to obtain the final video prediction result; S43, if the average value of the image prediction result is greater than 0.5, then the corresponding video is a fake video.

[0039] Reference Figure 2 As shown, a face forgery detection device based on a multi-scale VIT neural network includes: a segmentation module configured to acquire the FaceForensics++ dataset, divide the FaceForensics++ dataset into training and testing sets, and perform data preprocessing and data augmentation; a feature extraction module configured to construct a VIT neural network, extract features using a residual network, obtain multiple feature images of different scales using a feature pyramid, fuse the feature images of different scales to obtain a fused feature map, and feed the fused feature map into the VIT neural network to model inconsistencies in face forgery regions; a training module configured to feed the segmented FaceForensics++ dataset into the VIT neural network for training until the VIT neural network converges and obtains a weight file; and a detection module configured to use the trained VIT neural network and the weight file to detect test images and output classification results and classification accuracy.

[0040] Reference Figure 3As shown, training and testing were performed on a single 2080ti graphics card, using ResNet50 as the residual network. During training, we used Adam as the optimizer with a learning rate of 3e-5 and a weight decay of 3e-4. In the VIT neural network, we used an encoding network with an image size of 16 and a base size. The number of real and fake images in 30 frames of a video was counted, and the results were averaged. Assuming the prediction result for each sample is y1, y2, ..., yn, the final result is acc = (y1 + y2 + ... + yn) / 30. If the acc is greater than 0.5, the video is predicted as fake; otherwise, it is considered real. The above embodiments are only used to explain the technical solution of the present invention and not to limit it. Those skilled in the art should understand that any modifications and equivalent substitutions that do not depart from the spirit and scope of the present invention should fall within the protection scope of the claims of the present invention.

Claims

1. A face forgery detection method based on a multi-scale VIT neural network, characterized in that, include: S1, Obtain the FaceForensics++ dataset, and divide the FaceForensics++ dataset into training set and test set, as well as perform data preprocessing and data augmentation; S2, construct a VIT neural network, extract features using a residual network, obtain multiple feature images of different scales using a feature pyramid, fuse the feature images of different scales to obtain a fused feature map, and feed the fused feature map into the VIT neural network to model inconsistencies in the face forgery region. S3, the partitioned FaceForensics++ dataset is fed into the VIT neural network for training until the VIT neural network converges and a weight file is obtained; S4 uses the trained VIT neural network and weight file to detect test images and outputs classification results and classification accuracy. Step S2 includes: S21, Construct a VIT neural network and use the residual network to extract features from the training samples; S22, use the feature pyramid to add features of the same scale to obtain four feature maps of different scales; S23, feature maps of different scales are fused by channel overlay to obtain a fused feature map; S24, The fused feature map is fed into the VIT neural network to model the inconsistency problem of face forgery regions and capture contextual information.

2. The face forgery detection method based on a multi-scale VIT neural network according to claim 1, characterized in that, Step S1 includes: S11, use the dataset partitioning file to divide the video dataset into training set and test set; S12, extract frames from each video in the training set and the test set, and save the data frame every 4 frames. S13, use tools to perform face detection and cropping on the saved data frames to obtain the final training and testing data.

3. The face forgery detection method based on a multi-scale VIT neural network according to claim 1, characterized in that, Step S3 includes: S31, based on the size of the targets in the FaceForensics++ dataset, the FaceForensics++ dataset is processed using data augmentation methods of random cropping and random horizontal flipping; S32 uses stochastic gradient descent for optimization, with the learning rate decaying twice from the initial value, enabling the VIT neural network to achieve better detection results. S33. Try different training hyperparameters on the VIT neural network, augment the input image and train it. When the loss function converges or the maximum number of iterations is reached, stop training to obtain the trained VIT neural network and weight file.

4. The face forgery detection method based on a multi-scale VIT neural network according to claim 3, characterized in that, Step S31 includes: Before training the VIT neural network, assume that the FaceForensics++ dataset contains n data points, denoted as x1, x2, ..., xn. The mean of the FaceForensics++ dataset is expressed as the sum of the values ​​of all data points divided by the number of samples, i.e., Mean = (x1 + x2 + ... + xn) / n; Calculate the difference between each data point and the mean: (x1 - mean), (x2 - mean), ..., (xn - mean); Calculate the square of the difference: (x1 - mean)^2, (x2 - mean)^2, ..., (xn - mean)^2; Calculate the mean of the squared differences: [(x1 - mean)^2 + (x2 - mean)^2 + ... + (xn - mean)^2] / n; Normalization is performed on the data by transforming it into a distribution with a mean of 0 and a standard deviation of 1. That is, the standardized value = (original value - mean) / standard deviation.

5. The face forgery detection method based on a multi-scale VIT neural network according to claim 3, characterized in that, Step S32 includes: Random weights are used as initial weights to set the learning rate, number of iterations, batch size, etc.; and the learning rate is decayed from the initial value at 100 and 150 rounds to enable the VIT neural network to achieve better detection results.

6. The face forgery detection method based on a multi-scale VIT neural network according to claim 1, characterized in that, Step S4 includes: S41, The test image corresponding to the video is fed into the multi-scale VIT neural network to detect and obtain the image prediction result; S42, the image prediction results are averaged to obtain the final video prediction result; S43, if the average value of the image prediction results is greater than 0.5, then the corresponding video is a fake video.

7. A face forgery detection device based on a multi-scale VIT neural network, characterized in that, For implementing the method of any one of claims 1-6, comprising: The segmentation module is configured to acquire the FaceForensics++ dataset, and to divide the FaceForensics++ dataset into training and testing sets, as well as perform data preprocessing and data augmentation. The feature extraction module is configured to construct a VIT neural network, extract features using a residual network, obtain multiple feature images of different scales using a feature pyramid, fuse the feature images of different scales to obtain a fused feature map, and feed the fused feature map into the VIT neural network to perform inconsistency modeling on the face forgery region. Specifically, this includes: constructing a VIT neural network and using a residual network to extract features from training samples; By using the feature pyramid, features of the same scale are added together to obtain four feature maps of different scales; Feature maps of different scales are fused by channel overlay to obtain a fused feature map. The fused feature map is fed into the VIT neural network to model the inconsistency problem of face forgery regions and capture contextual information; The training module is configured to feed the partitioned FaceForensics++ dataset into the VIT neural network for training until the VIT neural network converges and a weight file is obtained; the detection module is configured to use the trained VIT neural network and the weight file to detect test images and output classification results and classification accuracy.

Images