Face forgery detection method based on separated three-dimensional convolutional neural network

Abstract

This invention is a face forgery detection method based on a separable 3D convolutional neural network. First, the video is read in and preprocessed. The face portion of the video image is obtained using the MTCNN library, and a FAN network is used to calculate the key points on the face. Then, the face image is divided into regions and masked. The masked image is then subjected to SRM feature extraction. Finally, the obtained SRM features are input into an improved S3D network for model training. This invention fully utilizes local and global facial features and optimizes the objective function, improving the accuracy of video detection while saving computational overhead. Furthermore, the face feature processing method adopted in this invention has good detection performance for faces from different viewpoints and directions, exhibiting strong generalization ability. The masking process for face image partitioning also explores aspects of neural network interpretability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a research method for detecting fake face videos based on a discrete three-dimensional convolutional neural network. Background Technology

[0002] Thanks to innovative breakthroughs in deep learning algorithms, computer vision and image processing technologies have ushered in a new wave of development, making it increasingly easier to generate highly realistic facial modification images or videos. The next generation of generative deep neural networks (GDNNs) can automatically generate highly realistic images or videos from massive amounts of training data. In particular, the emergence of Deepfake has significantly lowered the barrier to entry for facial modification technology. Deepfake uses generative adversarial networks (GANs) to replace the face in the original video with another person's face in a near-perfect way.

[0003] Deepfake technology, a double-edged sword, offers astonishing entertainment, but its misuse has also led to a surge in fake videos and news. The rapid rise of facial modification apps has raised concerns about the spread of forged images and videos. Because altered images or videos may pose serious risks to social stability and national security, an increasing number of researchers are dedicating themselves to the field of facial forgery detection.

[0004] For current image and video forgery detection using deep learning algorithms, the involvement of deep learning can be categorized into three levels. The lowest level is direct transfer, directly applying various commonly used deep learning network structures to image forgery detection. For example, directly training four standard CNN models: VGG16, ResNet50, ResNet101, and ResNet152, effectively distinguishing Deepfake-generated forged videos from real videos. Since image forgery detection and deep learning problems target different objects, fine-tuning the network input is performed to achieve specificity; this is the second level. For example, a novel CNN-based general forgery detection technique can automatically learn how to detect different image forgeries without relying on pre-selected features or any preprocessing. This convolutional network structure suppresses the image's original content information and focuses on the feature information generated after image modification, achieving good results. The third level involves appropriate modification of the network structure, that is, proposing network structures suitable for the specific problems of digital forensics. For example, models that combine CNN and LSTM have an advantage over those that only use CNN to learn image features for prediction because they introduce the learning of time series data.

[0005] Although there are many technologies for detecting face forgery, the following problems still exist: (1) Some detection technologies use traces of deep generative networks for detection, but when faced with videos with very small forgery traces through deep network forgery techniques such as FaceSwap and DeepFake, the effectiveness of these technologies decreases. (2) Methods for detecting face regions have been proposed, but on the one hand, existing methods have the problem of not sharing global and local information, and on the other hand, existing methods use horizontal, vertical, and grid partitioning methods for the face. However, in complex and diverse real-world scenarios, the face in the camera is not always facing forward, so such partitioning methods are not widely applicable. (3) The video deformation, encoding, and compression brought about by the trend of multi-platform video distribution and internet streaming have brought new challenges to existing face forgery detection technologies. Summary of the Invention

[0006] This invention provides a method for detecting fake face videos based on a discrete three-dimensional convolutional neural network, in order to solve the problem that existing methods cannot accurately detect fake face images and thus have an impact.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for detecting fake faces in videos based on a separable 3D convolutional neural network includes the following steps:

[0009] S1. Obtain video frames from the input video data;

[0010] S2. Extract face slices from the acquired video frames;

[0011] S3. Extract facial key points from the face slices, partition them, and then perform masking;

[0012] S4. Establish a face spoofing detection model;

[0013] S5. Model training and testing.

[0014] Furthermore, the specific steps of step S1 are as follows:

[0015] S11. The original dataset contains a video duration of 10 seconds, a resolution of 1920*1080 or 1080*1920, and a frame rate of 30FPS. FPS represents the number of frames transmitted per second. Use the VideoCapture class in OpenCV to process the video and obtain the video frame count CAP_PROP_FRAME_COUNT.

[0016] S12. Based on the obtained video frame count CAP_PROP_FRAME_COUNT, capture video frames using the grab and retrieve methods;

[0017] S13. The video frames obtained in S11 are randomly downsampled to half the resolution of the original frame image, i.e., 960*540 or 540*960. After sampling, a mapping relationship between the frame image and the original video is established.

[0018] Furthermore, the specific steps of step S2 are as follows:

[0019] S21. Use the Multi-task CNN (MTCNN) network in FaceNet for face detection. Determine the input size of the face detector based on the resolution. Process the face from coarse to fine using three cascaded CNNs: P-Net, R-Net, and O-Net. P-Net generates candidate boxes, R-Net corrects the candidate boxes, and O-Net locates and outputs the final face.

[0020] S22. The face bounding box with image size w×h obtained by MTCNN is not necessarily a square rectangle. In order to unify the face image processing operations in subsequent models, the face bounding box is expanded to the size of max(w,h)×max(w,h) and saved in bbox in the form of a .json file, where: w represents the image length and h represents the image width;

[0021] S23. Using the image cropping information in the bounding box saved in S22, crop the transformed frame images from the original video based on the obtained face bounding boxes. The face data contained in each frame image is represented by the face tag. j This indicates that j represents the face ID contained in the current frame, and j takes a value from 0 to N. face N face This indicates the number of faces in the current frame image. The frame information is then linked with the tag information of the original video to build the input data for the model.

[0022] Furthermore, the specific steps of step S3 are as follows:

[0023] S31. Input the face data constructed in S23 into the Face Alignment Network (FAN) to perform multi-scale feature extraction;

[0024] S32. After obtaining 68 facial key points through FAN, the image is first divided into facial key areas and non-facial areas using these coordinates (facial key areas mainly include areas such as facial features that can accurately identify different people, while the rest are non-facial areas). In addition, some empty space is reserved when dividing the facial landmark areas so that the non-facial areas become square.

[0025] S33. After obtaining the four key facial regions and non-facial regions through the above steps, randomly select n regions from the non-facial region of the image for masking operations. n represents the number of regions to be masked, that is, these n regions are processed with Gaussian noise. The same masking process is applied to face slices obtained from the same video, and face slices from different videos are processed independently, thereby obtaining the input data for the face forgery model established in S4.

[0026] Furthermore, the specific steps of step S4 are as follows:

[0027] S41. The input image is first processed through an SRM (Spatial Rich Model) layer. Several high-pass filters (HPF) are used to operate on the image to obtain its SRM features. The obtained features are then fed into subsequent modules for training.

[0028] S42. Establish an improved network model M based on a Separate 3D Convolutional Neural Network (S3D) by decomposing the standard 3D convolution into two consecutive convolutional layers: a 2D convolutional layer for learning spatial features and a 1D convolutional layer on the time axis.

[0029] The loss function for S3D optimization is calculated as follows:

[0030] l i =-w i [y·logσ(x i )+(1-y i )·log(1-σ(x i )),

[0031] in: The total loss is represented by x, the predicted value is represented by y, and l1 and l2 represent the actual values. N l i These represent the losses at position 1, position N, and position i, respectively, where N represents the batch size, and w i Let x represent the weight at position i. i and y iThen, σ represents the predicted value and the true value at point i, σ is the Sigmoid activation function, and T represents the transpose operation. The binary cross-entropy loss (BCE Loss) is combined with the Sigmoid function, and the model parameters are corrected by comparing the label and the prediction results.

[0032] Furthermore, the specific steps of step S5 are as follows:

[0033] S51. When loading image data into the model, considering the imbalance of input data labels, data compensation is first adopted to adjust the label distribution of the input image by adjusting the extraction frame interval;

[0034] S52. The improved model M is trained using the Adaptive Moment Estimation (AdaM) optimizer with an initial learning rate (lr) of 0.001. If the model loss does not improve significantly within 5 epochs, the learning rate is set to gamma times the original value, where gamma represents the learning rate scaling factor and has a value of 0.1. In addition, a weight decay strategy is used during the training of model M to reduce the adverse effects of model overfitting.

[0035] S53. Input the video into the trained model to perform face forgery detection, so as to obtain the face forgery video detection results.

[0036] Compared with existing technologies, the present invention has the following beneficial effects:

[0037] This invention discloses a method for detecting forged faces in videos based on a separable 3D convolutional neural network. First, the input video is processed to obtain video frames. Then, face image slices are extracted from the video frames using the MTCNN face recognition network. These face images are loaded into a Facial Animation Array (FAN) to obtain facial key points. The images are then partitioned and masked based on these key points. Finally, the images are input into a face forgery detection model based on a separable 3D convolutional neural network to obtain prediction results. Through this method, the forgery of faces in the video is determined. This invention fully utilizes local and global facial features and optimizes the objective function, improving the accuracy of video detection while saving computational overhead. Furthermore, the facial feature processing method adopted in this invention has good detection performance for faces from different viewpoints and directions, exhibiting strong generalization ability. The masking process for face image partitioning also explores aspects of neural network interpretability. Attached Figure Description

[0038] Figure 1 This is a flowchart of the present invention.

[0039] Figure 2 This is a diagram of the FAN facial landmark detection network model.

[0040] Figure 3 A diagram showing the key facial regions designed for this invention.

[0041] Figure 4 This is a model diagram of a face forgery detection model M based on a separable 3D convolutional neural network. Detailed Implementation

[0042] The present invention will be further described below with reference to embodiments.

[0043] like Figure 1 As shown, a method for detecting fake faces in videos based on three-dimensional convolutional neural networks includes the following steps:

[0044] S1. Obtain video frames from the input video data;

[0045] S11. The original dataset contains videos with a duration of 10 seconds, a resolution of 1920*1080 or 1080*1920, and a frame rate of 30 FPS. FPS represents the number of video frames per second. Use the VideoCapture class in OpenCV to process the video and obtain the video frame count CAP_PROP_FRAME_COUNT.

[0046] S12. Based on the obtained video frame count CAP_PROP_FRAME_COUNT, capture video frames using the grab and retrieve methods;

[0047] S13. The video frames obtained in S11 are randomly downsampled to half the resolution of the original frame image, i.e., 960*540 or 540*960. After sampling, a mapping relationship between the frame image and the original video is established.

[0048] S2. Use the Multi-task CNN (MTCNN) face extraction network to extract face slices from the acquired video frames;

[0049] S21. Considering kernel computation time, the MTCNN network in FaceNet was selected for face detection after comparison. The input size of the face detector was determined according to the resolution. The face was processed from coarse to fine by three cascaded CNNs: P-Net, R-Net, and O-Net. Specifically, P-Net generates candidate boxes, R-Net corrects the candidate boxes, and O-Net locates and outputs the final face. Bounding box regression and non-maximum suppression (NMS) are used in P-Net and R-Net to adjust and merge candidate boxes, with thresholds set to [0.85, 0.95, 0.95], respectively.

[0050] S22. The face bounding box with size w×h obtained by MTCNN is not necessarily a square rectangle. In order to unify the face image processing operations in subsequent models, the face bounding box is expanded to the size of max(w,h)×max(w,h) and saved in bbox in the form of a .json file, where: w represents the image length and h represents the image width;

[0051] S23. Using the image cropping information in the bounding box saved in S22, crop the transformed frame images from the original video based on the obtained face bounding boxes. The face data contained in each frame image is represented by the face tag. j This indicates that j represents the face ID contained in the current frame, and j takes a value from 0 to N. face N face This indicates the number of faces in the current frame image. The frame information is then linked with the tag information of the original video to build the input data for the model.

[0052] S3. Use a FAN network to obtain facial key points from face slices, and then perform masking after partitioning;

[0053] S31. Input the face data constructed in S23 into the Face Alignment Network (FAN), such as... Figure 2 As shown, FAN uses four stacked HG (Hour-Glass, HG) network blocks, replacing the residual blocks with the hierarchical parallel multi-scale blocks on the right. The network output is a set of heatmaps, which give the probability of a face appearing at each pixel. Figure 2As shown on the right, in the block, each convolutional layer is linked to the output through a direct path. Therefore, at any given time, the shortest path is equal to 1 for all layers within the block. Inside the block are convolutional layers with 3×3 filters. In this block, the number of channels in the first layer is [256→128, 3×3], the number of channels in the second layer is [128→64, 3×3], and the number of channels in the third layer is [64×64, 3×3].

[0054] Specifically, the HG blocks used in FAN process features to a very low resolution through convolutional and max-pooling layers. Each time the max-pooling layer is passed, the network generates branches and performs more convolutional operations on the pooled resolution. After reaching the lowest resolution, the network begins to alternately perform upsampling and cross-scale feature fusion. In order to fuse two cross-scale features, the lower resolution is upsampled to the nearest neighbor, and then the two sets of features are added element by element. The topology of the hourglass is symmetrical, so for each layer that exists downwards, there is a corresponding layer that exists upwards.

[0055] S32. After obtaining 68 facial landmarks through FAN, as follows: Figure 3 As shown, the image is first divided into key facial regions and non-facial regions using these coordinates (the key facial regions mainly include areas that can accurately identify different people, such as facial features, while the rest are non-facial regions). In addition, some empty spaces are reserved when dividing the key facial regions, such as expanding them by 10% on each side. These regions are logical, so overlap between regions is allowed. By expanding some key regions, the non-facial regions become square.

[0056] S33. After obtaining the four key facial regions and non-facial regions through the above steps, randomly select n regions from the non-facial region of the image for masking operations. n represents the number of regions to be masked, that is, these n regions are processed with Gaussian noise. The same masking process is applied to the face slices obtained from the same video, and the face slices from different videos are processed independently to obtain the input data of the face forgery model established in S4.

[0057] S4. Establish a face forgery detection model based on the separable 3D convolutional neural network S3D;

[0058] S41. The input image is first processed through an SRM (Spatial Rich Model) layer. Several high-pass filters (HPF) are used to operate on the image to obtain its SRM features. The obtained features are then fed into subsequent modules for training.

[0059] S42. Establish an improved network model M based on a separable 3D CNN (S3D), such as... Figure 4 As shown, the S3D model decomposes the standard 3D convolution into two consecutive convolutional layers: a 2D convolutional layer for learning spatial features and a 1D convolutional layer on the time axis, as follows. Figure 4 As shown in the Sep-Conv convolution module below, in other words, for a convolution kernel of size [k;k;k] in the original convolution module, where k is the kernel size (default value is 3), a [1;k;k] filter and a [k;1;1] filter can be used to implement 3D convolution calculation in I3D. Although 3D convolution is very effective in improving model accuracy, its computational cost is very high. However, by using the decomposed convolution module Sep-Cov, based on the complementary nature of the spatial and temporal domains, we can achieve the same or even better results as 3D convolution while ensuring that the computational cost is within a certain range.

[0060] Specifically, the network model M consists of the following modules connected together: a convolutional module Sep-Conv, pointwise convolutional layers, max pooling layers, average pooling layers, and a separable Inception module Sep-Inc, such as... Figure 4 As shown in the Sep-Inc module below, the Sep-Inc module differs from the 3D-Inception module (where a 3D-Inception block requires two 3×3×3 convolutions across its four branches, and only some features contain temporal information). The Sep-Inc module separates the temporal features, allowing temporal information to be added to all four branches, thus giving the model stronger expressive power. Except for the final fully-connected (FC) layer used to output the classification prediction result, all convolutional layers in the pointwise convolutional layers, Sep-Conv module, and Sep-Inc module are connected to a batch normalization (BN) layer and a ReLU activation function layer, ensuring higher learning efficiency and stronger generalization ability. The loss function for S3D optimization is calculated as follows:

[0061] l i =-w i [y·logσ(x i )+(1-y i )·log(1-σ(x i )),

[0062] in: The total loss is represented by x, the predicted value is represented by y, and l1 and l2 represent the actual values. N l iThese represent the losses at position 1, position N, and position i, respectively, where N represents the batch size, and w i Let x represent the weight at position i. i and y i Then, σ represents the predicted value and the true value at point i, σ is the Sigmoid activation function, and T represents the transpose operation. The binary cross-entropy loss (BCE Loss) is combined with the Sigmoid function, and the model parameters are corrected by comparing the label and the prediction results.

[0063] S5. Use the established model M for training and testing.

[0064] S51. When loading image data into the model, considering the imbalance of input data labels, data compensation is first adopted to adjust the label distribution of the input image by adjusting the extraction frame interval;

[0065] S52. The improved model M is trained using the Adaptive Moment Estimation (AdaM) optimizer with an initial learning rate (lr) of 0.001. If the model loss does not improve significantly within 5 epochs, the learning rate is set to gamma times the original value, where gamma represents the learning rate scaling factor and has a value of 0.1. In addition, a weight decay strategy is used during the training of model M to reduce the adverse effects of model overfitting.

[0066] S53. Input the video into the trained model to perform face forgery detection, so as to obtain the face forgery video detection results.

[0067] This invention presents a method for detecting forged faces in videos based on a separable 3D convolutional neural network. First, the input video is processed to obtain video frames. Then, face image slices are extracted from the video frames using the MTCNN face recognition network. These face images are loaded into a Facial Animation Array (FAN) to obtain facial key points. The images are then partitioned and masked based on these key points. Finally, the images are input into a face forgery detection model M to obtain prediction results. Through this process, the determination of whether a face in the video has been forged is completed. This invention fully utilizes local and global facial features and optimizes the objective function, improving the accuracy of video detection while saving computational overhead. Furthermore, the facial feature processing method adopted in this invention has good detection performance for faces from different viewpoints and directions, exhibiting strong generalization ability. The masking process for face image partitioning also explores aspects of neural network interpretability.

[0068] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting fake faces in videos based on a separable 3D convolutional neural network, characterized in that, Includes the following steps: S1. Obtain video frames from the input video data; S2. Extract face slices from the acquired video frames; S3. Extract facial key points from the face slices, partition them, and then perform masking; S4. Establish a face spoofing detection model; S5. Model training and testing; The specific steps of step S3 are as follows: S31. Input the face data constructed in S23 into the Facial Anchor Network (FAN) for multi-scale feature extraction; S32. After obtaining 68 facial key points through FAN, the image is first divided into facial key areas and non-facial areas using these coordinates. Facial key areas include areas such as facial features that can accurately identify different people, while the rest are non-facial areas. In addition, some empty space is reserved when dividing the facial landmark areas so that the non-facial areas become square. S33. After obtaining the four key facial regions and non-facial regions through the above steps, randomly select n regions from the non-facial region of the image for masking operations. n represents the number of regions to be masked, that is, these n regions are processed with Gaussian noise. The same masking process is applied to the face slices obtained from the same video, and the face slices from different videos are processed independently to obtain the input data of the face forgery model established in S4. The specific steps of step S4 are as follows: S41. The input image is first processed through an SRM layer. Several high-pass filters (HPF) are used to operate on the image to obtain its SRM features. The obtained features are then fed into subsequent modules for training. S42. Establish an improved network model M based on a separable 3D convolutional neural network S3D by decomposing the standard 3D convolution into two consecutive convolutional layers: a 2D convolutional layer for learning spatial features and a 1D convolutional layer on the time axis.

2. The method for detecting fake faces in videos based on a discrete three-dimensional convolutional neural network according to claim 1, characterized in that, The specific steps of step S1 are as follows: S11. The original dataset contains a video duration of 10 seconds, a resolution of 1920*1080 or 1080*1920, and a frame rate of 30FPS. FPS represents the number of frames transmitted per second. Use the VideoCapture class in OpenCV to process the video and obtain the video frame count CAP_PROP_FRAME_COUNT. S12. Based on the obtained video frame count CAP_PROP_FRAME_COUNT, capture video frames using the grab and retrieve methods; S13. The video frames obtained in S11 are randomly downsampled to half the resolution of the original frame image, i.e., 960*540 or 540*960. After sampling, a mapping relationship between the frame image and the original video is established.

3. The method for detecting fake faces in videos based on a discrete three-dimensional convolutional neural network according to claim 1, characterized in that, The specific steps of step S2 are as follows: S21. Use the MTCNN network in FaceNet for face detection. Determine the input size of the face detector based on the resolution. Process the face from coarse to fine using three cascaded CNNs: P-Net, R-Net, and O-Net. Specifically, P-Net generates candidate boxes, R-Net corrects the candidate boxes, and O-Net locates and outputs the final face. S22. The face bounding box with image size w×h obtained by MTCNN is not necessarily a square rectangle. In order to unify the face image processing operations in subsequent models, the face bounding box is expanded to the size of max(w,h)×max(w,h) and saved in bbox in the form of a .json file, where: w represents the image length and h represents the image width. S23. Using the image cropping information in the bounding box saved in S22, crop the transformed frame images from the original video based on the obtained face bounding boxes. The face data contained in each frame image is represented by the face tag. j This indicates that j represents the face ID contained in the current frame, and j takes a value from 0 to N. face N face This indicates the number of faces in the current frame image. The frame information is then linked with the tag information of the original video to build the input data for the model.

4. The method for detecting fake faces in videos based on a discrete three-dimensional convolutional neural network according to claim 1, characterized in that, The loss function for S3D optimization is calculated as follows: in: The total loss is represented by x, the predicted value is represented by y, and l1 and l2 represent the actual values. N l i These represent the losses at position 1, position N, and position i, respectively, where N represents the batch size, and w i Let x represent the weight at position i. i and y i Then, σ represents the predicted value and the true value at point i, σ is the Sigmoid activation function, and T represents the transpose operation. The binary cross-entropy loss (BCE Loss) is combined with the Sigmoid function, and the model parameters are corrected by comparing the labels with the prediction results.

5. The method for detecting fake faces in videos based on a discrete three-dimensional convolutional neural network according to claim 1, characterized in that, The specific steps of step S5 are as follows: S51. When loading image data into the model, considering the imbalance of input data labels, data compensation is first adopted to adjust the label distribution of the input image by adjusting the extraction frame interval; S52. The improved model M is trained using the Adaptive Moment Estimation (AdaM) optimizer with an initial learning rate (lr) of 0.

001. If the model loss does not improve significantly within 5 epochs, the learning rate is set to gamma times the original value, where gamma represents the learning rate scaling factor and has a value of 0.

1. In addition, a weight decay strategy is used during the training of model M to reduce the adverse effects of model overfitting. S53. Input the video into the trained model to perform face forgery detection, so as to obtain the face forgery video detection results.

Images