Face forgery detection method based on dual-flow network of frequency domain features

Abstract

The application discloses a face forgery detection method based on a dual-flow network of frequency domain features. The application proposes a face forgery detection method based on a dual-flow network in view of the problem that the existing face forgery method based on frequency domain features has limitations in retaining local texture features and time domain features. The three-dimensional Fourier transform is used to extract time domain features and face region segmentation is used to extract local texture features, and a corresponding loss function is designed for these features. The experimental results prove that the application can significantly improve the accuracy of true and false video classification by retaining local texture features and time domain features through the dual-flow network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of face forgery detection and computer vision, and particularly relates to a face forgery detection method based on frequency domain features. BACKGROUND

[0002] With the development of deep learning-based generative network technology, face manipulation technologies such as Deepfake and Face2Face can generate false videos that cannot be distinguished by the human eye. Face forgery detection algorithms are proposed for these fake videos.

[0003] In recent years, in addition to using convolutional neural networks (CNN) and recurrent neural networks (RNN) to mine deep-level forgery features, more and more scholars have tried to use frequency domain information to achieve ideal results while using fewer computing resources. Durall et al. first introduced the discrete Fourier transform into face forgery detection, and they found that the average amplitude of each frequency band can reveal abnormal information hidden in the manipulated face. They only used classic frequency domain analysis and a basic classifier (such as a support vector machine) to achieve an accuracy of 85% on the FF++ dataset. Inspired by them, several methods have been proposed to combine frequency domain clues with existing neural networks for detection.

[0004] The closest prior art and its analysis:

[0005] Among the closest prior art, Zhou et al. (Zhou P, Han X, I Morariu V, S Davis L. Two-stream neural networks for tampered face detection [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017: 1831-1839.) proposed a two-stream network, training GoogLeNet for one branch and a ternary network for the other. The MesoNet algorithm (Afchar D, Nozick V, Yamagishi J, Echizen I. MesoNet: a Compact Facial Video Forgery Detection Network [C] / / IEEE International Workshop on Information Forensics and Security, 2018: 1-7.) designed a shallow neural network composed of two initial modules and two classic convolutional layers. Li et al. (Li Y, Lyu S. Exposing deepfake videos by detecting face warping artifacts [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2019.) used CNN to detect artifacts that appear when the original face is transformed into the target face. The heartbeat rhythm algorithm (Qi H, Guo Q, Juefei-Xu F, et al. Deeprhythm: Exposing deepfakes with attentional visual heartbeat rhythms [C] / / 28th ACM International Conference on Multimedia on Proceedings, 2020: 4318-4327.) detects the interrupted or even broken heartbeat rhythm in fake videos by monitoring the slight changes in skin color that occur periodically when blood flows through the face.Face X-ray algorithm (Li L, Bao J, Zhang T, et al. Face x-ray for more general face forgery detection [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020: 5001-5010.) innovatively adds self-generated data to train the network to locate the mixed boundary, and shows the mixed boundary for the fake image and the missing mixed boundary for the real image. GANs fingerprint analysis (Liu H, Li X, W Zhou, et al. Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2021: 772-781.) combines the spatial image and the phase spectrum in DFT to capture the up-sampling artifacts of face forgery. SUMMARY

[0006] The purpose of the present application is to provide a face forgery detection method based on a dual-flow network of frequency domain features.

[0007] Technical solution points:

[0008] I. About frequency domain features:

[0009] For video frames, the face region recognition is performed to intercept the global region, and the face key point recognition is performed to intercept the local region of the five organs. The obtained pictures are respectively retained by two-dimensional discrete Fourier transform to retain the spatial domain features, and by three-dimensional discrete Fourier transform of the front and rear frames to retain the time domain features. The features obtained from the global region will enter the global branch, and the features obtained from the local region will enter the local branch.

[0010] II. About dual-flow loss function

[0011] Local loss and global loss are designed in the loss function. The local loss function can enhance the local texture features, and the global loss function can retain the global forgery features.

[0012] The present application aims at the problem that the existing face forgery method based on frequency domain features has limitations in retaining local texture features and time domain features, and proposes a face forgery detection method based on a double-flow network. Three-dimensional discrete Fourier transform is used to extract time domain features, and lateral link branches are used to fuse time domain features and spatial domain features. Face region segmentation is used to extract local texture features, and the local features and global features are put into a double-flow network for fusion, and a corresponding loss function is designed for these features. Experimental results show that the present application can significantly improve the accuracy of true and false video classification by retaining local texture features and time domain features through a double-flow network.

[0013] The technical scheme to be protected by the present application is characterized by:

[0014] A face forgery detection method based on a double-flow network of frequency domain features, characterized by comprising the following steps:

[0015] Step 1: divide the FaceForensics++ dataset and the Celeb-DF dataset into training sets and test sets;

[0016] Step 2: extract the training set and test set video frames respectively, and take 20 frames for each video;

[0017] Step 3: perform face region segmentation on the video frames, and then perform two-dimensional and three-dimensional Fourier transform on the segmented blocks and the whole face to obtain frequency domain distribution features;

[0018] Step 4: input the features obtained in step 3 into the face forgery detection double-flow network, calculate the local loss and the global loss, and obtain the total loss;

[0019] Step 5: loop step 4 until the double-flow network converges, and save the trained model file;

[0020] Step 6: input a group of test set features into the face forgery detection model obtained in step 5 to obtain the judgment result of whether the features come from a fake video.

[0021] Further, the step 3:

[0022] The Facecrop algorithm is used to obtain the whole face image I(0), the segmented blocks are calculated according to the 68 key points of the face, the left face I(1) is positioned by taking the 2nd, 3rd, 4th and 32nd key points, the right face I(2) is positioned by taking the 14th, 15th, 16th and 36th key points, the left eye socket I(3) is positioned by taking the 41st, 42nd and 29th key points, the right eye socket I(4) is positioned by taking the 47th, 48th and 29th key points, the mouth I(5) is positioned by taking the 51st and 53rd key points, and the nose I(6) is positioned by taking the 28th, 32nd, 34th and 36th key points.

[0023] The calculation formula of the two-dimensional discrete Fourier transform is as follows:

[0024]

[0025] where W k represents the width of region I(k), H k represents the height of region I(k), j is the imaginary unit, and π is the circular constant. I2represents a two-dimensional frequency domain image, I represents a time domain image, a is in the range of [0, W k -1], b is in the range of [0, H k -1], and I(x, y, k) is the pixel gray value at (x, y) in region I(k), and k is in the range of {0, 1, 2, 3, 4, 5, 6}.

[0026] The three-dimensional discrete Fourier transform calculation formula is as follows:

[0027]

[0028] where W k represents the width of region I(k), H k represents the height of region I(k), j is the imaginary unit, and π is the circular constant. For each image, a total of 20 frames are taken forward and backward, so z is in the range of [0, 19], a is in the range of [0, W k -1], b is in the range of [0, H k -1], c is in the range of [0, 19], I3represents a three-dimensional frequency domain image, I represents a time domain image, I(x, y, z, k) is the pixel gray value at (x, y) in region I(k) of the zth frame, and k is in the range of {0, 1, 2, 3, 4, 5, 6}.

[0029] After the two-dimensional discrete Fourier transform and the three-dimensional discrete Fourier transform are performed on region k respectively, a two-dimensional frequency domain image I2(k) = DFT2(I(k)) and a three-dimensional frequency domain image I3(k) = DFT3(I(k)) are obtained. The obtained results are respectively subjected to serialization processing, one-dimensional data is obtained by sorting according to the amplitude value size, the data value range is projected to [0, 80], that is, 80 frequency bands are divided, the mean value of the energy distribution under different frequency bands is calculated, F2(k) and F3(k) are obtained, and region k characteristics F(k) = {F2(k), F3(k)} are connected. After the above calculation is performed on all 7 regions, the characteristics of the video frame are obtained:

[0030] F = {F(0), F(1), F(2), F(3), F(4), F(5), F(6)}

[0031] Where F(0) is the full face feature, F(1) is the local left face feature, F(2) is the local right face feature, F(3) is the local left eye feature, F(4) is the local right eye feature, F(5) is the local mouth feature, and F(6) is the local nose feature.

[0032] Further, the step 4:

[0033] The input features are split into F(0), F(1), F(2), F(3), F(4), F(5), F(6), and further split F(k) into F2(k) and F3(k) to obtain 14 feature vectors. They are placed in corresponding branches, where F2(0) and F3(0) are regarded as global branches, and the rest are local branches. The above features are respectively input into a fully connected layer for feature dimensionality increase:

[0034]

[0035] Where F2(k) is a 1x80-dimensional vector, and the output is a 1x128-dimensional vector.

[0036] A three-dimensional branch is added to the lateral connection of the two-dimensional branch to fuse the two-dimensional and three-dimensional features:

[0037]

[0038] Where the output is a 1x128-dimensional vector.

[0039] Then, the two-dimensional and three-dimensional branches are concatenated and passed through two fully connected layers for feature dimensionality reduction to obtain the prediction result:

[0040]

[0041] Where F 4 (k) is a 1x512-dimensional vector, and the output y(k) is a 1x2-dimensional vector, i.e., the prediction classification result.

[0042] The local loss L local is used to calculate the influence of the local texture feature on the classification structure, and the formula is as follows:

[0043]

[0044] In the formula, y i (1) represents the prediction result obtained by the left face local branch of the i-th sample, y i (2) represents the prediction result obtained by the right face local branch of the i-th sample, y i (3) represents the prediction result obtained by the left eye local branch of the i-th sample, and y i(4) represents the prediction result obtained by the right eye local branch of the i-th sample, y i (5) represents the prediction result obtained by the mouth branch of the i-th sample, y i (6) represents the prediction result obtained by the nose local branch of the i-th sample, and N is the set batch size;

[0045] Global loss L global For calculating the influence of local texture features on the classification structure, the calculation formula is as follows:

[0046]

[0047] In the formula, y i (0) represents the prediction result obtained by the global branch of the i-th sample, and N is the set batch size;

[0048] The total loss function formula is as follows:

[0049] L=λL global +L local

[0050] In the formula, λ is a weight parameter, which is best set to 1 according to experimental results.

[0051] By adopting the above scheme, the application has the beneficial effects that:

[0052] 1. The application avoids the problem of limitations of the frequency domain face forgery detection method in retaining local texture features and time domain features, uses face region segmentation and three-dimensional discrete Fourier transform method, and improves the detection accuracy;

[0053] 2. The application proposes a dual-flow network for frequency domain face forgery detection, which can effectively fuse local and global, spatial and temporal features, and can effectively reduce the consumption of computer resources in algorithm implementation due to being a lightweight network;

[0054] 3. The application proposes a novel dual-flow loss function to enhance local texture features while retaining global forgery features. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is the basic flow chart of the face forgery detection method of the dual-flow network based on the frequency domain features of the application.

[0056] Figure 2 is the local and global region cropping process of the application.

[0057] Figure 3 is the frequency domain feature calculation (taking global two-dimensional features as an example) of the application.

[0058] Figure 4 is the comparison of two-dimensional features and three-dimensional frequency domain features of part of video frames (taking a global image as an example) of the present application.

[0059] Figure 5 is a structural diagram of the dual-flow network of the present application.

[0060] Figure 6 is a graph of experimental results using two-dimensional, three-dimensional and fused two-dimensional and three-dimensional features.

[0061] Figure 7 is the comparison of AUC of the present application and prior art.

[0062] Figure 8 is the comparison of model parameter quantity of the present application and prior art. DETAILED DESCRIPTION

[0063] The data set used in the experiment is from the public face fake video data set FaceForensics++ and Celeb-DF, and the FaceForensics++ data set contains 4000 different videos, which are 1000 real videos downloaded from the Internet and 1000 Deepfake tampered videos, 1000 Face2Face tampered videos and 1000 FaceSwap tampered videos synthesized from these videos. The experimental results are shown in Figure 6 The present application mainly proposes a face fake detection method of a dual-flow network of frequency domain features, and proposes a dual-flow loss function, and for this purpose, the present application also makes an additional comparison experiment, which is the test results obtained by using two-dimensional, three-dimensional and fused two-dimensional and three-dimensional features during training, as shown in Figure 5 The experimental results show that the features obtained by the three-dimensional discrete Fourier transform can effectively improve the judgment accuracy. The present application also makes an additional comparison experiment with the closest prior art, Figure 7 is the comparison of AUC of the present application and 12 kinds of prior art on Celeb-DFv1 and Celeb-DFv2, and the experimental results show that the AUC value of the present application is the highest in this test data. Figure 8 is the comparison of network parameter quantity of the present application and prior art, and it can be seen that the network parameter quantity of the present application is less, which can effectively reduce the consumption of computer resources in the implementation of the algorithm.

[0064] The present application will be further described below in conjunction with the accompanying Figure 1 flowchart, obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments.

[0065] Step 1: Divide 1000 real videos and 1000 Deepfake tampered videos in the FaceForensics++ data set into training set and test set according to 4:1 at random.

[0066] Step 2: Uniformly cut 20 frames from each video to obtain 32,000 images for the test set and 8,000 images for the training set.

[0067] Step 3: Crop the face region to obtain a 128*128 pixel image using facecrop, and then identify the face key points to crop the facial feature region to obtain 6 20*20 pixel images. Perform two-dimensional and three-dimensional Fourier transform on the above images and then serialize to obtain the frequency domain distribution features.

[0068] As shown in Figure 2 , the face region (blue frame) is obtained using the Facecrop algorithm, and then it is adjusted and cropped to obtain a 128*128 pixel full face image I(0) (green frame). The segmentation block is calculated according to the 68 key points {p 1 , p 2 , p 3 , …, p 68}, where p represents the x-coordinate of the i-th key point, p represents the y-coordinate of the i-th key point, and i takes the value range [1, 68].

[0069] The left face region I(1) is located by taking the 2nd, 3rd, 4th, and 32nd key points:

[0070]

[0071] where I represents the video frame, e represents the segmentation block length, i.e., 20, is the left endpoint, is the right endpoint, is the upper endpoint, is the lower endpoint, and the following is the same.

[0072] The right face region I(2) is located by taking the 14th, 15th, 16th, and 36th key points:

[0073]

[0074] The left eye socket region I(3) is located by taking the 41st, 42nd, and 29th key points:

[0075]

[0076] The right eye socket region I(4) is located by taking the 47th, 48th, and 29th key points:

[0077]

[0078] The mouth region I(5) is located by taking the 51st and 53rd key points:

[0079]

[0080] Take 28, 32, 34, 36 key points to locate the nose region I (6):

[0081]

[0082] The calculation formula of the two-dimensional discrete Fourier transform is as follows:

[0083]

[0084] where W k represents the width of region I (k), H k represents the height of region I (k), j is the imaginary unit, and π is the circular constant. I2 represents the two-dimensional frequency domain image, I represents the time domain image, the value range of a is [0, W k -1], the value range of b is [0, H k -1], and I (x, y, k) is the pixel gray value at (x, y) in region I (k).

[0085] The calculation formula of the three-dimensional discrete Fourier transform is as follows:

[0086]

[0087] where W k represents the width of region I (k), H k represents the height of region I (k), j is the imaginary unit, and π is the circular constant. For each image, a total of 20 frames are taken forward and backward, so the value range of z is [0, 19], the value range of a is [0, W k -1], the value range of b is [0, H k -1], the value range of c is [0, 19], I3 represents the three-dimensional frequency domain image, I represents the time domain image, and I (x, y, z, k) is the pixel gray value at (x, y) in region I (k) of the z-th frame.

[0088] After two-dimensional discrete Fourier transform and three-dimensional discrete Fourier transform are performed on region k respectively, the two-dimensional frequency domain image I2 (k) = DFT2 (I (k)) and the three-dimensional frequency domain image I3 (k) = DFT3 (I (k)) are obtained. The obtained results are respectively subjected to serialization processing, and one-dimensional data is obtained by sorting according to the amplitude value. The data value range is projected to [0, 80], that is, divided into 80 frequency bands, the mean value of the energy distribution under different frequency bands is calculated, F2 (k) and F3 (k) are obtained, and region k feature F (k) = {F2 (k), F3 (k)} is connected. Figure 3 An example of the process of obtaining F2 (0) from I (0) is given. Figure 4The differences between F2(0) and F3(0) on the FaceForensics++ dataset are compared. The above calculation is performed on all 7 regions to obtain the features of the video frame:

[0089] F={F(0), F(1), F(2), F(3), F(4), F(5), F(6)}

[0090] where F(0) is the full face feature, F(1) is the local left face feature, F(2) is the local right face feature, F(3) is the local left eye feature, F(4) is the local right eye feature, F(5) is the local mouth feature, and F(6) is the local nose feature.

[0091] Step 4: Input the features obtained in step 3 into the face forgery detection double-flow network in Figure 5 , calculate the local loss and global loss, and obtain the total loss;

[0092] Split the input features into F(0), F(1), F(2), F(3), F(4), F(5), and F(6), and further split F(k) into F2(k) and F3(k) to obtain 14 feature vectors. Put them into the corresponding branch, where F2(0) and F3(0) are considered as the global branch, and the rest are considered as the local branch. For example, Figure 5 The above features are input into a fully connected layer for feature dimensionality increase:

[0093]

[0094] where F2(k) is a 1x80-dimensional vector, and the output is a 1x128-dimensional vector.

[0095] Add a three-dimensional branch to the two-dimensional lateral connection, and fuse the two-dimensional and three-dimensional features:

[0096]

[0097] where the output is a 1x128-dimensional vector.

[0098] Then concatenate the two-dimensional and three-dimensional branches and pass them through two fully connected layers to reduce the dimensionality of the features and obtain the prediction results:

[0099]

[0100] where F 4 (k) is a 1x512-dimensional vector, and the output y(k) is a 1x2-dimensional vector, which is the prediction classification result.

[0101] Replace k to get the local loss L local, for calculating the influence of local texture features on the classification structure, the formula is as follows:

[0102]

[0103] In the formula, y i (1) represents the prediction result obtained by the left face local branch of the i-th sample, y i (2) represents the prediction result obtained by the right face local branch of the i-th sample, y i (3) represents the prediction result obtained by the left eye local branch of the i-th sample, y i (4) represents the prediction result obtained by the right eye local branch of the i-th sample, y i (5) represents the prediction result obtained by the mouth branch of the i-th sample, y i (6) represents the prediction result obtained by the nose local branch of the i-th sample, and N is the set batch size.

[0104] Global loss L global For calculating the influence of local texture features on the classification structure, the calculation formula is as follows:

[0105]

[0106] In the formula, y i (0) represents the prediction result obtained by the global branch of the i-th sample, and N is the set batch size.

[0107] The total loss function formula is as follows:

[0108] L = λL global + L local

[0109] In the formula, λ is a weight parameter, which is best set to 1 according to experimental results.

[0110] Step 5: Step 4 is executed in a loop until the dual-flow network converges, and the trained model file is saved;

[0111] Step 6: A set of test set features is input into the face forgery detection model obtained in step 5 to obtain a judgment result of whether the features come from a fake video.

Claims

1. A face forgery detection method based on a dual-flow network of frequency domain features, characterized in that, Comprising the following steps: Step 1: divide the FaceForensics++ dataset and the Celeb-DF dataset into training sets and test sets; Step 2: extract the video frames of the training set and the test set respectively, and take 20 frames for each video; Step 3: perform face region segmentation on the video frames, and then perform two-dimensional and three-dimensional Fourier transform on the segmented blocks and the whole face to obtain frequency domain distribution features; Step 4: input the features obtained in step 3 into the face forgery detection double-flow network, calculate the local loss and the global loss, and obtain the total loss, the specific steps are as follows: The input features are split into , and further split into , obtaining 14 feature vectors. Put into the corresponding branch, where , is regarded as a global branch, and the rest are local branches; The above features are input into a fully connected layer for feature dimensionality increase: wherein is a 1 x 80 dimensional vector, and output is a 1 x 128 dimensional vector; Add a three-dimensional branch to the lateral connection of the two-dimensional branch to fuse the two-dimensional and three-dimensional features: where the output is a 1 x 128 dimensional vector; Then concatenate the two-dimensional and three-dimensional branches and pass them through two fully connected layers to reduce the dimensionality of the features and obtain the prediction results: wherein is a 1x512 dimensional vector, the output is a 1x2 dimensional vector, i.e. the prediction classification result; local loss For computing the impact of local texture features on the classification structure, the formula is as follows: In the formula Indicates by the first The prediction results obtained from the local branch of the left face of each sample Indicates by the first The prediction results obtained from the local branch of the right face of a sample Indicates by the first The prediction results obtained from the left eye local branch of each sample Indicates by the first The prediction results obtained from the right eye local branch of each sample Indicates by the first The prediction results obtained from the mouth portion of each sample. Indicates by the first The prediction results obtained from the local branch of the nose of each sample, where N is the set batch size; global loss For computing the impact of local texture features on the classification structure, the following formula is used: In the formula represents the prediction result obtained by the global branch of the i-th sample, and N is the set batch size. represents the prediction result obtained by the global branch of the i-th sample, and N is the set batch size. The total loss function formula is as follows: wherein is a weight parameter, which is set to 1 as optimal according to experimental results; Step 5: loop step 4 until the double-flow network converges, and save the trained model file; Step 6: input a group of test set features into the face forgery detection model obtained in step 5 to obtain the judgment result of whether the features come from a forged video.

2. The face fake detection method based on frequency domain features of a dual-flow network according to claim 1, characterized in that, The step 3: Get full face image using Facecrop algorithm Calculate segmentation block according to human face 68 key points, take 2, 3, 4, 32 key points to position left face Take 14, 15, 16, 36 key points to position right face Take 41, 42, 29 key points to position left eye socket Take 47, 48, 29 key points to position right eye socket Take 51, 53 key points to position mouth Take 28, 32, 34, 36 key points to position nose ; The calculation formula of the two-dimensional discrete Fourier transform is as follows: wherein represents the width of the region , represents the height of the region , is the imaginary unit, is the circle constant. represents a two-dimensional frequency domain image, represents a time domain image, has a value range of , has a value range of , is the pixel gray value at in the region , has a value range of ; The calculation formula of the three-dimensional discrete Fourier transform is as follows: wherein represents the width of the region , represents the height of the region , is the imaginary unit, is the circle constant; for each image, 20 frames are taken in total, so the value range of z is [0, 19], the value range of x is , the value range of y is , the value range of z is , represents a three-dimensional frequency domain image, represents a time domain image, is the pixel gray value at the region in the zth frame , the value range of x is ; Region After two-dimensional discrete Fourier transform and three-dimensional discrete Fourier transform, respectively, two-dimensional frequency domain image , three-dimensional frequency domain image ; the obtained results are respectively sequenced, sorted by amplitude value to obtain one-dimensional data, the data value range is projected to [0, 80], that is, divided into 80 frequency bands, the mean value of energy distribution under different frequency bands is calculated, and , , the region characteristics ; Perform the above calculation on all 7 regions to obtain the features of the video frames: wherein is a full face feature, is a local left face feature, is a local right face feature, is a local left eye feature, is a local right eye feature, is a local mouth feature, is a local nose feature.

Images