Deep counterfeit video detection frame selection algorithm based on key reconstruction error

Abstract

The invention discloses a deep counterfeit video detection frame selection algorithm based on a key reconstruction error. The algorithm comprises the following steps: constructing and training a face attitude auto-encoder used for reconstructing a face three-dimensional model diagram; the method comprises the following steps: after reconstructing a face three-dimensional model diagram by using a face posture auto-encoder, extracting a key reconstruction error score of each video frame of a video to be detected; and sorting the key reconstruction error scores of all video frames of the to-be-detected video from small to large, and selecting video frames corresponding to the first K key reconstruction error scores as detection frames of the deep counterfeit video detection model. The method has the beneficial effects that according to the characteristics of a standard face video data set, the technologies such as a face three-dimensional model, a face posture self-encoder and local region texture are adopted, and low-quality samples with significant face deflection are screened out; the video frames input into the detection model are adaptively selected through image content analysis, so that the reliability and accuracy of the corresponding deep counterfeit video detection model are improved.

Description

Technical Field

[0001] The present invention relates to the field of multimedia information security technology, and in particular to a deep fake video detection frame selection algorithm based on key reconstruction error. Background Art

[0002] Deep fake video detection is an important research issue in forensic technology in the field of multimedia information security, and it is in constant competition and development with deep fake video technology.

[0003] At present, a large number of studies have been conducted to design detection algorithms from multiple dimensions such as spatial domain, frequency domain and time domain to capture abnormal traces generated by the deep fake process within the video frame and between two frames. However, most of the existing deep fake video detection algorithms use uniform frame extraction, random frame extraction or continuous frame extraction strategies in the preprocessing stage to convert the input video into an input form that can be used by the detection model, and then perform feature extraction and classification. Among them, uniform frame extraction refers to extracting frames from the video at a fixed time interval as the sampling result, random frame extraction refers to randomly selecting a number of frames from the video as the sampling result, and continuous frame extraction refers to extracting the required number of continuous frames from the video as the sampling result. Obviously, the above commonly used video frame extraction schemes do not consider the changes in video content, and cannot select video frames that are more suitable for deep fake video detection models. In addition, the existing deep fake video detection algorithms mainly use data-driven methods to train the detection model, and the samples processed in the test phase have differences in the head posture of the characters at different time points. When the head of the character in the video deflects to an angle that does not appear in the training sample, it is more difficult for the detection model to extract reliable forensic features, resulting in reduced reliability of the detection results provided by the classifier. Summary of the invention

[0004] The present invention provides a deep fake video detection frame selection algorithm based on key reconstruction error, which is used to select video frames that can provide more reliable classification results during deep fake video detection, so as to improve the reliability of the deep fake video detection model.

[0005] The technical solution of the present invention is as follows:

[0006] Deep fake video detection frame selection algorithm based on key reconstruction error, including:

[0007] Step 1, constructing and training a face pose autoencoder for reconstructing a three-dimensional face model image; the face pose autoencoder includes an encoder and a decoder; wherein the encoder is composed of two or more convolution blocks in cascade, and the convolution block includes one or more convolution layers and one or more non-linear activation functions; the decoder is composed of two or more deconvolution blocks in cascade, and the deconvolution block includes one or more deconvolution layers and one or more non-linear activation functions; the loss function of the face pose autoencoder is a mean square error loss function;

[0008] Step 2, extracting the key reconstruction error score of each video frame of the video to be tested;

[0009] 2.1 The face region of the video frame of the video to be tested is obtained by using the face recognition algorithm, and then the 3D face model extraction algorithm is used to obtain the 3D face model image, and then the face posture autoencoder is used to obtain the reconstructed 3D model image; the difference between the reconstructed 3D model image and the 3D face model image is used as the reconstruction error of the video frame;

[0010] 2.2 Divide the reconstruction error of the video frame into b×b non-overlapping square areas, and calculate the texture complexity of each square area; the texture complexity of the square area is the average value of the horizontal two-dimensional image entropy, the vertical two-dimensional image entropy and the diagonal two-dimensional image entropy of the square area;

[0011] 2.3 Select a square area with the maximum texture complexity, and sum the absolute values ​​of all elements in the area to obtain the value as the key reconstruction error score of the video frame;

[0012] Step 3: Sort the key reconstruction error scores of all video frames of the video to be tested from small to large, and then select the video frames corresponding to the top K key reconstruction error scores as the detection frames of the deep fake video detection model.

[0013] Preferably, the facial posture autoencoder is trained using a data set for training the deep fake video detection model.

[0014] Preferably, the convolution block further includes a batch normalization layer, and the deconvolution block further includes a batch normalization layer.

[0015] Compared with the prior art, the beneficial effects of the present invention are: based on the characteristics of the standard face video dataset, the face 3D model, face posture autoencoder, local area texture and other technologies are used to screen out low-quality samples with significant face deflection; through image content analysis, the video frames of the input detection model are adaptively selected, thereby improving the reliability and accuracy of the corresponding deep fake video detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention.

[0017] Figure 2 It is a schematic diagram of face area recognition and three-dimensional model calculation. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0019] like Figure 1As shown, the present invention proposes a deep fake video detection frame selection algorithm based on key reconstruction error, comprising the following steps:

[0020] S1: Using video dataset D t , construct and train the face pose autoencoder. In order to better adapt the sample features learned by the deep fake video detection model, in this embodiment, the video dataset D t The training set for training deep fake video detection models was used.

[0021] S1.1 to D t Decompress the video in to get the video frame, expressed as: F = [f1,f2,f3,…,f i ,…], i=1,2,3…,N, where N represents the total number of video frames.

[0022] S1.2 For each video frame f i Use the face recognition algorithm Dlib to locate the face area and output the box i =[xmin i ,ymin i ,xmax i ,ymax i ] represents the bounding box coordinates of the face area in the video frame, where (xmin i ,ymin i ) is the upper left corner of the border, (xmax i ,ymax i ) is the lower right corner of the border. i Crop the facial region f′ of the person in the video frame i For the facial region f′ i Use the facial 3D model extraction algorithm 3DDFA-V2 to calculate the 3D model graph of the face i The face recognition algorithm and the facial 3D model extraction algorithm may use other algorithms in the prior art.

[0023] S1.3 build and train face pose autoencoder AE(s;θ):

[0024] s′ i =AE(s i ;θ)

[0025] Among them, s i ′is a three-dimensional model image of a face reconstructed by an autoencoder, and θ is a parameter of the autoencoder. The autoencoder AE(·) consists of an encoder and a decoder. In this embodiment, the structure of the encoder is composed of three cascaded convolution blocks, each of which is composed of a 3×3 convolution layer with a convolution step size of 2×2, a ReLU activation function, and a batch normalization layer cascaded, and the number of channels of the three convolution layers is 32, 64, and 128, respectively. The decoder is composed of three cascaded deconvolution blocks, each of which is composed of a 3×3 deconvolution layer with a convolution step size of 2×2, a ReLU activation function, and a batch normalization layer cascaded, and the number of channels of the three deconvolution layers is 64, 32, and 3, respectively. In this embodiment, a batch normalization layer is added to the autoencoder, which can further improve the convergence speed and performance of the autoencoder.

[0026] The mean square error loss function loss is used to train the autoencoder AE(·), which can be expressed as:

[0027]

[0028] Among them, (x, y) traverses the face 3D model graph s i Each pixel, N x =xmax i -xmin i , N y =ymax i -ymin i .

[0029] S2: Use the trained autoencoder to extract the key reconstruction error score for each frame of the input video.

[0030] S2.1 For the input video, decompress it into video frames. Input the video frames into the trained autoencoder AE (s i θ * ) to obtain the reconstructed 3D model graph s′ i . Calculate the reconstruction error of each frame image R = [r′1, r′2, r′3, …, r′ i ,…], where r′ i =s′ i -s i .

[0031] S2.2 selects the region with the richest texture in each reconstruction error and calculates the key reconstruction error. Specifically, the reconstruction error r′ of each frame i Divide into b×b non-overlapping square regions Where k represents the number of the square area, Indicates the rounding down operation. Calculate the area of ​​each square The texture complexity Texture complexity is calculated based on the 2D image entropy in different directions, including horizontal 2D image entropy Vertical 2D Graph Entropy and diagonal 2D image entropy The calculation method is:

[0032]

[0033] where j∈{hor,ver,dia}, Indicates that in the square area The number of occurrences of a pair of horizontally adjacent elements in , where the values ​​are u and v respectively. Indicates that in the square area The number of occurrences of a pair of vertically adjacent elements taking values ​​u and v respectively. Indicates that in the square area The number of occurrences of a pair of diagonal adjacent elements taking values ​​u and v respectively. The final square area The texture complexity of In this embodiment, b=4.

[0034] S2.3 Calculate each reconstruction error r′ i All square areas in The texture complexity of the square area with the largest texture complexity is selected The square area The absolute values ​​of all elements in are summed to obtain the key reconstruction error score s of the video frame. i For the video frame sequence to be tested, a key reconstruction error score sequence can be constructed, S = [s 1 ,s 2 ,s 3 ,…,s i ,…].

[0035] S3: For the input video to be tested, calculate the key reconstruction error score of each video frame. Sort the video frames from small to large according to the reconstruction error score, and select the first K video frames. This is the output result of the frame selection algorithm. In the subsequent deep fake video detection process, the detection model only processes the video frames obtained after frame selection to obtain frame-level or video-level detection results. In this embodiment, K=10.

[0036] In order to verify the performance improvement of the above-mentioned video frame selection algorithm on deep fake video detection. The following experiment was designed, and the FaceForensics++ (FF++) dataset was used for training and testing. The FF++ dataset includes 1000 real videos and 4000 fake videos. The fake videos use four fake technologies, DeepFakes (DF), Face2Face (F2F), FaceSwap (FS) and NeuralTextures (NT), to generate corresponding fake videos based on real videos. In this experiment, the XceptionNet network is used as a deep fake detector. According to the official division method, the FF++ dataset is divided into training sets, test sets and validation sets in a ratio of 720:140:140. The training set and validation set are used to train the deep fake detector, and the autoencoder is trained. Accuracy (ACC, %), F1 score (F1-score, %) and AUC (Area Under Curve, %) are used as evaluation indicators of detection performance. The detection results are obtained by using random frame extraction, uniform frame extraction and the frame selection algorithm of the present invention, combined with the XceptionNet detector, as shown in the following table:

[0037]

[0038] Experiments show that, under the premise of using the same detector, compared with random frame sampling and uniform frame sampling, the frame selection algorithm based on key reconstruction error can effectively improve the performance of the detection model, that is, exclude samples in the test set that are significantly different from the training set, and avoid unreliable detection results. In addition, the frame selection algorithm of the present invention can also be used in conjunction with various other deep fake detection models, with good flexibility.

Claims

1. A deep fake video detection frame selection algorithm based on key reconstruction error, characterized by: include: Step 1, constructing and training a face pose autoencoder for reconstructing a 3D face model image; The face pose autoencoder includes an encoder and a decoder; wherein the encoder is composed of two or more convolution blocks in cascade, and the convolution blocks include one or more convolution layers and one or more nonlinear activation functions; the decoder is composed of two or more deconvolution blocks in cascade, and the deconvolution blocks include one or more deconvolution layers and one or more nonlinear activation functions; the loss function of the face pose autoencoder is a mean square error loss function; Step 2, extracting the key reconstruction error score of each video frame of the video to be tested; 2.1 The face region of the video frame of the video to be tested is obtained by using the face recognition algorithm, and then the 3D face model extraction algorithm is used to obtain the 3D face model image, and then the face posture autoencoder is used to obtain the reconstructed 3D model image; the difference between the reconstructed 3D model image and the 3D face model image is used as the reconstruction error of the video frame; 2.2 Divide the reconstruction error of the video frame into b×b non-overlapping square areas, and calculate the texture complexity of each square area; the texture complexity of the square area is the average value of the horizontal two-dimensional image entropy, the vertical two-dimensional image entropy and the diagonal two-dimensional image entropy of the square area; 2.3 Select a square area with the maximum texture complexity, and sum the absolute values ​​of all elements in the area to obtain the value as the key reconstruction error score of the video frame; Step 3: Sort the key reconstruction error scores of all video frames of the video to be tested from small to large, and then select the video frames corresponding to the top K key reconstruction error scores as the detection frames of the deep fake video detection model.

2. The deep fake video detection frame selection algorithm based on key reconstruction error as claimed in claim 1, characterized in that The face pose autoencoder is trained using the dataset used to train the deep fake video detection model.

3. The deep fake video detection frame selection algorithm based on key reconstruction error as claimed in claim 1, characterized in that The convolution block further includes a batch normalization layer, and the deconvolution block further includes a batch normalization layer.

Images