A method and system for detecting a fake video

Abstract

The application provides a method for detecting a fake video, comprising the following steps: S1. obtaining a set of fake videos, marking the authenticity of the target objects in the set of fake videos, and performing video data preprocessing to obtain a training sample set, a test sample set and a verification sample set; S2. constructing a single-stream residual dilated 3D convolution network; S3. inputting the training sample set into the constructed network and performing training to obtain optimal parameters; S4. inputting the verification sample set into the constructed network and loading the optimal parameters, detecting and classifying the authenticity of the target objects in the test sample set and the verification sample set to obtain a classification result; and S5. comprehensively classifying the classification result to obtain the judgment probability of the authenticity of the target objects in the test sample set and the verification sample set. The application constructs a single-stream residual dilated 3D convolution network, removes the optical flow network in the double-stream dilated 3D convolution network to reduce it to an RGB single-stream network, thereby improving the adaptability of the detection of the fake video, and adds a residual module to improve the training ability of the network in the deep layer.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a method and system for detecting fake videos. Background Technology

[0002] With the development of mobile internet and the explosive growth of video data, coupled with the rapid open-source popularization of high-quality video forgery and tampering technology, various forms of deepfake video cases have emerged, seriously impacting social stability and public safety.

[0003] Most existing video forgery techniques forge videos frame by frame independently. Because the forgery lacks temporal continuity, it leads to contextual anomalies in the video. That is, in addition to the forgery features of a single frame, forged videos also exhibit temporal forgery features such as inconsistencies between consecutive frames. Therefore, the key technology in video detection lies in how to enable networks to better represent spatial and temporal information.

[0004] However, the optical flow network used in existing video detection technologies has poor adaptability to face video forgery recognition, and the training difficulties caused by network depth also need to be solved. Summary of the Invention

[0005] In response to at least one defect or improvement requirement of the prior art mentioned in the background section, the present invention provides a method for detecting forged videos, comprising the following steps:

[0006] S1. Obtain a set of fake videos, label the authenticity of the targets in the set of fake videos, and perform video data preprocessing to obtain a training sample set, a test sample set, and a verification sample set;

[0007] S2. Construct a single-stream residual dilated 3D convolutional network;

[0008] S3. Input the training sample set into the single-stream residual dilated 3D convolutional network and train it to obtain the optimal parameters of the single-stream residual dilated 3D convolutional network;

[0009] S4. Input the verification sample set into the single-stream residual dilated 3D convolutional network and load the optimal parameters to detect and classify the authenticity of the target objects in the test sample set and the verification sample set, and obtain the classification result;

[0010] S5. Based on the classification results, obtain the probability of determining the authenticity of the target object in the test sample set and the verification sample set.

[0011] According to the method for detecting fake videos provided by the present invention, the single-stream residual dilated 3D convolutional network includes a single-stream module;

[0012] The single-stream module is formed by removing the optical flow network from the dual-stream dilated 3D convolutional network.

[0013] According to the method for detecting fake videos provided by the present invention, the single-stream residual dilated 3D convolutional network further includes a residual module;

[0014] The residual module is accessed by connecting it in a skip connection manner among multiple serial Inception Modules.

[0015] According to the method for detecting fake videos provided by the present invention, the video data preprocessing specifically includes the following steps:

[0016] S11. Separate the video object into image objects frame by frame;

[0017] S12. Locate the position of the target object from the frame images obtained by video framing, and crop the position of the target object in the frame images to obtain a target object image set;

[0018] S13. Group the target object images of each consecutive first preset number of frames together as a set of frame images.

[0019] According to the method for detecting fake videos provided by the present invention, the first preset number is 16.

[0020] According to the method for detecting fake videos provided by the present invention, the objective function of the single-stream residual dilated 3D convolutional network adopts the cross-entropy function:

[0021]

[0022] Where H(X) represents information entropy, P(x) i ) represents the probability predicted for this category.

[0023] According to the method for detecting fake videos provided by the present invention, the training method in step S3 specifically includes the following steps:

[0024] S31. Initialize the parameters of the single-stream residual dilated 3D convolutional network;

[0025] S32. Perform one round of training and adjust the parameters accordingly;

[0026] S33. Conduct one round of testing and record the results of that round;

[0027] S34. Repeat step S32 until convergence.

[0028] According to the method for detecting fake videos provided by the present invention, each training round reads 8 batch sizes of data, 4 processes run in parallel, and the training learning rate is 0.001.

[0029] According to the method for detecting fake videos provided by the present invention, the specific method for determining the authenticity probability of the target object in the test sample set and the verification sample set by combining the classification results is as follows:

[0030] The set of frame images is sequentially input into the trained single-stream residual dilated 3D convolutional network, and the network outputs the comprehensive true probability for each frame image set. A probability higher than 50% is labeled as true, and a probability lower than 50% is labeled as false. The true / false determination results for all output frame image sets are statistically analyzed and compared with the labels, and the results are recorded. The probability of determining the authenticity of the target object is used as the final output.

[0031] Secondly, the present invention provides a system for detecting fake videos, which can implement the steps of the method for detecting fake videos described in any of the above claims.

[0032] This invention improves the adaptability to fake video detection by constructing a single-stream residual dilated 3D convolutional network and removing the optical flow network from the dual-stream dilated 3D convolutional network to reduce it to an RGB single-stream network. At the same time, the addition of a residual module improves the network's training capability at deeper levels. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the following drawings are drawings of some preferred embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating a method for detecting fake videos provided in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of a single-stream residual dilated 3D convolutional network provided in an embodiment of the present invention;

[0036] Figure 3 This is a model construction diagram of the single-stream residual dilated 3D convolutional network provided in an embodiment of the present invention;

[0037] Figure 4This is a schematic diagram of the residual module in the Mixed layer provided in an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to some accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0039] like Figure 1 As shown, the first embodiment provides a method for detecting fake face videos, including the following steps S1-S5:

[0040] S1. Obtain a set of fake face videos, label the authenticity of the faces (target objects) in the fake video set, and perform video data preprocessing to obtain a training sample set, a test sample set, and a verification sample set.

[0041] The fake face video set mentioned in step S1 includes a training video set, a test video set, and a verification video set.

[0042] The first-generation fake dataset (FaceForensics++ dataset) was used as both the training and testing video set for building the network. The training video set was used to update the network parameters during training.

[0043] The FaceForensics++ dataset includes 1000 real videos and 1000 fake videos generated using four publicly known forgery methods: Deepfake, Face2Face, FaceSwap, and NeuralTextures. Of the 1000 fake videos generated by each forgery method, 720 videos were preprocessed to form a training sample set used to train the network parameters, while the remaining 280 videos were preprocessed to form a training sample set used to test the network performance during training.

[0044] The second-generation fake dataset (Celeb-DF dataset) was used as the validation video set for the constructed network. The constructed network trained on the FaceForensics++ dataset was tested on the Celeb-DF dataset to verify the generalization ability of the constructed network.

[0045] The Celeb-DF dataset consists of 590 real videos and 5649 fake videos generated using publicly available forgery methods. A validation sample set was created after video data preprocessing for verification purposes.

[0046] The video data preprocessing in step S1 specifically includes:

[0047] Video frame segmentation: Using algorithms provided by OpenCV, video objects are separated into image objects based on the frames played by the data media.

[0048] Face localization: The RetinaFace method is used to locate the face position in the frame images obtained from the video frame segmentation, and the face position in the frame image is cropped to obtain a face image set.

[0049] Frame set reassembly: Every 16 consecutive frames of face images are grouped together and considered as a frame image set. That is, the original video set is processed into multiple sets of 16 face frame images, representing a set of video frames with certain spatiotemporal information.

[0050] The larger the number of consecutive frames, the better it reflects the overall integrity of the video; the smaller the number, the better it shows the details of facial movements within a short period. Considering the graphics card's memory capacity, the limited size of the input vector, and the binary computation mechanism, after considering these factors and conducting experiments, a frame length of 16 was intentionally set. These 16 consecutive video frames are grouped into a 'small video' and used as the input to build the network.

[0051] S2. Construct a video classifier based on a single-stream residual dilated 3D convolutional network.

[0052] like Figure 2 As shown, the single-stream residual dilated 3D convolutional network (RGB-Inception-ResNet3D convolutional network) described in step S2 includes a batch normalization module, a single-stream module, and a residual module.

[0053] Batch Normalization Module: Normalizes the input values ​​of a batch in a layer through the BN layer.

[0054] The batch normalization module consists of the following layers: convolutional layer C3D_1a (7x7x7), normalization layers BN1 and Scal1, and activation function layer ReLU1.

[0055] Single-stream module: such as Figure 3 As shown, the optical flow network in the dual-stream dilated 3D convolutional network is removed, forming an RGB single-stream network. This improves the adaptability of the 3D convolutional network to face spoofing video detection. The RGB single-stream network dilates the convolutional kernels of the Inception module in the InceptionV2 model into three-dimensional convolutional kernels, which can extract spatiotemporal information from face spoofing videos.

[0056] The hierarchy of a single-stream module includes:

[0057] The first layer is: a 3D max-pooling layer MaxPool3d_2a(1x3x3).

[0058] The second layer consists of: a convolutional layer C3D_2a (3x3x3), normalization layers BN2 and Scal2, and an activation function layer ReLU2.

[0059] The third layer is the 3DInception layer, which includes the following:

[0060] MaxPool3d_3a(1x3x3)

[0061] Mixed-3b, Mixed-3c

[0062] MaxPool3d_4a(3x3x3)

[0063] Mixed-4b, Mixed-4c, Mixed-4d, Mixed-4e, Mixed-4f

[0064] MaxPool3d_5a(2x2x2)

[0065] Mixed-5b, Mixed-5c

[0066] Residual modules: Residual modules are connected in a skip connection manner between multiple serial Inception Modules to improve the network's training capability at deeper layers.

[0067] A schematic diagram of the residual module in the Mixed layer is shown below. Figure 4 As shown, the Mixed layer includes Branch_0, Branch_1, Branch_2, and a residual module, with output dimensions of (8,16,25,25,32), (8,16,25,25,32), (8,16,25,25,64), and (8,16,25,25,128), respectively. The total output dimension after concatenation is (8,16,25,25,256). The residual module adopts a bottleneck residual module form, which reduces the number of parameters and lowers the risk of overfitting.

[0068] The objective function of a single-stream residual dilated 3D convolutional network uses the cross-entropy function:

[0069]

[0070] Where H(X) represents information entropy, P(x) i ) represents the probability predicted for this category.

[0071] Optical flow is the instantaneous velocity of pixels moving on the imaging plane of a spatially moving object, and it is a commonly used method for extracting temporal features. Optical flow is divided into dense optical flow and sparse optical flow. Dense optical flow is a method that performs point-by-point matching on an image or a specified region, calculating the offset of all points on the image.

[0072] 3D convolution works by stacking multiple consecutive frames into a cube, and then applying 3D convolutional kernels within that cube. The 3D convolution kernels convolve both the temporal and spatial dimensions simultaneously; that is, they use higher-dimensional convolutional kernels and higher-dimensional pooling operators to build the convolutional network, thereby capturing changes in both the temporal and spatial dimensions. In this structure, each feature map in the convolutional layer is connected to multiple neighboring consecutive frames from the previous layer, thus capturing motion information.

[0073] In the experiment, dense optical flow maps were input into a 3D convolutional network to test its performance in detecting fake videos. The original set of fake video frames and the set of optical flow maps were input into the 3D convolutional network, and tests were conducted on all fake video methods and the NeuralTextures (NT) method, which has the greatest impact on the detection results. The accuracy comparison is shown in Table 1.

[0074]

[0075] Table 1

[0076] The results from dense optical flow methods are slightly inferior to those from conventional methods. This indicates that incorporating optical flow information does not significantly improve the extraction of video forgery and tampering traces. This may be because optical flow maps overemphasize cross-frame motion, thus losing the original intra-frame information of the image, which plays a crucial role in forgery detection.

[0077] Among existing 3D convolutional network architectures, I3D and R3D networks and their derivative models have achieved good results in action detection research, and can provide some selection experience for video forgery detection tasks. This study experimentally tested the detection performance of I3D and R3D on the FF++ dataset (FaceForensics++ dataset). The accuracy comparison table is shown in Table 2:

[0078]

[0079] Table 2

[0080] It can be seen that in the field of face forgery detection, networks with residual modules are more suitable. This is because the I3D network is a two-stream network based on RGB and optical flow, and existing experiments have shown that optical flow information cannot effectively extract forgery and tampering traces. At the same time, the addition of residual modules improves the network's training ability to a certain extent.

[0081] S3. Input the training sample set into the single-stream residual dilated 3D convolutional network and train it to obtain the optimal parameters of the single-stream residual dilated 3D convolutional network.

[0082] The training method in step S3 specifically includes the following steps:

[0083] S31. Initialize the parameters of the single-stream residual dilated 3D convolutional network;

[0084] S32. Perform one round of training and adjust the parameters accordingly;

[0085] S33. Conduct one round of testing and record the results of that round;

[0086] S34. Repeat step S32 until convergence.

[0087] Each training round is configured to read data in batches of 8, with 4 processes running in parallel and a training learning rate of 0.001. The above three parameters are hyperparameters, representing optimal values ​​derived from experimental testing.

[0088] S4. Input the verification sample set into the single-stream residual dilated 3D convolutional network and load the optimal parameters to detect and classify the authenticity of the faces (target objects) in the test sample set and the verification sample set, and obtain the classification result.

[0089] S5. Based on the classification results, obtain the probability of determining the authenticity of the face (target object) in the test sample set and the verification sample set.

[0090] After the input frame image set is processed by a single-stream residual dilated 3D convolutional network, the network outputs the combined true probability of the 16 output frame image sets. A probability higher than 50% is denoted as SR (True) and a probability lower than 50% is denoted as SF (False). The true / false probabilities of all output frame image sets are statistically analyzed and compared with the labels to obtain the accuracy and ROC curve for forgery detection. Finally, the classification of all frame image sets in the test video is statistically analyzed, and SR / (SR+SF) is taken as the true probability (accuracy) of the video as the final output.

[0091] Performance testing experiment:

[0092] From the FF++ dataset, 4000 fake frame images were divided into sets of 16 frames each. These sets included 1000 DeepFake fake frames, 1000 FaceSwap fake frames, 1000 Face2Face fake frames, and 1000 NeuralTexture fake frames. These were then compiled into a test frame image set. A pre-trained single-stream residual dilated 3D convolutional network was loaded, and the 4000 test frame images were used to determine whether they were genuine or fake. The accuracy is shown in Table 3.

[0093]

[0094] Table 3

[0095] Generalization performance experiment:

[0096] From the Celeb-DF dataset, images were divided into sets of 1000 real and 1000 fake videos, each set consisting of 16 frames. These were then combined into a test video set of 2000.

[0097] A 3D convolutional detection model trained on the FF++ dataset was loaded, and the authenticity of 2000 test videos was tested. The accuracy of the network on the Celeb-DF dataset was 66.7%.

[0098] From the FF++ dataset and the Celeb-DF dataset, at a 1:1 extraction ratio, the images were divided into 1000 real and 1000 fake video sets, with each set consisting of 16 frames.

[0099] A set of 2000 test videos was compiled. A single-stream residual dilated 3D convolutional network trained on the FF++ and Celeb-DF datasets was loaded, and the authenticity of the 2000 test videos was tested. The accuracy of the experimental network on the FF++ and Celeb-DF datasets was 65.6%.

[0100] The second embodiment provides a system for detecting fake videos, which can implement the steps of the method for detecting fake videos described in any of the above embodiments.

[0101] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure, and all such combinations and / or combinations fall within the scope of this disclosure.

[0102] Although this disclosure has been shown and described with reference to specific exemplary embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made to this disclosure without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Therefore, the scope of this disclosure should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents.

Claims

1. A method for detecting forged videos, characterized in that, Includes the following steps: S1. Obtain a set of fake videos, label the authenticity of the targets in the set of fake videos, and perform video data preprocessing to obtain a training sample set, a test sample set, and a verification sample set; S2. Construct a single-stream residual dilated 3D convolutional network; the single-stream residual dilated 3D convolutional network includes a single-stream module and a residual module; the single-stream module forms an RGB single-stream network by removing the optical flow network in the dual-stream dilated 3D convolutional network, and the RGB single-stream network dilates the convolutional kernel of the Inception module in the InceptionV2 model into a three-dimensional convolutional kernel; The single-stream module consists of the following layers: the first layer is a 3D max pooling layer MaxPool3d_2a; the second layer consists of a convolutional layer C3D_2a, normalization layers BN2 and Sca12, and an activation function layer ReLU2; the third layer is a 3D Inception layer, consisting of MaxPool3d_3a, Mixed-3b, Mixed-3c, MaxPool3d_4a, Mixed-4b, Mixed-4c, Mixed-4d, Mixed-4e, Mixed-4f, MaxPool3d_5a, Mixed-5b, and Mixed-5c; the Mixed layer includes Branch_0, Branch_1, Branch_1, and the residual module; the residual module is accessed by skipping between multiple serial Inception modules, and the residual module adopts the form of a bottleneck residual module to reduce the number of parameters and reduce the risk of overfitting; S3. Input the training sample set into the single-stream residual dilated 3D convolutional network and train it to obtain the optimal parameters of the single-stream residual dilated 3D convolutional network; S4. Input the verification sample set into the single-stream residual dilated 3D convolutional network and load the optimal parameters to detect and classify the authenticity of the target objects in the test sample set and the verification sample set, and obtain the classification result; S5. Based on the classification results, obtain the probability of determining the authenticity of the target object in the test sample set and the verification sample set.

2. The method for detecting fake videos according to claim 1, characterized in that, The video data preprocessing specifically includes the following steps: S11. Separate the video object into image objects frame by frame; S12. Locate the position of the target object from the frame images obtained by video framing, and crop the position of the target object in the frame images to obtain a target object image set; S13. Group the target object images of each consecutive first preset number of frames together as a set of frame images.

3. The method for detecting fake videos according to claim 2, characterized in that, The first preset quantity is 16.

4. The method for detecting fake videos according to claim 1, characterized in that, The objective function of the single-stream residual dilated 3D convolutional network is the cross-entropy function: in, Represents information entropy. This represents the probability obtained from the category prediction.

5. The method for detecting fake videos according to claim 1, characterized in that, The training method in step S3 specifically includes the following steps: S31. Initialize the parameters of the single-stream residual dilated 3D convolutional network; S32. Perform one round of training and adjust the parameters accordingly; S33. Conduct one round of testing and record the results of that round; S34. Repeat step S32 until convergence.

6. The method for detecting fake videos according to claim 5, characterized in that, Each training round reads 8 batch sizes of data, with 4 processes running in parallel and a training learning rate of 0.

001.

7. The method for detecting fake videos according to claim 2, characterized in that, Based on the classification results, the specific probabilities of determining the authenticity of the target object in the test sample set and the verification sample set are as follows: The set of frame images is sequentially input into the trained single-stream residual dilated 3D convolutional network, and the network outputs the comprehensive true probability for each frame image set. A probability higher than 50% is labeled as true, and a probability lower than 50% is labeled as false. The true / false determination results for all output frame image sets are statistically analyzed and compared with the labels, and the results are recorded. The probability of determining the authenticity of the target object is used as the final output.

Images