A deep fake video detection method based on dual fine-grained artifacts

Abstract

The present invention discloses a deep fake video detection method based on dual fine-grained artifacts, comprising: obtaining a true and false paired video data set; obtaining a position coordinate value sequence of a face area in a real video through a face detection model, and cropping the face area from the video in the true and false paired video data set to obtain a true and false paired face image sequence; obtaining dual fine-grained artifact annotations of fake face images from the true and false paired face image sequence; expanding the fake face images in the video data set using an online fake face data augmentation method; calculating a corresponding loss function based on the true and false face image annotation data and the output of a deep neural network, and training the deep fake detection model; the trained deep fake detection model detects deep fake face images to obtain detection results. The present invention significantly improves the generalization performance of the detector by revealing the dual fine-grained artifacts inherent in deep fake videos.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a deep fake video detection method based on dual fine-grained artifacts. Background Art

[0002] DeepFake technology can easily tamper with or even synthesize highly realistic digital media content such as sounds and images with the help of deep learning models and big data. In recent years, deep fake videos about human faces have been widely abused to create face-changing pornographic videos, spoof politicians, etc. This has brought serious threats to society. Therefore, the detection of deep fake videos is very important and urgent.

[0003] Most of the current major deep fake detection technologies are based on data-driven deep network model methods, such as MesoNet, Capsule, XceptionNet, MTD-Net, PRRNet, F3-NET, STIL, etc. These methods require a large number of labeled real videos and deep fake videos for supervised training and learning, and usually rely on specific training data. Once the test scene appears with a different distribution from the training data, and an unknown fake method generates a fake video, these methods are usually difficult to accurately identify its authenticity, which means that the detection effect of existing methods in real scenes will be greatly reduced.

[0004] Therefore, when faced with fake videos generated by unknown deep fake technology, the detection performance of existing technologies is greatly reduced, and the generalization performance needs to be improved and developed. Summary of the invention

[0005] In response to the above-mentioned defects of the existing methods, the present invention provides a deep fake video detection method, device, intelligent terminal and computer-readable storage medium based on dual fine-grained artifacts, aiming to solve the problem in the prior art that the detection performance is reduced due to data distribution differences, so as to improve the generalization of the detection model.

[0006] To achieve the above technical objectives, the technical solution of the present invention is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a deep fake video detection method based on dual fine-grained artifacts, the method comprising:

[0008] Obtain a dataset of real and fake paired videos;

[0009] Acquire a sequence of position coordinate values ​​of a face region in a real video through a face detection model, and crop the face region from the video in the true and false paired video data set to obtain a sequence of true and false paired face images;

[0010] Obtaining dual fine-grained artifact annotations of forged face images from the true and false paired face image sequences;

[0011] The video data set is trained using a deep neural network and video frames of different videos are randomly selected, and the fake face images in the video data set are expanded using an online fake face data augmentation method;

[0012] Calculate the corresponding loss function based on the dual fine-grained artifact annotation data of the true and false face images and the deep neural network output, and train the deep fake detection model;

[0013] Use the trained deep fake detection model to detect deep fake face images and obtain the detection results.

[0014] In one implementation, the step of obtaining a sequence of position coordinate values ​​of a face region in a real video by using a face detection model, and cropping the face region from the video of the true and false paired video data set to obtain a sequence of true and false paired face images specifically includes:

[0015] Converting the video in the data set into a sequence of image frames;

[0016] Based on the image frame sequence, using a face detection model to detect a face region in the image frame sequence of a real video, so as to obtain a position coordinate value sequence of the face region in the real image frame sequence;

[0017] Based on the position coordinate value sequence of the face region in the real image frame sequence, a corresponding face image sequence is cropped from the paired real and fake image frame sequences.

[0018] In one implementation, the step of obtaining dual fine-grained artifact annotations of forged face images from the sequence of paired true and false face images specifically includes:

[0019] After subtracting the corresponding pixels between the paired true and false face images, the absolute value is taken to obtain a difference map, which is then binarized using a predefined threshold to obtain a binary difference map. Finally, the convex hull of the binary difference map is calculated to obtain a labeled mask map of the intrinsic fine-grained artifacts.

[0020] The annotated mask map of the intrinsic fine-grained artifacts is subjected to morphological dilation and erosion operations to obtain an expanded intrinsic fine-grained artifact mask map and an eroded intrinsic fine-grained artifact mask map respectively. Finally, the expanded intrinsic fine-grained artifact mask map is subtracted from the eroded intrinsic fine-grained artifact mask map to obtain an annotated mask map of the extrinsic fine-grained artifacts.

[0021] The intrinsic fine-grained artifacts are caused by a series of common operations in the generative model (such as upconvolution or upsampling), and the extrinsic fine-grained artifacts are introduced by a common step in post-processing, which fuses the synthesized face with the original video. The dual fine-grained artifact annotated mask map includes the annotated mask map of the intrinsic fine-grained artifacts and the annotated mask map of the extrinsic fine-grained artifacts.

[0022] In one implementation, the step of training the video dataset using a deep neural network and randomly selecting video frames of different videos, and expanding the forged face images in the video dataset using an online forged face data augmentation method specifically includes:

[0023] Generate a new fake image by using a deep fake face image and a real face with similar face angle but not corresponding to it, and generate a new fake image by using a real face image and another real face with similar face angle. Use the two synthesis methods to expand the fake face images in the dataset;

[0024] In one implementation, the deep fake video detection method based on dual fine-grained artifacts, wherein the corresponding loss function is calculated based on the true and false face image annotation data and the deep neural network output to train the deep fake detection model:

[0025] The real and fake face images are input into the deep neural network to obtain the probability confidence output of the real and fake classification, as well as the mask outputs of the predicted internal fine-grained artifacts and external fine-grained artifacts. The output of the deep neural network and the annotated data are used to calculate the loss function, and the deep fake detection model is trained according to the loss function.

[0026] In one implementation, the deep forgery video detection method based on dual fine-grained artifacts, wherein the deep neural network specifically includes:

[0027] The true and false face images are input into the encoder of the deep neural network to obtain deep features; the deep features are then passed through a global pooling layer and a linear classification layer to obtain a probability confidence output of the true and false classification; at the same time, the deep features are passed through two parallel decoders to obtain the predicted mask map outputs of the intrinsic fine-grained artifacts and the extrinsic fine-grained artifacts respectively.

[0028] In one implementation, the deep fake video detection method based on dual fine-grained artifacts, wherein the loss function specifically includes:

[0029] For the probability confidence output of true and false classification, calculate the cross entropy loss function:

[0030] L Cls =y log(O Cls(x))+(1-y)log(1-O Cls (x))

[0031] Where x is the input face image sample, y is the category label of the sample, y = 0 means the input is a real face image, y = 1 means the input is a fake face image, Cls (x) represents the probability confidence of the true or false classification output by the deep neural network.

[0032] For the predicted mask outputs of intrinsic fine-grained artifacts and extrinsic fine-grained artifacts, the pixel-level cross entropy loss function is calculated:

[0033]

[0034]

[0035] Among them, In (x) and O Ex (x) represent the predicted mask images of the intrinsic fine-grained artifacts and extrinsic fine-grained artifacts output by the deep neural network, respectively, and M In and M Ex They respectively represent the annotated mask images of the intrinsic fine-grained artifacts and the extrinsic fine-grained artifacts, and i and j represent the pixel locations on the mask images.

[0036] A deep fake video detection device based on dual fine-grained artifacts, comprising:

[0037] An acquisition module is used to obtain a dataset of true and false paired videos;

[0038] A cropping processing module, used to obtain a sequence of position coordinate values ​​of a face region in a real video through a face detection model, and crop the face region from the video in the true and false paired video data set to obtain a sequence of true and false paired face images;

[0039] An annotation processing module, used for obtaining dual fine-grained artifact annotations of forged face images from the real and fake paired face image sequence;

[0040] An augmentation processing module, used to train the video data set using a deep neural network and randomly select video frames of different videos, and augment the forged face images in the video data set using an online forged face data augmentation method;

[0041] A training module, configured to calculate a corresponding loss function based on the dual fine-grained artifact annotation data of the true and false face images and the output of the deep neural network, and train the deep fake detection model;

[0042] The detection application module is used to detect deep fake face images using the trained deep fake detection model to obtain detection results.

[0043] An intelligent terminal includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, including the method for executing any one of the methods described above.

[0044] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods described above.

[0045] Beneficial effects: The present invention provides a deep fake video detection method, device, intelligent terminal and computer-readable storage medium based on dual fine-grained artifacts, which effectively improve the feature representation ability of the network by detecting dual fine-grained artifacts in deep fake face images. At the same time, the online fake face data augmentation method is used to expand the fake face images in the data set, which increases the diversity of fake samples and prompts the network to learn more discriminative features. The present invention uses dual fine-grained artifacts and fake face data augmentation to perform end-to-end deep fake model training. Without external intervention, the trained deep fake detection model can effectively alleviate the problem that the detection performance of the existing technology will be significantly reduced when facing large differences in data distribution, and greatly improves the generalization performance of the detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is an overall flow chart of a deep fake video detection method based on dual fine-grained artifacts provided by an embodiment of the present invention.

[0047] Figure 2 This is an effect diagram of a dual fine-grained artifact annotation example provided by an embodiment of the present invention.

[0048] Figure 3 The present invention provides a schematic diagram of a process for augmenting forged face data based on a forged face image and a real face image.

[0049] Figure 4 It is a schematic diagram of a process for augmenting forged face data by synthesizing two real face images provided by an embodiment of the present invention.

[0050] Figure 5 It is a schematic diagram of the structure of a deep fake detection model provided by an embodiment of the present invention.

[0051] Figure 6 It is a principle block diagram of a deep fake video detection method and device based on dual fine-grained artifacts provided by an embodiment of the present invention.

[0052] Figure 7It is a block diagram of the internal structure principle of the intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] Figure 1 Schematic diagram of the process of deep forgery detection method based on dual fine-grained artifacts described in a preferred embodiment of the present invention. Figure 1 As shown, a deep forgery detection method based on dual fine-grained artifacts according to an embodiment of the present invention specifically includes:

[0055] Step S100: Obtain a dataset of true and false paired videos.

[0056] In the embodiment of the present invention, specifically, the training of the deep fake detection model requires obtaining a real-fake paired video dataset. The present invention chooses to use the Celeb-DF deep fake video dataset, which contains 5639 high-quality deep fake videos and 890 corresponding original videos, and provides a pairing relationship between all deep fake videos and source real face videos. The dataset covers 59 identities of different genders, ages, etc. The average length of all videos is about 13 seconds, and the standard frame rate is 30 frames per second.

[0057] The Celeb-DF deep face extraction dataset of the present invention may include the Celeb-DF (v1) dataset and the Celeb-DF (v2), wherein the Celeb-DF (v2) dataset contains real and DeepFake synthetic videos, and its video quality is similar to that of videos circulated online. The Celeb-DF (v2) dataset is greatly expanded compared to the previous Celeb-DF (v1), which only contains 795 DeepFake videos. The Celeb-DF that can be used in the present invention includes 590 original videos collected from YouTube, which have subjects of different ages, races and genders, and 5639 corresponding DeepFake videos.

[0058] In the embodiment of the present invention, a true and false paired video data set is obtained to provide a basis for helping to identify AI face-swapped videos.

[0059] Step S200: obtaining a sequence of position coordinate values ​​of a face region in a real video through a face detection model, and cropping the face region from the video of the true and false paired video data set to obtain a sequence of true and false paired face images.

[0060] Specifically, after obtaining a dataset of true and false paired video data, the acquired video is first converted into an image frame sequence, and then a face detection model is used to detect the face area of ​​the image frame sequence of the real video to obtain a sequence of position coordinate values ​​of the face area in the real image frame sequence. Finally, the obtained sequence of position coordinate values ​​of the face area in the real image frame sequence is used to crop the corresponding face image sequence from the paired true and false image frame sequences.

[0061] Step S300: Obtain dual fine-grained artifact annotations of forged face images from the sequence of paired genuine and fake face images.

[0062] In the embodiment of the present invention, the paired true and false face images can be used (I f ,I r ) indicates that I f and I r They represent the deep fake face and its paired real face respectively. First, the corresponding pixels between the paired real and fake face images are subtracted, and then their absolute values ​​are taken to obtain the difference map p. Then, the difference map is binarized using a predefined threshold γ to obtain the binary difference map. Finally, the convex hull of the binary difference map is calculated to obtain the annotated mask map M of the intrinsic fine-grained artifacts. In , the specific conversion formula is:

[0063] M In =Convex(p>γ), where Convex represents the convex hull.

[0064] Specifically, through the annotated mask map M of the intrinsic fine-grained artifacts In The expansion operation and erosion operation in morphology are performed to obtain the expansion intrinsic fine-grained artifact mask map D(M In ) and the erosion intrinsic fine-grained artifact mask E(M In ), and finally, the expanded intrinsic fine-grained artifact mask map is subtracted from the eroded intrinsic fine-grained artifact mask map to obtain the annotated mask map M of the extrinsic fine-grained artifacts Ex , the specific conversion formula is:

[0065] M Ex =D(M In )-E(M In ), where D(M In ) represents the dilated intrinsic fine-grained artifact mask map; E(M In ) represents the erosion intrinsic fine-grained artifact mask map; M Ex Annotated mask map representing extrinsic fine-grained artifacts.

[0066] Specifically, the double fine-grained artifact annotation mask corresponding to the true and false face images is as follows: Figure 2 As shown. Figure 2The real face image above conforms to the natural imaging model, i.e., there is no double fine-grained artifact; Figure 2 The deep fake face image below has both intrinsic and extrinsic fine-grained artifacts.

[0067] Step S400: training the video data set using a deep neural network and randomly selecting video frames from different videos, and expanding the forged face images in the video data set using an online forged face data augmentation method.

[0068] Specifically, the present invention applies a fake face data augmentation method to expand the diversity of fake face images based on the real and fake paired video data set and the double fine-grained artifact annotated images in step S300, so as to help the model learn more discriminative features. A new fake face image synthesized based on a deep fake face image and an image of a real face with a similar face angle but not corresponding to the real face is represented as (fake, real), and a new fake face image synthesized based on a real face image and another real face image with a similar face angle is represented as (real, real). The fake face data augmentation method can realize online data augmentation and expansion of new fake face data during the training process, wherein the specific steps of synthesizing fake face images (fake, real) are as shown in the following algorithm 1, and the corresponding flow chart of synthesizing fake face images (fake, real) is as shown in Figure 3 As shown; in addition, the specific steps of synthesizing a forged face image (real, real) are as follows Algorithm 2, and the corresponding flowchart of synthesizing a forged face image (real, real) is as follows Figure 4 shown.

[0069] Algorithm 1 is as follows:

[0070] Input: A deep fake face image I f Another real face image I with a similar but not corresponding face angle r , and deep fake face image I f The annotated mask map M of the intrinsic fine-grained artifacts In ;

[0071] Output: A new fake face image I f ′, and the corresponding annotated mask map M of the intrinsic fine-grained artifacts and the extrinsic fine-grained artifacts In ′ and M Ex ′.

[0072] like Figure 3 As shown, step 1: Use the face detection library Dlib to extract I f The facial key points K(I f ) and I r The facial key points K(I r).

[0073] Step 2: Through K(I f ) Calculation I f The convex hull mask map M of the facial key points is obtained by applying an affine transformation to M to obtain the convex hull mask map T(M) of the facial key points after the affine transformation.

[0074] Step 3: Finally define the tampering mask map M′=M In ⊙T(M), ⊙ represents element-wise multiplication.

[0075] Step 4: Synthesize the new fake face image I f ′=M′⊙I f +(1-M′)⊙I r The corresponding intrinsic fine-grained artifact annotation mask is the tampering mask, i.e., M In ′=M′, external fine-grained artifact annotation mask M Ex ′=D(M In ′)-E(M In ′).

[0076] Step 5: Assume the end;

[0077] Step 6: Return a new forged face image I f ′, and I f The annotated mask map M of the intrinsic fine-grained artifacts and extrinsic fine-grained artifacts corresponding to ′ In ′ and M Ex ′.

[0078] Algorithm 2 is as follows:

[0079] Input: A real face image I r Another real face image I with a similar face angle r ′.

[0080] Output: A new fake face image I f ′, and the corresponding annotated mask map M of the intrinsic fine-grained artifacts and extrinsic fine-grained artifacts In ′ and M Ex ′.

[0081] like Figure 4 As shown, step 11: Use the face detection library Dlib to extract I r The facial key points K(I r ) and I r ′’s facial key points K(I r ′).

[0082] Step 12: Through K(I r ) Calculation I rThe convex hull mask map M of the facial key points is obtained by applying an affine transformation to M to obtain the convex hull mask map T(M) of the facial key points after the affine transformation.

[0083] Step 13: Synthesized New Fake Face Image I f ′=T(M)⊙I r +(1-T(M))⊙I r ′. The corresponding intrinsic fine-grained artifact annotation mask has values ​​of all 0, that is, M In ′ is a full 0 matrix, and the external fine-grained artifact annotation mask map M Ex ′=D(T(M))-E(T(M)).

[0084] Step 14: Assume the end;

[0085] Step 15: Return a new forged face image I f ′, and I f The annotated mask map M of the intrinsic fine-grained artifacts and extrinsic fine-grained artifacts corresponding to ′ In ′ and M Ex ′.

[0086] For example, the experiment of the present invention is trained on the training set of Celeb-DF, and tested on the test sets of UADFV, DF-TIMIT-LQ (DT-LQ), DF-TIMIT-HQ (DT-HQ), FaceForensics++DF (FF++DF), DFD, DFDC-P, Celeb-DF and other data sets. In the comparison of experimental results, AUC is used as the test index. The higher the AUC index, the better the performance. The results are shown in Table 1. Among them, the UADFV data set contains 49 real videos and 49 deep fake videos; the DT-LQ data set contains 320 real videos and 320 deep fake videos; the DT-HQ data set contains 320 real videos and 320 deep fake videos; FF++DF contains 1000 real videos and 1000 deep fake videos; DFD contains 363 real videos and 3068 deep fake videos; DFDC-P contains 1131 real videos and 4119 deep fake videos.

[0087] Table 1 AUC performance under different fake face data augmentation settings.

[0088]

[0089] Among them, it can be seen from Table 1 that the model trained by using both Algorithm 1 and Algorithm 2 for fake face data augmentation achieved the best average performance on all data sets. Compared with the model trained without data augmentation, the average AUC score increased by 5%, which greatly improved the generalization of the model.

[0090] That is, in this step, a deep neural network is used to train the video data set and video frames from different videos are randomly selected, and the fake face images in the video data set are expanded using an online fake face data augmentation method. Among them, the video data set is trained using the "true and false paired video data set" in step S100, and then a deep neural network is used during the training process to randomly select "two videos" for data expansion. The online expansion method is used, which does not generate a lot of data first and then train, but augments while training. The "two videos" here means selecting one frame from each of the two videos, and then synthesizing a new face. The process is as follows: Figure 3 and Figure 4 As shown. In this way, the generalization of the model can be improved. The specific experiment is a cross-domain detection experiment (cross-database detection). From the results in Table 1, it can be seen that the use of the proposed data augmentation improves the average AUC score by 5% compared with no data augmentation.

[0091] Step S500: Calculate a corresponding loss function based on the dual fine-grained artifact annotation data of the true and false face images and the deep neural network output to train the deep fake detection model.

[0092] In this step, the true and false face images are input into the deep neural network to obtain the probability confidence output of the true and false classification, as well as the mask outputs of the predicted internal fine-grained artifacts and external fine-grained artifacts. The output of the deep neural network and the annotated data are used to calculate the loss function, and the deep fake detection model is trained according to the loss function.

[0093] Specifically, the schematic diagram of the deep neural network model structure is as follows: Figure 5 As shown, Figure 5 is a schematic diagram of a deep fake detection model structure provided by an embodiment of the present invention, such as Figure 5 As shown, the true and false face images are input into the encoder of the deep neural network to obtain deep features. The encoder adopts the ResNet18 network structure as the backbone structure and includes five downsampling modules, such as Figure 5 The following are respectively downsampling module 1, downsampling module 2, downsampling module 3, downsampling module 4, and downsampling module 5; the deep features are then passed through the global pooling layer and the linear classification layer to obtain the probability confidence output of the true and false classification; at the same time, the deep features are passed through two parallel decoders to obtain the predicted mask outputs of the internal fine-grained artifacts and the external fine-grained artifacts. The two parallel decoders have the same architecture but different parameters. The decoder is composed of five upsampling modules. At the same time, there is a jump connection between the first four upsampling modules and the downsampling modules of the encoder at the same depth. Figure 5As shown, for example, the downsampling module 5 is connected to the upsampling module 1 above and another upsampling module 1 below; the downsampling module 4 is connected to the upsampling module 2 above and another upsampling module 2 below; the downsampling module 3 is connected to the upsampling module 3 above and another upsampling module 3 below; the downsampling module 2 is connected to the upsampling module 4 above and another upsampling module 4 below; finally, the output features of the decoder are passed through a convolution layer with a convolution kernel size of 1*1 and a sigmoid activation function to obtain a predicted dual fine-grained artifact mask map. The two branches of dual-grained artifact prediction are intended to improve the performance of the main branch encoder. By training the three branches end-to-end simultaneously, the backbone network of the encoder can be helped to learn more discriminative features.

[0094] Specifically, the loss functions calculated by training the deep fake detection neural network include the classification of real and fake face images, and the localization of dual fine-grained artifact masks at the pixel level.

[0095] Specifically, for the classification of true and false face images, the cross entropy loss function is calculated through the probability confidence output of the true and false classification:

[0096] L Cls =y log(O Cls (x))+(1-y)log(1-O Cls (x))

[0097] Where x is the input face image sample, y is the category label of the sample, y = 0 means the input is a real face image, y = 1 means the input is a fake face image, Cls (x) represents the probability confidence of the true or false classification output by the deep neural network.

[0098] Specifically, for pixel-level dual fine-grained artifact mask localization, the predicted mask map outputs of the intrinsic fine-grained artifacts and the extrinsic fine-grained artifacts are used to calculate the pixel-level cross entropy loss function:

[0099]

[0100]

[0101] Among them, In (x) and O Ex (x) represent the predicted mask images of the intrinsic fine-grained artifacts and extrinsic fine-grained artifacts output by the deep neural network, respectively, and M In and M Ex They respectively represent the annotated mask images of the intrinsic fine-grained artifacts and the extrinsic fine-grained artifacts, and i and j represent the pixel locations on the mask images.

[0102] Specifically, the entire loss function is finally defined as follows:

[0103] L=λ In L In +λ Ex L Ex +λ Cls L Cls

[0104] The weight coefficient λ in the loss function Cls , In , Ex They are set to 1, 15, and 15 respectively. End-to-end training is performed using the deep fake detection model.

[0105] Step S600: Use the trained deep fake detection model to detect the deep fake face image to obtain the detection result.

[0106] Finally, the deep fake detection model of this embodiment is trained on the training set of Celeb-DF, tested on the test sets of different data sets, and compared with the existing deep fake detection methods (XceptionNet, MesoNet, Capsule, Face X-ray, GSRM and RFM). The results are shown in Table 2.

[0107] Table 2. AUC performance compared with existing deepfake detection methods.

[0108] method UADFV DT-LQ DT-HQ FF++DF DFD DFDC-P Celeb-DF average XceptionNet 0.9610 0.9550 0.6539 0.7551 0.7706 0.7401 0.9985 0.8334 MesoNet 0.7750 0.8133 0.5609 0.7296 0.6707 0.7674 0.9242 0.7487 Capsule 0.8751 0.8519 0.6162 0.7219 0.6676 0.7056 0.9900 0.7754 Face X-ray 0.9305 0.9899 0.8917 0.8185 0.8253 0.7535 0.9984 0.8868 GSRM 0.9645 0.9709 0.6749 0.8029 0.8048 0.7924 0.9962 0.8580 RFM 0.9110 0.9569 0.6582 0.8288 0.7870 0.7678 0.9973 0.8439 Ours 0.9404 0.9952 0.9377 0.8523 0.8185 0.8189 0.9980 0.9087

[0109] Among them, it can be seen from Table 2 that compared with these methods, the deep fake detection method based on dual-granularity artifacts proposed in the present invention shows superior performance in cross-dataset scenarios, that is, it reaches an average AUC score of 0.9087 on all datasets, which is at least 2% better than other methods and has excellent generalization performance.

[0110] The present invention effectively improves the feature representation capability of the network by detecting dual fine-grained artifacts in deep fake face images. At the same time, the online fake face data augmentation method is used to expand the fake face images in the data set, which increases the diversity of fake samples and prompts the network to learn more discriminative features. The present invention uses dual fine-grained artifacts and fake face data augmentation to perform end-to-end deep fake model training. Without external intervention, the trained deep fake detection model can effectively alleviate the problem that the detection performance of the existing technology will be significantly reduced when facing large differences in data distribution, greatly improving the generalization performance of the detection model.

[0111] Exemplary Devices

[0112] like Figure 6As shown in , an embodiment of the present invention provides a deep fake video detection device based on dual fine-grained artifacts, the device comprising:

[0113] An acquisition module 510 is used to acquire a true and false paired video data set;

[0114] A cropping processing module 520 is used to obtain a sequence of position coordinate values ​​of a face region in a real video through a face detection model, and to crop the face region from the video of the true and false paired video data set to obtain a sequence of true and false paired face images;

[0115] Annotation processing module 530, used for obtaining dual fine-grained artifact annotations of forged face images from the sequence of true and false paired face images;

[0116] An augmentation processing module 540 is used to train the video data set using a deep neural network and randomly select video frames of different videos, and augment the forged face images in the video data set using an online forged face data augmentation method;

[0117] A training module 550 is used to calculate a corresponding loss function based on the dual fine-grained artifact annotation data of the true and false face images and the deep neural network output, and train the deep fake detection model;

[0118] The detection application module 560 is used to detect deep fake face images using the trained deep fake detection model to obtain detection results, as described above.

[0119] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 7 As shown. The smart terminal includes a processor, a memory, a network interface, a display screen, and a sensor connected through a system bus. Among them, the processor of the smart terminal is used to provide computing and control capabilities. The memory of the smart terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the smart terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a deep fake video detection method based on dual fine-grained artifacts is implemented. The display screen of the smart terminal can be a liquid crystal display screen or an electronic ink display screen.

[0120] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations:

[0122] Obtain a dataset of real and fake paired videos;

[0123] Acquire a sequence of position coordinate values ​​of a face region in a real video through a face detection model, and crop the face region from the video in the true and false paired video data set to obtain a sequence of true and false paired face images;

[0124] Obtaining dual fine-grained artifact annotations of forged face images from the true and false paired face image sequences;

[0125] The video data set is trained using a deep neural network and video frames of different videos are randomly selected, and the fake face images in the video data set are expanded using an online fake face data augmentation method;

[0126] Calculate the corresponding loss function based on the dual fine-grained artifact annotation data of the true and false face images and the deep neural network output, and train the deep fake detection model;

[0127] The deep fake face image is detected using the trained deep fake detection model to obtain the detection results, as described above.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] In summary, the present invention discloses a deep fake video detection method based on dual fine-grained artifacts, the detection method comprising: obtaining a true and false paired video data set; obtaining a position coordinate value sequence of the face area in the real video through a face detection model, and cropping the face area from the video of the true and false paired video data set to obtain a true and false paired face image sequence; obtaining a dual fine-grained artifact annotation of the forged face image from the true and false paired face image sequence; training the video data set using a deep neural network and randomly selecting video frames of different videos, and expanding the forged face images in the video data set using an online forged face data augmentation method; calculating the corresponding loss function according to the true and false face image annotation data and the output of the deep neural network, and training the deep fake detection model; the trained deep fake detection model detects the deep fake face image to obtain the detection result. Compared with the prior art, the present invention detects the inherent artifacts left by the deep fake video, and uses two data augmentation algorithms for forged face synthesis, thereby effectively improving the detection generalization performance of the model when there is a difference in the distribution of training data and test data.

[0130] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep fake video detection method based on dual fine-grained artifacts, characterized in that: The detection method comprises: Obtain a dataset of real and fake paired videos; Acquire a sequence of position coordinate values ​​of a face region in a real video through a face detection model, and crop the face region from the video in the true and false paired video data set to obtain a sequence of true and false paired face images; Obtaining dual fine-grained artifact annotations of forged face images from the true and false paired face image sequences; The video data set is trained using a deep neural network and video frames of different videos are randomly selected, and the fake face images in the video data set are expanded using an online fake face data augmentation method; Calculate the corresponding loss function based on the dual fine-grained artifact annotation data of the true and false face images and the deep neural network output, and train the deep fake detection model; Use the trained deep fake detection model to detect deep fake face images and obtain the detection results.

2. The deep fake video detection method based on dual fine-grained artifacts according to claim 1 is characterized in that: The steps of obtaining a sequence of position coordinate values ​​of a face region in a real video by using a face detection model, and cutting out the face region from the video of the true and false paired video data set to obtain a sequence of true and false paired face images include: Converting the video in the data set into a sequence of image frames; Based on the image frame sequence, using a face detection model to detect a face region in the image frame sequence of a real video, so as to obtain a position coordinate value sequence of the face region in the real image frame sequence; Based on the position coordinate value sequence of the face region in the real image frame sequence, a corresponding face image sequence is cropped from the paired real and fake image frame sequences.

3. The deep fake video detection method based on dual fine-grained artifacts according to claim 1 is characterized in that: The step of obtaining dual fine-grained artifact annotations of forged face images from the real and fake paired face image sequence comprises: Subtract the corresponding pixels between the pairs of true and false face images, and take their absolute values ​​to obtain the difference map; A predefined threshold is used to binarize the difference map to obtain a binarized difference map, and finally the convex hull of the binarized difference map is calculated to obtain a mask map of the intrinsic fine-grained artifacts; Performing a dilation operation and an erosion operation in morphology on the annotated mask map of the intrinsic fine-grained artifacts to obtain an expanded intrinsic fine-grained artifact mask map and an eroded intrinsic fine-grained artifact mask map respectively, and finally calculating the expanded intrinsic fine-grained artifact mask map minus the eroded intrinsic fine-grained artifact mask map to obtain an annotated mask map of the extrinsic fine-grained artifacts; Among them, intrinsic fine-grained artifacts are caused by a series of common operations in the generative model (such as upconvolution or upsampling), and extrinsic fine-grained artifacts are introduced by a common step in post-processing, which fuses the synthesized face with the original video; the dual fine-grained artifact annotated mask map includes the annotated mask map of intrinsic fine-grained artifacts and the annotated mask map of extrinsic fine-grained artifacts.

4. The deep fake video detection method based on dual fine-grained artifacts according to claim 1 is characterized in that: The step of training the video data set using a deep neural network and randomly selecting video frames of different videos, and expanding the forged face images in the video data set using an online forged face data augmentation method comprises: A new forged image is generated by a deep forged face image and a real face with a similar face angle but not corresponding to it, and a new forged image is generated by a real face image and another real face with a similar face angle; the forged face images in the dataset are expanded by the two synthesis methods.

5. The deep fake video detection method based on dual fine-grained artifacts according to claim 1 is characterized in that: The corresponding loss function is calculated based on the true and false face image annotation data and the deep neural network output to train the deep fake detection model: The genuine and fake face images are input into a deep neural network to obtain a probability confidence output for the genuine and fake classification, as well as two mask image outputs for predicting internal fine-grained artifacts and external fine-grained artifacts; and the loss function is calculated using the output of the deep neural network and the labeled data, and the deep fake detection model is trained according to the loss function.

6. The deep fake video detection method based on dual fine-grained artifacts according to claim 5 is characterized in that: The deep neural network specifically includes: The true and false face images are input into the encoder of the deep neural network to obtain deep features; the deep features are then passed through a global pooling layer and a linear classification layer to obtain a probability confidence output of the true and false classification; at the same time, the deep features are passed through two parallel decoders to obtain the predicted mask map outputs of the intrinsic fine-grained artifacts and the extrinsic fine-grained artifacts respectively.

7. The deep fake video detection method based on dual fine-grained artifacts according to claim 5 is characterized in that: The loss function specifically includes: For the probability confidence output of true and false classification, calculate the cross entropy loss function: L Cls ylog(O Cls (x))+(1-y)log(1-O). Cls (x)) Where x is the input face image sample, y is the category label of the sample, y = 0 means the input is a real face image, y = 1 means the input is a fake face image, Cls (x) represents the probability confidence of the true or false classification output by the deep neural network; For the predicted mask outputs of intrinsic fine-grained artifacts and extrinsic fine-grained artifacts, the pixel-level cross entropy loss function is calculated: Among them, In (x) and O Ex (x) represent the predicted mask images of the intrinsic fine-grained artifacts and extrinsic fine-grained artifacts output by the deep neural network, respectively, and M In and M Ex They respectively represent the annotated mask images of the intrinsic fine-grained artifacts and the extrinsic fine-grained artifacts, and i and j represent the pixel locations on the mask images.

8. A deep fake video detection device based on dual fine-grained artifacts, characterized in that: include: An acquisition module is used to obtain a dataset of true and false paired videos; A cropping processing module, used to obtain a sequence of position coordinate values ​​of a face region in a real video through a face detection model, and crop the face region from the video in the true and false paired video data set to obtain a sequence of true and false paired face images; An annotation processing module, used for obtaining dual fine-grained artifact annotations of forged face images from the real and fake paired face image sequence; An augmentation processing module, used to train the video data set using a deep neural network and randomly select video frames of different videos, and augment the forged face images in the video data set using an online forged face data augmentation method; A training module, used to calculate a corresponding loss function based on the dual fine-grained artifact annotation data of the true and false face images and the output of the deep neural network, and train the deep fake detection model; The detection application module is used to detect deep fake face images using the trained deep fake detection model to obtain detection results.

9. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in any one of claims 1 to 7.

Images