Face tampering video detection method and system based on dual-stream contrast learning model

Through the dual-stream comparison learning model, the problem of poor cross-store performance of face tamper detection in the existing technology is solved, and the high detection accuracy and cross-store detection capability are improved, and the face tampering is solved. Detect the problem of the model being overly reliant on the training dataset.

CN115116108BActive Publication Date: 2024-11-26SUN YAT SEN UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210447080.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-26
Publication Date
2024-11-26
Estimated Expiration
2042-04-26

AI Technical Summary

Technical Problem

The existing end-to-end binary face tamper detection method lacks the support for face tamper priors, and the cross-store detection performance is poor and over-reliance on the training dataset.

Method used

Using the face tamper video detection method based on the dual-stream contrast learning model, the face tamper prior knowledge is injected by designing the semantic segmentation network, and two semantic segmentation network branches with the same structure are used for information collaborative learning to alleviate excessive dependence on training data. The disadvantages of collection improve cross-store detection capabilities.

Benefits of technology

While maintaining high detection accuracy in the library, it significantly improves cross-store detection capabilities, can effectively identify tampered videos of unknown methods, and improves the robustness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115116108B_ABST
    Figure CN115116108B_ABST
Patent Text Reader

Abstract

The present invention proposes a method and system for detecting face tampering in videos based on a dual-stream contrast learning model, which relates to the technical fields of computer vision, deep learning, and information security. Each video is framed, face detection is performed on a single-frame image, and a face image containing a background area is cropped out to detect the authenticity of the single-frame image. The discrimination results of the single-frame image are statistically analyzed to obtain video-level detection results. In order to inject more prior knowledge of face tampering into the training of the model, a dual-stream contrast learning model is designed from the perspective of integrating prior knowledge of face tampering. The dual-stream contrast learning model includes two semantic segmentation network branches with the same structure, which ensures that each branch makes adjustments by taking into account the information of another stream when extracting and learning its own information stream, thereby alleviating the disadvantage of traditional binary classification detection algorithms that are overly dependent on training data sets, maintaining a high in-library detection accuracy, and effectively improving the model's cross-library detection capability for unknown methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, deep learning and network security, and more specifically, to a method and system for detecting face tampering in videos based on a dual-stream contrast learning model. Background Art

[0002] In recent years, with the popularity of various digital media editing software, tampering with digital media has become rampant, reducing the originality and reliability of digital media content. A type of deep fake technology called Deepfakes has attracted widespread attention from the society due to its realistic video face-changing effect. Using this technology to generate fake videos can cause serious harm. Therefore, it is very necessary to develop and research detection methods for such fake face-changing video media.

[0003] The prior art discloses an image tampering detection method, in which an untampered image and a tampered image are first prepared, and the untampered image and the tampered image are combined to form a training set, and then a binary classification network model is constructed, and the binary classification network model is trained through the training set, so that the trained binary classification network model can distinguish whether the image has been tampered with; the binary classification network model includes a feature extraction layer, an image attention layer and a classifier, and finally the trained binary classification network model is used to identify whether the image has been tampered with. In this technical solution, a neural network is used to train a binary classification network to guide the network to obtain an accurate tampering detection probability value, but the solution ultimately models face tampering detection as an end-to-end binary classification problem. This end-to-end binary classification model may cause the trained model to be overly dependent on the training data set and have poor cross-database detection performance during network training due to the lack of prior support for face tampering. Summary of the invention

[0004] In order to solve the problem that the existing end-to-end binary classification face tampering detection method has poor cross-database detection performance when there is a lack of support for face tampering prior, the present invention proposes a face tampering video detection method and system based on a dual-stream contrast learning model, which injects more face tampering prior knowledge, alleviates the disadvantage of deep learning networks' over-reliance on training data sets, and effectively improves the cross-database detection capability while maintaining a high in-database detection accuracy.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] A method for detecting face tampering in video based on a dual-stream contrast learning model, comprising the following steps:

[0007] S1. Perform a frame operation on the original video to obtain an original frame image;

[0008] S2. Perform a face tampering operation on the original framed image to obtain a face tampered image, mix the face tampered image and the original framed image to form a data set, and divide the data set into a training set, a validation set, and a test set;

[0009] S3. Perform face detection on each single frame image in the data set and crop the face image containing the background area;

[0010] S4. locate facial key points of the input face image and generate a pixel-level mask;

[0011] S5. Construct a two-stream contrastive learning model, including two semantic segmentation network branches with the same structure; use the training set to supervise the two-stream contrastive learning model, then use the validation set to evaluate the two-stream contrastive learning model, select the optimal detection model, and use the test set to test the effectiveness of the optimal detection model;

[0012] S6. Obtain the video to be detected, perform frame operations on it, and obtain a face image in the manner of step S2, and then obtain a pixel-level mask for the face image in the manner of step S4;

[0013] S7. Input the face video frame to be detected and the pixel-level mask generated in step S6 into the optimal detection model for classification, and obtain the classification results of all face video frames to be detected;

[0014] S8. Based on the classification and discrimination results of all face video frames to be detected, the overall video-level prediction classification result is obtained.

[0015] In this technical solution, face tampering detection is achieved by designing a semantic segmentation network. Different from the end-to-end binary classification supervision mode of most existing detection networks, this technical solution provides a new paradigm of pixel-level supervision. Each video is framed, face detection is performed on a single frame image, and the face image containing the background area is cropped out to detect the authenticity of the single frame image. The video-level detection results are obtained by counting the single frame image discrimination results. In order to inject more prior knowledge of face tampering into the training of the model, a two-stream contrast learning model is designed from the perspective of integrating prior knowledge of face tampering. The two-stream contrast learning model contains two semantic segmentation network branches with the same structure, which ensures that each branch makes adjustments by taking into account the information of the other stream when extracting and learning its own information flow, thereby alleviating the disadvantage of the traditional binary classification detection algorithm that over-relies on the training data set, maintaining a high in-library detection accuracy, and effectively improving the model's cross-library detection capability for unknown methods.

[0016] Preferably, the process of performing the face tampering operation on the original framed image in step S2 is specifically as follows:

[0017] S21. Match the framed images of different face identities in the original framed images in pairs, and use them as source material and target material respectively;

[0018] S22. Perform face tampering operations on the source material and the target material using a face tampering method to obtain a face tampering image. The face tampering method includes a Deepfakes method and a FaceSwap method.

[0019] Preferably, in step S3, the process of performing face detection on each single frame image in the data set after the frame division operation and cutting out the face image containing the background area is specifically as follows:

[0020] S31. Perform face detection on each single frame image in the data set after the frame operation to obtain the original face cropping frame;

[0021] S32. Fix the center position of the face cropping frame, and enlarge the face cropping frame by 1:1.3 to obtain the enlarged face cropping frame;

[0022] S33. Based on the enlarged face cropping frame, perform region cropping on the single frame image to obtain a face image including the background region.

[0023] Here, after enlarging the face cropping frame in each single-frame image, the enlarged face cropping frame is used as a reference to perform region cropping on the single-frame image to obtain a face image including the background area. The included background area tends to be untampered with, and the untampered background area is used as prior knowledge to design a semantic segmentation network.

[0024] Preferably, the pixel-level mask in step S4 includes a face mask and a background mask, and the process of locating the key points of the face on the input single-frame image and generating the pixel-level mask is as follows:

[0025] S41. Determine the number q of facial key points, and obtain the positioning coordinate values ​​of q facial key points;

[0026] S42. Perform a convex hull algorithm on the q facial key points to obtain a closed U-shaped facial region mask, that is, a facial mask;

[0027] S43. Perform a logical complement operation on the face mask to obtain a background mask.

[0028] Preferably, the dual-stream contrast learning model constructed in step S5 includes two semantic segmentation network branches with the same structure, one is an RGB stream branch, and the other is a gradient stream branch; the semantic segmentation network includes two basic parts: one part is an encoder built with the EfficientNet-b3 network as the backbone, and the trained network weight parameters of the network on ImageNet are loaded as the pre-trained network weight parameters of the encoder; the other part is a shallow decoder obtained by alternately stacking a bilinear upsampling module and a normal convolution module, and the weight parameters of the decoder network of this part are preset by using Kaiming initialization.

[0029] Preferably, step S5 also includes constructing a perspective generator for receiving video frames input thereto and generating two RGB images under different perspectives. The perspective generator integrates four perspective transformation methods, namely: Gaussian blur, Gaussian noise, grayscale and SRM noise superposition.

[0030] Preferably, the training set is used to supervise the dual-stream contrast learning model, and then the validation set is used to evaluate the dual-stream contrast learning model, the optimal detection model is selected, and the process of testing the optimal detection model using the test set is as follows:

[0031] S51. Input the training set into the view generator to obtain two RGB images under different view angles, select one of the RGB images as RGB stream data, and perform gradient filtering on the other RGB image to obtain gradient stream data;

[0032] S52. Input the RGB stream data and gradient stream data obtained in S51 into the RGB stream branch and gradient stream branch of the dual-stream contrast learning model for feature extraction, and obtain feature maps F of two different flow domains. s and F g ;

[0033] S53. Use the background mask obtained in S43 to perform mask average pooling on the feature map to obtain the background feature prototypes p of two different watersheds s and p g , and add a gradient blocking mechanism after mask average pooling to prevent gradient backpropagation;

[0034] The mask average pooling formula is expressed as:

[0035]

[0036] Among them, p c represents the output background prototype, i and j represent the pixel position index, c is the watershed category, M bg is the background mask, F cis the characteristic output of the corresponding flow domain, s and g represent the spatial flow and gradient flow respectively;

[0037] S54. Feature map F for two different watersheds s and F g Perform domain adaptive mapping to obtain the mapped feature map and The feature matching formula is used to cross-match the background prototypes between different watersheds to obtain the initial semantic segmentation map O. Each pixel position of the initial semantic segmentation map is normalized to obtain the normalized probability segmentation map output O. new ; Among them, the feature matching formula is:

[0038]

[0039] O represents the initial segmentation output, p g and p s Represents the background prototype vectors of different watersheds, and represents the feature map output after the domain adaptation layer, and δ(·,·) represents the cosine similarity metric function; the normalization formula is:

[0040]

[0041] S55. Design the total loss function of the two-stream contrast learning model, including the background area loss function and the face area loss function, where the background area loss function is expressed as:

[0042]

[0043] Among them, M bg represents the background mask, O new represents the segmentation output image after probability normalization; the facial area loss function expression is:

[0044]

[0045] Among them, M fg represents the face mask, I represents the input image; the total loss function is:

[0046] L seg =L bg +L fg ,

[0047] S56. Update the network parameters of the two-stream contrastive learning model using gradient descent;

[0048] S57. Introduce different training hyperparameters, and repeat steps S51 to S56 using different training hyperparameters to obtain several dual-stream comparative learning training models. Use the validation set to evaluate the several dual-stream comparative learning training models, select the optimal dual-stream comparative learning training model as the trained dual-stream comparative learning model, and use the test set to test the effectiveness of the optimal detection model.

[0049] Here, based on the two branches of the two-stream contrastive learning model, the background mask obtained in S43 is used to perform mask average pooling on the feature map to obtain background feature prototypes of two different watersheds. The background feature prototype represents the untampered area. Based on this, a scan is performed on the feature map of the original image to confirm the tampered area.

[0050] Preferably, in step S7, S51 to S54 are executed on the face video frame to be detected to obtain a semantic segmentation output map O test , and use the region scoring formula to get the probability of non-tampering p um , the specific regional scoring formula is:

[0051]

[0052] For the probability p of non-tampering um Perform probability complement operation to obtain the predicted tampering probability p m , the expression is: m =1-p um ; Set the discrimination threshold to p th , the tampering probability p m Compare it with the discrimination threshold to obtain the tampering discrimination classification result of the face video frame to be detected.

[0053] Preferably, if p m ≥p th , then it is determined that the current face video frame to be detected has been tampered with by the face; if p m <p th , the current video frame has not been tampered with by the face. By confirming the position and number of video frames that have been judged to have been tampered with by the face in all video frames, the video-level tampering discrimination prediction result is obtained.

[0054] The present application also proposes a face tampering video detection system based on a dual-stream contrast learning model, the system comprising:

[0055] A frame processing module is used to perform frame operations on the original video to obtain original frame images;

[0056] A data set construction module is used to perform a face tampering operation on the original framed image to obtain a face tampering image, the face tampering image and the original framed image are mixed to form a data set, and the data set is divided into a training set, a validation set and a test set;

[0057] The face image processing module performs face detection on each single frame image in the data set and crops the face image containing the background area;

[0058] The mask generation module locates the key points of the input face image and generates a pixel-level mask;

[0059] The model building training module is used to build a two-stream contrastive learning model, including two semantic segmentation network branches with the same structure; the two-stream contrastive learning model is supervised and trained using the training set, and then the two-stream contrastive learning model is evaluated using the validation set to select the optimal detection model, and the optimal detection model is tested using the test set;

[0060] The video preprocessing module to be detected is used to perform frame division operation on the video to be detected, and obtain a face image, and obtain a pixel-level mask based on the face image;

[0061] The detection module is used to input the face video frames to be detected and the pixel-level masks generated by the video preprocessing module to be detected into the optimal detection model for classification, so as to obtain the classification and discrimination results of all face video frames to be detected, and obtain the overall video-level predicted classification results based on the classification and discrimination results of all face video frames to be detected.

[0062] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0063] The present invention proposes a method and system for detecting face tampering in videos based on a dual-stream contrast learning model, provides a new paradigm of supervision based on pixel level, divides each video into frames, performs face detection on a single-frame image and crops out the face image containing the background area, performs authenticity detection on the single-frame image, and statistically identifies the video-level detection results. In order to inject more prior knowledge of face tampering into the training of the model, a dual-stream contrast learning model is designed from the perspective of integrating prior knowledge of face tampering. The dual-stream contrast learning model includes two semantic segmentation network branches with the same structure, which ensures that each branch makes adjustments by taking into account the information of another stream when extracting and learning its own information stream, thereby alleviating the disadvantage of the traditional two-classification detection algorithm that over-relies on the training data set, maintaining a high in-library detection accuracy, and effectively improving the model's cross-library detection capability for unknown methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1A schematic diagram showing a flow chart of a method for detecting face tampering in video based on a dual-stream contrast learning model proposed in Embodiment 1 of the present invention;

[0065] Figure 2 A flowchart showing the process of obtaining the face mask and the background mask proposed in Embodiment 1 of the present invention;

[0066] Figure 3 A structural framework diagram showing the dual-stream contrastive learning model proposed in Example 1 of the present invention.

[0067] Figure 4 A structural diagram of a face tampering video detection system based on a dual-stream contrast learning model proposed in Example 3 of the present invention. DETAILED DESCRIPTION

[0068] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0069] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0070] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0071] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0072] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent;

[0073] Example 1

[0074] In this embodiment, if Figure 1 As shown in the figure, a face tampering video detection method based on a two-stream contrast learning model is proposed, which includes the following steps:

[0075] S1. Perform a frame operation on the original video to obtain an original frame image;

[0076] S2. Perform a face tampering operation on the original framed image to obtain a face tampered image, mix the face tampered image and the original framed image to form a data set, and divide the data set into a training set, a validation set, and a test set;

[0077] The specific process of performing face tampering operation on the original frame image is as follows:

[0078] S21. Match the framed images of different face identities in the original framed images in pairs, and use them as source material and target material respectively;

[0079] S22. Perform face tampering operations on the source material and the target material using a face tampering method to obtain a face tampering image. The face tampering method includes a Deepfakes method and a FaceSwap method.

[0080] Here, the training set, validation set, and test set are randomly divided according to a certain ratio.

[0081] S3. Perform face detection on each single frame image in the data set and crop the face image containing the background area;

[0082] The process of performing face detection on each single frame image in the data set after the frame operation and cropping the face image containing the background area is as follows:

[0083] S31. Perform face detection on each single frame image in the data set after the frame operation to obtain the original face cropping frame;

[0084] S32. Fix the center position of the face cropping frame, and enlarge the face cropping frame by 1:1.3 to obtain the enlarged face cropping frame;

[0085] S33. Based on the enlarged face cropping frame, perform region cropping on the single frame image to obtain a face image including the background region.

[0086] In this embodiment, the ffmpeg video processing software package is used to perform a frame operation on the video, and then the MTCNN is used to identify the face cropping frame of the single frame image. Since the directly identified face frame only contains a small part of the face area, in this embodiment, the algorithm fixes the center position of the face cropping frame and enlarges the face cropping frame at 1:1.3, and then uses the enlarged cropping frame to perform a cropping operation on the original video frame to obtain a cropped image containing a small amount of background area, which is referred to as a face image.

[0087] S4. locate facial key points of the input face image and generate a pixel-level mask;

[0088] The pixel-level mask includes the face mask and the background mask, and locates the key points of the face in the input single-frame image, such as Figure 2 As shown, the process of generating pixel-level masks is:

[0089] S41 determines the number q of facial key points. In this embodiment, q is 68, that is, the positioning coordinate values ​​of 68 facial key points are obtained;

[0090] S42. Perform a convex hull algorithm on the 68 facial key points to obtain a closed U-shaped facial region mask, that is, a facial mask;

[0091] S43. Perform a logical complement operation on the face mask to obtain a background mask.

[0092] S5. Construct a two-stream contrastive learning model, including two semantic segmentation network branches with the same structure; use the training set to supervise the two-stream contrastive learning model, then use the validation set to evaluate the two-stream contrastive learning model, select the optimal detection model, and use the test set to test the effectiveness of the optimal detection model;

[0093] In this process, see Figure 3 The constructed two-stream contrastive learning model includes two semantic segmentation network branches with the same structure, one is the RGB stream branch and the other is the gradient stream branch; the semantic segmentation network includes two basic parts: one is the encoder built with the EfficientNet-b3 network as the backbone, and the trained network weight parameters of the network on ImageNet are loaded as the pre-trained network weight parameters of the encoder; the other part is a shallow decoder obtained by alternately stacking bilinear upsampling modules and ordinary convolution modules, and the weight parameters of the decoder network of this part are preset by using Kaiming initialization.

[0094] Step S5 also includes constructing a perspective generator for receiving video frames input thereto and generating two RGB images under different perspectives. The perspective generator integrates four perspective transformation methods, namely: Gaussian blur, Gaussian noise, grayscale, and SRM noise superposition.

[0095] The process of using the training set to supervise the two-stream contrastive learning model, then using the validation set to evaluate the two-stream contrastive learning model, selecting the optimal detection model, and using the test set to test the optimal detection model is as follows:

[0096] S51. Input the training set into the view generator to obtain two RGB images under different view angles, select one of the RGB images as RGB stream data, and perform gradient filtering on the other RGB image to obtain gradient stream data;

[0097] S52. Input the RGB stream data and gradient stream data obtained in S51 into the RGB stream branch and gradient stream branch of the dual-stream contrast learning model for feature extraction, and obtain feature maps F of two different flow domains. s and F g ;

[0098] After obtaining the characteristic maps F of two different watersheds s and F g Finally, based on the prior knowledge that the background area of ​​the face image always tends to be untampered, a background prototype cross-matching algorithm is designed to obtain the semantic segmentation output. The formula for background prototype cross-matching is:

[0099]

[0100] Among them, O represents the initial segmentation output, p g and p s Represents the background prototype vectors of different watersheds, and represents the feature map output after the domain adaptation layer, δ(·,·) represents the cosine similarity metric function, and then the normalized segmentation output map O is obtained through the probability normalization operation. new Based on the two branches of the two-stream contrastive learning model, the background mask obtained by S43 is used to perform mask average pooling on the feature map to obtain background feature prototypes of two different watersheds. The background feature prototype can express the abstract semantic category of untampered. Based on this background feature prototype, pixel-by-pixel position matching is performed on the feature map of the original image to confirm the untampered area.

[0101] Specifically, the steps S53 to S54 are as follows:

[0102] S53. Use the background mask obtained in S43 to perform mask average pooling on the feature map to obtain the background feature prototypes p of two different watersheds s and p g , and add a gradient blocking mechanism after mask average pooling to prevent gradient backpropagation;

[0103] The mask average pooling formula is expressed as:

[0104]

[0105] Among them, p c represents the output background prototype, i and j represent the pixel position index, c is the watershed category, M bg is the background mask, F c is the characteristic output of the corresponding flow domain, s and g represent the spatial flow and gradient flow respectively;

[0106] S54. Feature map F for two different watersheds s and F g Perform domain adaptive mapping to obtain the mapped feature map and The feature matching formula is used to cross-match the background prototypes between different watersheds to obtain the initial semantic segmentation map O. Each pixel position of the initial semantic segmentation map is normalized to obtain the normalized probability segmentation map output O. new ; Among them, the feature matching formula is:

[0107]

[0108] O represents the initial segmentation output, pg and p s Represents the background prototype vectors of different watersheds, and represents the feature map output after the domain adaptation layer, and δ(·,·) represents the cosine similarity metric function; the normalization formula is:

[0109]

[0110] S55. Design the total loss function of the two-stream contrast learning model, including the background area loss function and the face area loss function, where the background area loss function is expressed as:

[0111]

[0112] Among them, M bg represents the background mask, O new represents the segmentation output image after probability normalization; the facial area loss function expression is:

[0113]

[0114] Among them, M fg represents the face mask, I represents the input image; the total loss function is:

[0115] L seg =L bg +L fg ,

[0116] S56. Update the network parameters of the two-stream contrastive learning model by gradient descent. Specifically, calculate the partial derivatives of various network parameters with respect to the total loss function to achieve iterative update of the network. Repeat the iterative update process on different small batches until the network loss function converges or reaches the preset maximum number of iterations, and obtain the trained hyperparameters of the current round of network.

[0117] S57. Introduce different training hyperparameters, and repeat steps S51 to S56 using different training hyperparameters to obtain several dual-stream comparative learning training models. Use the validation set to evaluate the several dual-stream comparative learning training models, and select the optimal dual-stream comparative learning training model as the trained dual-stream comparative learning model.

[0118] Start the detection operation:

[0119] S6. Obtain the video to be detected, perform frame operations on it, and obtain a face image in the manner of step S2, and then obtain a pixel-level mask for the face image in the manner of step S4;

[0120] S7. Input the face video frame to be detected and the pixel-level mask generated in step S6 into the optimal detection model for classification, and obtain the classification results of all face video frames to be detected;

[0121] S8. Based on the classification and discrimination results of all face video frames to be detected, the overall video-level prediction classification result is obtained.

[0122] Execute S51 to S54 on the face video frame to be detected to obtain a semantic segmentation output map O test , and use the region scoring formula to get the probability of non-tampering p um , the specific regional scoring formula is:

[0123]

[0124] For the probability p of non-tampering um Perform probability complement operation to obtain the predicted tampering probability p m , the expression is: m =1-p um ; Set the discrimination threshold to p th In this embodiment, 0.5 is used as the discrimination threshold, and the tampering probability p m Compare with the discrimination threshold to obtain the tampering discrimination classification result of the face video frame to be detected;

[0125] If p m ≥p th , then it is determined that the current face video frame to be detected has been tampered with by the face; if p m <p th , the current video frame has not been tampered with by the face. By confirming the position and number of video frames that have been judged to have been tampered with by the face in all video frames, the video-level tampering discrimination prediction result is obtained.

[0126] Example 2

[0127] In this embodiment, in order to test the effectiveness of the optimal detection model compared with other methods, the accuracy (Accuracy, Acc) and the area under the receiver operating characteristic curve (Receiver Operating Characteristic Curve, ROC) (Area Under Curve, AUC) are used as indicators to test the performance of the dual-stream comparison learning model. This embodiment uses three face tampering video datasets, FaceForensics++ (FF++), Celeb-DF and Wild-DF, to train and test the network. The datasets used in this embodiment are briefly introduced below: FaceForensics++ contains a total of 1000 real videos and 4000 forged videos obtained by 4 different forging methods. In this embodiment, only the datasets obtained by the Deepfakes and FaceSwap tampering methods in this dataset are experimentally tested, and are divided into two versions: high-quality set (HQ) and low-quality set (LQ); The Celeb-DF dataset includes 590 real videos and 5639 high-quality videos generated by advanced face tampering algorithms. One feature of this dataset is that all the materials used are celebrity videos from public social media, so the identities of the characters in the videos are more recognizable; The Wild-DF dataset contains 3805 real videos and 3509 face tampering videos crawled from public social media. Since all the materials are from Internet social media, its scenes and tampering methods are richer.

[0128] After obtaining the video data set, this embodiment performs frame segmentation operation on all videos, and then crops out face images for training and testing of the dual-stream contrast network.

[0129] Specifically, the comparison methods used in this embodiment are from the following documents:

[0130] 1)Chollet F.Xception: Deep learning with depthwise separableconvolutions[C] / / Proceedings of the IEEE conference on computer vision andpattem recognition.2017: 1251-1258;

[0131] 2)Rao Y, Ni JA deep learning approach to detection of splicing andcopy-move forgeries inimages[C] / / 2016 IEEE International Workshop onInformation Forensics and Security(WIFS).IEEE, 2016: 1-6;

[0132] 3)Afchar D, Nozick V, Yamagishi J, et al.Mesonet: a compact facial videoforgery detection network[C] / / 2018 IEEE international workshop on informationforensics and security(WIFS).IEEE,2018:1-7;

[0133] 4)Bayar B,Stamm M CA deep learning approach to universal imagemanipulation detection using a new convolutional layer[C] / / Proceedings of the4th ACM workshop on information hiding and multimedia security.2016:5-10.

[0134] Specifically, this embodiment conducts two sets of experiments, namely: an in-library experimental test based on the FaceForensics++ dataset; a cross-library experimental test based on the Celeb-DF and Wild-DF datasets.

[0135] The first group: In-library test experiments were carried out based on the FaceForensics++ dataset. The test results are shown in Table 1, which represents the in-library test results (%) of different detection methods on FaceForensics++.

[0136] Table 1

[0137]

[0138] It can be seen from the results in Table 1 that on the high-quality set (the data set marked as HQ), this embodiment has achieved almost error-free detection performance both under the Acc indicator and the AUC indicator. On the low-quality set (the data set marked as LQ), the detection performance of all methods has declined to varying degrees, but the method proposed in this embodiment can still achieve the best detection performance. In particular, from the perspective of the Acc indicator, this embodiment surpasses the strong baseline model Xception for face detection with a detection accuracy advantage of 3.04% on FF++-DF (LQ), and surpasses Cap.V2 with a dynamic routing mechanism as the core with a detection accuracy advantage of 4.83% on FF++-FS (LQ), verifying that this embodiment is not only effective for high-quality data, but also exhibits good robustness for low-quality data.

[0139] The second group: cross-database test experiments were carried out based on the Celeb-DF and Wild-DF data sets. The test results are shown in Table 2, which shows the cross-database test results (AUC-%) of different detection methods on Celeb-DF and Wild-DF.

[0140] Table 2:

[0141]

[0142]

[0143] It can be seen from Table 2 that the method proposed in this embodiment surpasses the best model Cap.V2 in the comparison method with an AUC advantage of 1.78% in the Wild-DF database test, and can surpass Cap.V2 with a significant advantage of 6.68% in the cross-database test, which verifies that this embodiment can not only achieve higher detection performance in the database test, but also show better cross-database detection performance when facing unknown tampering methods.

[0144] The above two groups of experiments show that the method proposed in this application has a certain degree of versatility and can be applied to a variety of face tampering methods. Moreover, when the data quality changes or the tampering method changes, the present invention can exhibit good robustness.

[0145] Example 3

[0146] like Figure 4 As shown, this embodiment proposes a face tampering video detection system based on a dual-stream contrast learning model, see Figure 4 , the system comprising:

[0147] The frame processing module 101 is used to perform frame operations on the original video to obtain original frame images;

[0148] The data set construction module 102 is used to perform a face tampering operation on the original framed image to obtain a face tampering image, the face tampering image and the original framed image are mixed to form a data set, and the data set is divided into a training set, a verification set and a test set;

[0149] The face image processing module 103 performs face detection on each single frame image in the data set and cuts out the face image including the background area;

[0150] The mask generation module 104 locates the key points of the face in the input face image and generates a pixel-level mask;

[0151] The model building training module 105 is used to build a two-stream contrastive learning model, including two semantic segmentation network branches with the same structure; supervise the two-stream contrastive learning model using the training set, and then use the validation set to evaluate the two-stream contrastive learning model, select the optimal detection model, and use the test set to test the optimal detection model;

[0152] The video preprocessing module 106 is used to perform a frame division operation on the video to be detected, and obtain a face image, and obtain a pixel-level mask based on the face image;

[0153] The detection module 107 is used to input the face video frames to be detected and the pixel-level masks generated by the video preprocessing module 106 to be detected into the optimal detection model for classification, so as to obtain the classification and discrimination results of all face video frames to be detected, and obtain the overall video-level predicted classification results based on the classification and discrimination results of all face video frames to be detected.

[0154] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A face tampering video detection method based on a dual-stream contrast learning model, characterized in that: The following steps are involved: S1. Perform a frame operation on the original video to obtain an original frame image; S2. Perform a face tampering operation on the original framed image to obtain a face tampered image, mix the face tampered image and the original framed image to form a data set, and divide the data set into a training set, a validation set, and a test set; S3. Perform face detection on each single frame image in the data set and crop the face image containing the background area; S4. Positioning facial key points of the input facial image and generating a pixel-level mask; the pixel-level mask includes a face mask and a background mask; S5. Construct a two-stream contrastive learning model, including two semantic segmentation network branches with the same structure; The training set is used to supervise the training of the two-stream contrastive learning model, and then the validation set is used to evaluate the two-stream contrastive learning model, the optimal detection model is selected, and the effectiveness of the optimal detection model is tested using the test set; the process is as follows: S51. Input the training set into the view generator to obtain two RGB images under different view angles, select one of the RGB images as RGB stream data, and perform gradient filtering on the other RGB image to obtain gradient stream data; S52. Input the RGB stream data and gradient stream data obtained in S51 into the RGB stream branch and gradient stream branch of the dual-stream contrast learning model for feature extraction, and obtain feature maps F of two different flow domains. s and F g ; S53. Use the background mask to perform mask average pooling on the feature map to obtain the background feature prototypes p of two different watersheds s and p g , and add a gradient blocking mechanism after mask average pooling to prevent gradient backpropagation; The mask average pooling formula is expressed as: Among them, p c represents the output background prototype, i and j represent the pixel position index, c is the watershed category, M bg is the background mask, F c is the characteristic output of the corresponding flow domain, s and g represent the spatial flow and gradient flow respectively; S54. Feature map F for two different watersheds s and F g Perform domain adaptive mapping to obtain the mapped feature map and The feature matching formula is used to cross-match the background prototypes between different watersheds to obtain the initial semantic segmentation map O. Each pixel position of the initial semantic segmentation map is normalized to obtain the normalized probability segmentation map output O. new ; Among them, the feature matching formula is: O represents the initial segmentation output, p g and p s Represents the background prototype vectors of different watersheds, and represents the feature map output after the domain adaptation layer, and δ(·,·) represents the cosine similarity metric function; the normalization formula is: S55. Design the total loss function of the two-stream contrast learning model, including the background area loss function and the face area loss function, where the background area loss function is expressed as: Among them, M bg represents the background mask, O new represents the segmentation output image after probability normalization; the facial area loss function expression is: Among them, M fg represents the face mask, I represents the input image; the total loss function is: L seg =L bg +L fg , S56. Update the network parameters of the two-stream contrastive learning model using gradient descent; S57. Introduce different training hyperparameters, and repeat steps S51 to S56 using different training hyperparameters to obtain several dual-stream comparative learning training models. Use the validation set to evaluate the several dual-stream comparative learning training models, select the optimal dual-stream comparative learning training model as the trained dual-stream comparative learning model, and use the test set to test the effectiveness of the optimal detection model. S6. Obtain the video to be detected, perform frame operations on it, and obtain a face image in the manner of step S2, and then obtain a pixel-level mask for the face image in the manner of step S4; S7. Input the face video frame to be detected and the pixel-level mask generated in step S6 into the optimal detection model for classification, and obtain the classification results of all face video frames to be detected; S8. Based on the classification and discrimination results of all face video frames to be detected, the overall video-level prediction classification result is obtained.

2. The face tampering video detection method based on the dual-stream contrast learning model according to claim 1 is characterized in that: The process of performing face tampering operation on the original framed image in step S2 is specifically as follows: S21. Match the framed images of different face identities in the original framed images in pairs, and use them as source material and target material respectively; S22. Perform face tampering operations on the source material and the target material using a face tampering method to obtain a face tampering image. The face tampering method includes a Deepfakes method and a FaceSwap method.

3. The face tampering video detection method based on the dual-stream contrast learning model according to claim 2 is characterized in that: In step S3, face detection is performed on each single frame image in the data set after the frame operation, and the process of cropping the face image containing the background area is specifically as follows: S31. Perform face detection on each single frame image in the data set after the frame operation to obtain the original face cropping frame; S32. Fix the center position of the face cropping frame, and enlarge the face cropping frame by 1:1.3 to obtain the enlarged face cropping frame; S33. Based on the enlarged face cropping frame, perform region cropping on the single frame image to obtain a face image including the background region.

4. The face tampering video detection method based on the dual-stream contrast learning model according to claim 3 is characterized in that: The process of locating the key points of the face on the input single-frame image and generating a pixel-level mask is as follows: S41. Determine the number q of facial key points, and obtain the positioning coordinate values ​​of q facial key points; S42. Perform a convex hull algorithm on the q facial key points to obtain a closed U-shaped facial region mask, that is, a facial mask; S43. Perform a logical complement operation on the face mask to obtain a background mask.

5. The face tampering video detection method based on the dual-stream contrast learning model according to claim 4 is characterized in that: The dual-stream contrastive learning model constructed in step S5 includes two semantic segmentation network branches with the same structure, one is an RGB stream branch, and the other is a gradient stream branch; the semantic segmentation network includes two basic parts: one part is an encoder built with the EfficientNet-b3 network as the backbone, and the trained network weight parameters of the network on ImageNet are loaded as the pre-trained network weight parameters of the encoder; the other part is a shallow decoder obtained by alternately stacking a bilinear upsampling module and a normal convolution module, and the weight parameters of the decoder network of this part are preset by using Kaiming initialization.

6. The face tampering video detection method based on the dual-stream contrast learning model according to claim 5 is characterized in that: Step S5 also includes constructing a perspective generator for receiving video frames input thereto and generating two RGB images under different perspectives. The perspective generator integrates four perspective transformation methods, namely: Gaussian blur, Gaussian noise, grayscale, and SRM noise superposition.

7. The face tampering video detection method based on the dual-stream contrast learning model according to claim 6 is characterized in that: In step S7, S51 to S54 are executed on the face video frame to be detected to obtain a semantic segmentation output map O test , and use the region scoring formula to get the probability of non-tampering p um , the specific regional scoring formula is: For the probability p of non-tampering um Perform probability complement operation to obtain the predicted tampering probability p m , the expression is: m =1-p um ; Set the discrimination threshold to p t , the tampering probability p m Compare it with the discrimination threshold to obtain the tampering discrimination classification result of the face video frame to be detected.

8. The face tampering video detection method based on the dual-stream contrast learning model according to claim 7 is characterized in that: If p m ≥p t , then it is determined that the current face video frame to be detected has been tampered with by the face; if p m <p t , the current video frame has not been tampered with by the face. By confirming the position and number of video frames that have been judged to have been tampered with by the face in all video frames, the video-level tampering discrimination prediction result is obtained.

9. A face tampering video detection system based on a dual-stream contrast learning model, characterized in that: The system is used to implement the face tampering video detection method based on the dual-stream contrast learning model according to claim 1, and the system includes: A frame processing module is used to perform frame operations on the original video to obtain original frame images; A data set construction module is used to perform a face tampering operation on the original framed image to obtain a face tampering image, the face tampering image and the original framed image are mixed to form a data set, and the data set is divided into a training set, a validation set and a test set; The face image processing module is used to perform face detection on each single frame image in the data set and crop the face image containing the background area; The mask generation module locates the key points of the input face image and generates a pixel-level mask; The model building training module is used to build a two-stream contrastive learning model, including two semantic segmentation network branches with the same structure; the two-stream contrastive learning model is supervised and trained using the training set, and then the two-stream contrastive learning model is evaluated using the validation set to select the optimal detection model, and the effectiveness of the optimal detection model is tested using the test set; The video preprocessing module to be detected performs frame division operation on the video to be detected, obtains the face image, and obtains the pixel-level mask based on the face image; The detection module is used to input the face video frames to be detected and the pixel-level masks generated by the video preprocessing module to be detected into the optimal detection model for classification, so as to obtain the classification and discrimination results of all face video frames to be detected, and obtain the overall video-level predicted classification results based on the classification and discrimination results of all face video frames to be detected.

Citation Information

Patent Citations

  • Video face tampering detection method and system based on cyclic convolutional neural network

    CN110880172A

  • False face video tampering detection method and system based on time domain self-attention mechanism

    CN112733625A

  • Deep fake face video detection method and system based on face key points, and medium

    CN113449657A