Face forgery detection method based on desensitization contrast learning

Abstract

The invention discloses a face forgery detection method based on desensitization contrast learning, and belongs to the field of computer vision. The method comprises the following steps: 1, acquiring a face image from a video frame; 2, inputting the face image into a first layer of convolutional network, then performing domain transformation on low-level visual features of two random samples belonging to the same category from a statistical feature level, but keeping internal features of the two samples unchanged; 3, performing desensitization learning on the features output by the domain transformation to enable model learning to delete domain style mode information therein, and projecting the domain style mode information back to the original manifold of the sample; 4, utilizing comparison loss to keep enough distance between different types of image domains so as to limit the boundary of the domains; and 5, constructing a desensitization contrast learning network, and feeding the domain invariant representation to a downstream task module for binary classification to obtain a face forgery detection result. According to the cross-domain desensitization face forgery detection network designed by the invention, the domain information in the sample is desensitized through domain transfer, so that the finally obtained features have higher resolution and robustness, more robust face forgery detection is realized, and the cross-domain desensitization face forgery detection network has very high practical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for detecting forged faces based on desensitization contrast learning. Background Art

[0002] With the development of deep learning technology, the application of face recognition has achieved great success recently. However, existing face recognition systems are vulnerable to face forgery attacks, which refers to deep learning-based technology. It forges images and videos by replacing faces, creating faces that do not exist in reality, or changing the original features of the face. However, due to the easy availability of face forgery technology, forged videos and images have become more and more widespread, which has led to people being misled and brought new risks to personal privacy, economy, law and even national security.

[0003] With the development of deep learning in recent years, the detection methods of deep fake face videos have gradually shifted from mining manually marked features and traditional machine learning methods to using various deep neural networks for detection. In terms of feature utilization, researchers have detected deep fake face videos from the perspectives of spatial features, temporal features, spatiotemporal fusion features, and biological signal features.

[0004] Many forgery detection methods have been proposed recently, most of which usually frame the forgery detection task as a real-fake binary classification problem. The face forgery detection problem is usually modeled as an image-level classification problem, which focuses on finding the pattern differences between real and fake images. Although these methods have achieved certain results in detection accuracy, they are easily affected by specific forgery patterns. When the forgery methods encountered in the real world lose this fixed pattern, the detector will fail to detect.

[0005] Based on this problem, many researchers have begun to study forgery detection methods with more generalized discriminative capabilities, with the goal of learning a universal representation from a variety of forgery data sets so that it can distinguish real samples from forged samples. Recently, some studies have adopted a self-supervised learning method to build an additional reconstruction learning module in several convolutional layers of the feature extractor to learn to reconstruct real face images. The motivation is to assume that the self-encoder-decoder trained only on real images has a small reconstruction error only on real images, but cannot perfectly reconstruct on forged images. Based on this assumption, it is used to further assist the discriminator in forgery detection. However, this method is prone to overfitting to the real images in the data set, and it is difficult to learn a generalized real face representation. Specifically, in the face of real images that have not been seen in training, the self-encoder cannot reconstruct well, resulting in false alarms and misjudgments as forged images. Based on this, how to effectively learn the essential features of real face images through unsupervised learning and achieve accurate face forgery detection is still a challenging problem. In order to solve the above problems, the present invention proposes a face forgery detection method based on desensitization contrast learning. Summary of the invention

[0006] The present invention aims to solve the problems of the prior art and proposes a face forgery detection method based on desensitization contrast learning.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0008] A method for detecting forged faces based on desensitization contrastive learning, comprising the following steps:

[0009] Step 1: Use the face recognition algorithm to crop the face area in the data set;

[0010] Step 2: transform the low-level visual features of the face image into a domain from the statistical feature level;

[0011] Step 3: Desensitize the features so that the model learns to delete the domain style information and project it back to the original manifold of the sample;

[0012] Step 4: Use contrast loss to keep a sufficient distance between image domains of different categories to limit the boundaries of the domains;

[0013] Step 5: Construct a desensitized contrastive learning network and feed the domain-invariant representation to the downstream task module for binary classification. Use the trained desensitized contrastive learning network to apply it on multiple face forgery datasets, construct a desensitized contrastive learning network, feed the domain-invariant representation to the downstream task module for binary classification, obtain face forgery detection results, and evaluate the detection results.

[0014] Furthermore, the step 1 uses a face recognition algorithm to crop the face area in the data set, specifically including the following steps:

[0015] A1. Dividing the video data in the face forgery video dataset into a group of continuous non-overlapping video frames;

[0016] A2, performing face detection on the video frames in sequence, locating facial feature points, and capturing a square image of the facial part;

[0017] A3. Modify the face area into a 224*224 RGB image.

[0018] Furthermore, the face image is input into the first layer of the convolutional network, and then the low-level visual features of two random samples belonging to the same category are domain transformed from the statistical feature level, but the intrinsic features of the two samples are kept unchanged, specifically including the following steps:

[0019] B1: Input a batch size image and first pass it through a 3*3 convolution layer to obtain the feature map z

[0020] B2: The true features z of the pairs in the feature set z of batch size A and z B Perform domain transformation, where the domain transformation formula is as follows:

[0021]

[0022] Among them, μ A , μ B , σ A , σ B is the statistical characteristic of the feature. A Represents the mean of the real sample A, μ B represents the mean of the real sample B, σ A Represents the standard deviation of the true sample A, σ B represents the standard deviation of the real sample B. Formula (1) can be regarded as the feature z of the real sample A. A The domain of z is normalized, which essentially aligns B The characteristic statistics of out Contains and z B Same style.

[0023] B3: To ensure z out Contains and z A Based on the essential features of , we introduced intrinsic feature loss and domain alignment loss. The intrinsic features help to maintain the structural information in the reconstructed image. The formula is as follows:

[0024]

[0025] On the other hand, domain alignment loss plays an important role in pairing images. Its formula is as follows:

[0026] Where μ and σ represent the feature statistics obtained by the multi-layer convolutional network mentioned above. Domain alignment can effectively help us align paired images x at the image level. A and x B Domain information.

[0027] Furthermore, the features of the domain transformation output are desensitized so that the model learns to delete the domain style pattern information and project it back to the original manifold of the sample. Specifically, the following steps are included:

[0028] C1: To obtain a given real face image x A The domain-invariant representation z of this method is obtained by domain transformation out Learn desensitization.

[0029] C2: Input the input face image into the desensitization learning network to obtain z out The desensitization learning network uses the Xception structure as the basic network structure for z out Extract features, and then gradually restore the feature map to the original image size through upsampling and convolution operations of the decoder, thereby realizing face image x A reconstruction.

[0030] C3: Due to z out By x A The intrinsic features of the image x are combined with other images B We learn a decoder to represent the style features Thus removing z out Medium B The style information is then compressed back to the original x A In the data manifold that it depends on, the desensitization learning loss formula is as follows:

[0031]

[0032] Furthermore, contrast loss is used to keep a sufficient distance between image domains of different categories to limit the boundaries of the domains, which specifically includes the following steps:

[0033] D1: Input the input face image into the desensitization learning network, and the desensitization learning network obtains the features F of each layer 1 , F 2 , F 3 .

[0034] D2: For each layer of batch size F 1 , F2 , F 3 In, assuming and Represent the real images and The domain invariant feature of is the representation of the forged image. We use contrast loss to make the features of the real image and the forged image as far apart as possible in each layer of features. The contrast loss formula is as follows:

[0035]

[0036] Where R, F represent the real and fake image sets, N r , N f Represents their size. Dis() represents the matrix-based cosine distance, and its formula is as follows

[0037]

[0038] Furthermore, the above four losses are added together to obtain the final face image forgery detection loss, which specifically includes:

[0039] By integrating the prediction module and the domain desensitization network into a unified network framework, a face forgery detection network based on desensitization learning is formed, and its parameters can be calculated by L all To optimize:

[0040] L all =L cls +λ 1 (L d +L i +L s )+λ 2 L b

[0041] Among them, λ 1 and λ 2 Represents the hyperparameters in the overall loss function.

[0042] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for detecting face forgery based on global context structure difference as described in any one of the items is implemented.

[0043] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting face forgery based on global context structure difference as described in any one of the items is implemented.

[0044] The advantages and beneficial effects of the present invention are as follows:

[0045] 1. This invention mainly aims at the problem that the existing face forgery detection methods have poor accuracy when facing face forgery detection with changeable forgery methods and realistic factors, and self-supervised forgery detection is prone to false alarms. A face forgery detection system based on desensitization learning is designed. The core of this system is to obtain the essential features of real faces through desensitization learning, and then make classification judgments. Experimental verification was carried out on benchmark datasets such as FaceForensics++ and Celeb-DF. The results show that this method significantly avoids the false alarm problem in forgery detection and is much better than the existing face forgery detection methods based on prior experience.

[0046] 2. The present invention can effectively capture the essential features of facial images. Experimental verification has been carried out on benchmark datasets such as FaceForensics++ and Celeb-DF. The results show that the method significantly avoids the problem of false alarms in forgery detection and is significantly better than the existing face forgery detection method based on specific forgery patterns, overcoming the problem of cross-domain detection performance degradation in face forgery detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic diagram of a face forgery detection method based on desensitization learning provided by an embodiment of the present invention;

[0048] Figure 2 It is a frame diagram of a face forgery detection network model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and in detail describe the technical solutions in the embodiments of the present invention. The following embodiments will help those skilled in the art to further understand the present invention, but will not limit the present invention in any form. The described embodiments are only part of the embodiments of the present invention.

[0051] Embodiment 1

[0052] See also Figure 1 , Figure 1 is a schematic diagram of a face forgery detection method based on desensitization learning provided by an embodiment of the present invention, wherein the desensitization learning

[0053] The face forgery detection method first inputs the face image into the feature extraction basic network Xception, extracts the corresponding face representation information z from the first convolutional layer, and then passes it to the domain transformation module. The domain transformation module decouples the above representation information into essential feature information and style essential feature information. The essential feature information includes the domain-independent features of the real face image in the real data manifold, and the style feature represents the information from the domain. In desensitization learning, we first mix the style features in the sample image, and then learn an encoder that can remove the style information from the domain mixed image, extract the essential feature information, restore the original image from the mixed features, and compress it back to the data manifold it originally depends on. In order to ensure z out Keep z A We use intrinsic loss, domain alignment loss, and the data manifold on which we rely to preserve the structural information and domain information in the reconstructed image, thereby improving the robustness of the forgery detection model. Finally, we input the extracted features into other convolutional layers of the base network for feature extraction, and further learn features related to forgery classification for the final classification, mainly including:

[0054] Step 1: Perform domain desensitization on the input real image, such as Figure 2 shown.

[0055] This embodiment uses a self-supervised network based on desensitization learning. Through domain desensitization, it can simultaneously learn the essential features and domain style features of real faces, thereby increasing the generalization ability of the model and solving the problem of insufficient generalization ability of the model for data domain conversion.

[0056] Step 2: See Figure 2 , Figure 2 It is an overall block diagram of a face forgery detection method based on desensitization learning provided by an embodiment of the present invention.

[0057] Step 3: Input the 224*224 face image into the desensitization learning network. The desensitization learning network includes a linear projection unit, an encoder, and a decoder. The encoder consists of three blocks, each of which contains a 3*3 convolutional layer. The encoder obtains the features F of each layer. 1 , F 2 , F3, then domain transformation is performed on two samples belonging to the same category, where domain transformation can be regarded as normalizing the statistics of the features of the real sample A, which essentially aligns the feature statistics of another sample B, so that the updated features contain the style of another sample B.

[0058] Step 4: Desensitize the features output by the domain transformation so that the model learns to delete the domain style pattern information and projects it back to the original manifold of the sample. Specifically, the updated features are generated by combining the style features of B and the intrinsic features of A. In order to enable the model to learn to delete such pattern information, we need to learn to desensitize the style information of B in the decoder and project it back to the original manifold of A, thereby maintaining the domain-invariant representation of A, which helps to maintain the structural information in the reconstructed image.

[0059] The domain-invariant representation is used as the input to the prediction module as the downstream convolution, and the features are extracted through the Xception convolutional network and finally input into a fully connected layer to output the predicted probability.

[0060] Step 5: Use contrast loss to keep enough distance between image domains of different categories to limit the boundary of the domain. For each layer feature F of batch size 1 , F 2 , F 3 In, assuming and Represent the real images and The domain invariant feature of is the representation of the forged image. We use contrast loss to make the features of the real image and the forged image as far away as possible in each layer of features.

[0061] Step 6: Build a desensitized contrastive learning network and use the domain-invariant representation as the input of the downstream convolution to the prediction module. By integrating the prediction module and the domain desensitization network into a unified network framework, the features are extracted through the Xception convolutional network and finally input into a fully connected layer to output the prediction probability.

[0062] This embodiment adopts a training method similar to that of a generative adversarial network. First, the face reconstruction module is trained only on real face images, and then the forgery detector is trained in an end-to-end manner. The detailed process can be referred to the prior art and will not be introduced here.

[0063] The present invention provides a method for detecting face forgery based on desensitization learning.

[0064] This method simultaneously considers the robustness of extracting true and false features and the characterization of features from local and global robustness, making the final features more discriminative and robust, thereby achieving more robust face forgery detection.

[0065] Embodiment 2

[0066] The method in the above embodiment is experimentally verified with the existing method on all training sets of the commonly used face forgery detection datasets WildDeepfake, Celeb-DF and FaceForensics++, and compared with several existing methods to verify the effectiveness and robustness of the above method.

[0067] Test 1:

[0068] This experiment used 7 existing methods and the method of the present invention to compare the effects on the WildDeepfake dataset and Celeb-DF. The quantitative evaluation indicators include accuracy ACC, area under the receiver operating characteristic curve AUC (the area under the receiver operating characteristic curve), and equal error rate ERR (EqualError Rate). Among them, the larger the AUC value, the better, the larger the ACC value, the better, and the smaller the ERR value, the better. Among them, the 7 methods are as follows:

[0069] Method 1 is the method proposed by Chollet et al. in the paper "Xception: Deep learning with depthwiseseparable convolutions."; Method 2 is the method proposed by Zi et al. in the paper "Wild deepfake: A challenging real-world dataset for deepfake detection"; Method 3 is the method proposed by Zhao et al. in the paper "Multi-attentional deepfake detection"; Method 4 is the method proposed by Wang and Deng et al. in the paper "Representative forgery mining for fake face detection"; Method 5 is the method proposed by Cao et al. in the paper "End-to-end 510 reconstruction-classification learning for face forgery detection."; Method 6 is the method proposed by Zhang et al. in the paper "Deeplearning for medical image segmentation: tricks, challenges and future directions."; Method 7 is the method proposed by Shi et al. in the paper "Discrepancy-guided reconstruction 616 lcarning for image detection."

[0070] The test results are shown in Table 1 below.

[0071] Table 1 Performance comparison of different methods on the WildDeepfake dataset

[0072]

[0073] As can be seen from Table 1, the method of the present invention achieves a high accuracy of 85.21% in the WildDeepfake dataset. Compared with other frame-level methods, the accuracy of the present invention is 0.68% higher than that of method 7Dis-GRL, 6.96% higher than that of method 2Add-Net, and 7.96% higher than that of method 1Xception. The method of the present invention achieves a high accuracy of 99.94% in the Celeb-DF dataset. Compared with other frame-level methods, the accuracy of the present invention is 1.23% higher than that of method 7Dis-GRL, 1.35% higher than that of method 5RECCE, and 2.04% higher than that of method 1Xception.

[0074] Experiment 2

[0075] This experiment was trained on the FaceForensics++ dataset and tested on the Celeb-DF dataset. The results are shown in Table 2 below.

[0076] Table 2 Performance comparison of different methods on the Celeb-DF dataset

[0077] Method AUC (%) EER(%) Method 1: Xception 61.80 41.73 Method 2: Add-Net 65.29 38.90 Method 3: MultiAtt 67.02 37.90 Method 4: RFM 65.63 38.54 Method 5: RECCE 68.71 35.73 Method 7: Dis-GRL 70.03 34.23 Method of the present invention: CDN 70.73 34.66

[0078] It can be seen from Table 2 that the generalization performance of the method of the present invention is better than that of the other six modern detection networks. Specifically, when the model trained on the FaceForensics++ dataset is migrated to the Celeb-DF dataset for testing, the AUC value of our method can reach 70.73%, which is 5.10% higher than that of method 5RECCE and 8.93% higher than that of method 1Xception.

[0079] Combining Table 1 and Table 2, it can be seen that the method of the present invention is superior to the existing methods in terms of generalization ability and accuracy.

[0080] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A face forgery detection method based on desensitization contrast learning, characterized in that The steps include: Step 1: pre-process the input video containing human faces to obtain a frame-by-frame sequence of human face images; Step 2: Input the face image into the first layer of the convolutional network, and then transform the low-level visual features of two random samples belonging to the same category from the statistical feature level, but keep the intrinsic features of the two samples unchanged; Step 3: Desensitize the features output by the domain transformation so that the model learns to delete the domain style pattern information and project it back to the original manifold of the sample; Step 4: Use contrast loss to keep a sufficient distance between image domains of different categories to limit the boundaries of the domains; Step 5: Build a desensitized contrastive learning network and feed the domain-invariant representation to the downstream task module for binary classification to obtain the face forgery detection result.

2. According to claim 1, a face forgery detection method based on desensitization contrast learning is characterized in that: The preprocessing of the input video containing human faces includes the following steps: Step 1: dividing the video data in the face forgery video data set into a group of continuous non-overlapping video frames; Step 2: performing face detection on the video frames in turn to extract the face area; Step 3: Modify the face area into a 224*224 RGB image.

3. According to claim 1, a face forgery detection method based on desensitization contrast learning is characterized in that: The face image is input into the first layer of the convolutional network, and then the domain transformation is performed on two images belonging to the same category, including the following steps: Step 1: Two random samples x from different domains belonging to the same category A and x B , the input image of a batch size first passes through a 3*3 convolutional layer, and the encoder outputs its low-level feature set z A and z B ; Step 2: Paired true features z in the feature set z of batch size A and z B Perform domain transformation, where the domain transformation formula is as follows: Among them, μ A , μ B , σ A , σ B is the statistical characteristic of the feature. μ A Represents the mean of the real sample A, μ B represents the mean of the real sample B, σ A Represents the standard deviation of the true sample A, σ B represents the standard deviation of the real sample B. Formula (1) can be regarded as the feature z of the real sample A. A The domain of z is normalized, which essentially aligns B The characteristic statistics of out Contains and z B Same style. Step 3: To ensure z out Keep z A To further validate the intrinsic features of , we need to use two additional losses, namely the intrinsic loss and the domain alignment loss, as described below: Step 4: Intrinsic features help preserve the structural information in the reconstructed image. The formula is as follows: On the other hand, domain alignment loss plays an important role in pairing images. Its formula is as follows: Where μ and σ represent the feature statistics obtained by the multi-layer convolutional network mentioned above; domain alignment allows us to align paired images x at the image level A and x B Domain information.

4. According to claim 1, a face forgery detection method based on desensitization contrast learning is characterized in that: The features output by the domain transformation are desensitized so that the model can learn to delete the domain style pattern information and project it back to the original manifold of the sample, including the following steps: Step 1: Transform the domain output z out Desensitization learning; specifically, due to z out It is combined with x B In order for the model to learn to remove such pattern information, we need to project it back to its x A The original manifold of Step 2: By learning the decoder φ, we can learn to out Desensitization in x B The domain style information of x A The domain of z remains unchanged; Step 3: The desensitization learning network uses the Xception structure as the basic network structure to out Extract features, and then gradually restore the feature map to the original image size through upsampling and convolution operations of the decoder, thereby realizing face image x A reconstruction. Step 4: Due to z out By x A The intrinsic features of the image x are combined with other images B We learn a decoder to represent the style features Thus removing z out Medium B The style information is then compressed back to the original x A In the data manifold that it depends on, the desensitization learning loss formula is as follows:

5. According to claim 1, a method for detecting face forgery based on desensitization contrastive learning is characterized in that: The contrast loss is used to keep a sufficient distance between image domains of different categories to limit the boundaries of the domains, including the following steps: Step 1: Input the input face image into the desensitization learning network, and the desensitization learning network obtains the features F1, F2, and F3 of each layer; Step 2: For each layer feature F1, F2, F3 of batch size, assume and Represent the real images and The domain invariant feature of is the representation of the forged image; we use contrast loss to make the features of the real image and the forged image as far away as possible in each layer of features; the contrast loss formula is as follows: Where R, F represent the real and fake image sets, N r , N f Represents their size; Dis() represents the matrix-based cosine distance, and its formula is as follows:

6. According to claim 1, a face forgery detection method based on desensitization contrast learning is characterized in that: Construct a desensitized contrastive learning network, feed the domain-invariant representation to the downstream task module for binary classification, and obtain the face forgery detection result, including the following steps:' Step 1: The domain-invariant representation is used as the input of the downstream convolution to the prediction module, and the features are extracted through the Xception convolutional network and finally input into a fully connected layer to output the prediction probability; Step 2: By integrating the prediction module and the domain desensitization network into a unified network framework, a face forgery detection network based on desensitization learning is formed, and its parameters can be calculated by L all To optimize: L all =L cls +λ1(L d +L i +L s )+λ2L b Among them, λ1 and λ2 represent the hyperparameters in the total loss function; Step 3: When the negative log-likelihood loss function value of the current round on the validation set increases by more than the set value compared to the minimum negative log-likelihood loss function value during the training process, stop training.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for detecting face forgery based on desensitization learning as described in any one of the items is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting face forgery based on desensitization learning as described in any one of the items is implemented.

Images