A deep fake detection method based on balanced contrastive learning
By combining unsupervised and supervised contrastive learning strategies, the deep forgery detection method based on balanced contrastive learning addresses the problems of overfitting and insufficient generalization in existing technologies, and achieves accurate identification and detection of deep forgery content.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2024-07-10
- Publication Date
- 2026-07-03
AI Technical Summary
Existing deepfake detection methods suffer from overfitting to training datasets, resulting in poor generalization ability to unseen datasets and different manipulation methods, and difficulty in dealing with constantly evolving deepfake content.
We employ a deep forgery detection method based on balanced contrastive learning, combining unsupervised and supervised contrastive learning strategies. We expand the data through data augmentation methods and enhance the classification effect by jointly using supervised and unsupervised contrastive loss. We use a Siamese network encoder, a multi-scale attention interaction module, and a prediction head for feature extraction and fusion.
It achieves excellent detection performance for unseen deepfake images and videos, and can more comprehensively learn the subtle differences between real and fake content, thus improving the security and accuracy of detection.
Smart Images
Figure CN119068319B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a deep forgery detection method based on balanced contrastive learning. Background Technology
[0002] Deepfake detection can be viewed as a binary classification task. Past research has focused on detecting features in fake images or videos, such as blinking, heart rate, texture, and frequency characteristics. Other studies have emphasized network models suitable for deepfake detection, such as Xception, EfficientNet, and VisionTransformer. While existing deepfake detection methods perform well, they also have some limitations. Many deepfake detection methods suffer from overfitting to the training dataset, their predictions tending to reflect the probabilities of the data distribution rather than the forgery artifacts. This overfitting model generalizes poorly to unseen datasets or different manipulation methods.
[0003] Furthermore, as deepfake technology continues to develop and evolve, detection methods also need constant updates and improvements to meet new challenges. For example, with the continuous advancement of generative models, new deepfake content may become more realistic and difficult to distinguish, requiring detection methods to possess higher sensitivity and accuracy. In addition, deepfake technology may employ adversarial training to evade detection methods, rendering traditional detection algorithms ineffective. Therefore, researchers need to continuously explore new technologies and algorithms to address the ongoing evolution of deepfake technology.
[0004] In view of this, in-depth research was conducted on the above issues, which led to the creation of this case. Summary of the Invention
[0005] The purpose of this invention is to provide a deep forgery detection method based on balanced contrastive learning, so as to solve the problem of overfitting in the deep forgery detection model and the need for continuous updating and improvement of the detection method mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a deep forgery detection method based on balanced contrastive learning;
[0007] The deepfake detection model is a deepfake detection model based on a balanced contrastive learning strategy. This model combines unsupervised and supervised contrastive learning strategies, uses data augmentation methods to expand the data, and jointly enhances the deepfake classification effect through supervised and unsupervised contrastive loss.
[0008] The model comprises a Siamese encoder, two multi-scale attention interaction modules (MAI), two representation modules (Projection), and a prediction head (Prediction). Its steps are as follows:
[0009] Step 1: For the input video, first divide it into video frames, and randomly select video frames for face region cropping;
[0010] Step 2: For the input image, crop the face region;
[0011] Step 3: For the cropped face region, data augmentation is used to obtain the enhanced face region image;
[0012] Step 4: Pass the face regions of the original image and the face regions of the enhanced image through a Siamese network encoder to obtain multi-scale features of the original image and the enhanced image respectively.
[0013] Step 5: Input the multi-scale features of the original image and the multi-scale features of the enhanced image into the multi-scale attention interaction module to obtain the multi-scale attention fusion features of the face region in the original image and the face region in the enhanced image;
[0014] Step 6: The fused features are passed through a representation layer and then concatenated into a prediction head to obtain the results of deepfake detection.
[0015] Preferably, in step 1, for the input video, the video is first segmented into frames using the OpenCV tool, and a video frame is randomly selected for face region cropping; the face region is obtained through dlib, and the face region is enlarged by 1.2 times and then cropped to obtain the face region to be detected.
[0016] Preferably, in step 2, for the input image, the face region is obtained using dlib, and the face region is enlarged by 1.2 times and then cropped to obtain the face region to be detected.
[0017] Preferably, in step 3, data augmentation employs random flipping (RF), random resizing and cropping (RRC), and random erasure (RE); the probability of RF is set to 0.5; the RRC employs a scaling factor (1 / 1.3, 1) and an aspect ratio (0.9, 1.1); and the RE employs a scaling factor (0.02, 0.2), an aspect ratio (0.5, 2), and a probability (0.5).
[0018] Preferably, in step 4, for the face region of the input original image and enhance the face region of the image The encoders respectively obtain multi-scale features of the face region in the original image. , , )and( , , The formula is as follows:
[0019]
[0020]
[0021] In this network, the encoder is an Xception network, E represents the feature vector of the first stage EntryFlow of the Xception network, M represents the feature vector of the second stage Middle Flow of the Xception network, and O represents the feature vector of the third stage Exit Flow of the Xception network.
[0022] Preferably, in step 5, the multi-scale attention interaction module (MAI) first uses an attention mechanism to extract attention from features at multiple scales, and then obtains the relevant attention between different features by multiplying them in pairs. Finally, it performs feature fusion by adding the features together.
[0023] Preferably, in the multi-scale attention interaction module, each input ( , , All features are processed through a SEAttention module, and then through an adaptive average pooling layer to reduce the size of the feature map to 7. 7 pixels; then, a 1x1 Conv2D layer adjusts the dimension of its features to 2048, followed by batch normalization to obtain a dimension of 7. 7 Attention features of 2048; input obtained in this structure , , attention characteristics , , Then, a pairwise method is used to calculate their cross-attention; and Pixel-wise multiplication to obtain cross-attention features Similarly, calculate ( , )and( , ) combination to obtain and Through the above steps, , and Integrated information at different scales; , and The features are concatenated together, then the feature dimension is adjusted to 2048 through a 1x1 Conv2D layer; finally, a batch normalization layer is used to obtain multi-scale attention features. And as the output of the multi-scale attention interaction module, the formula can be expressed as:
[0024]
[0025] For the input ( , , Similarly, multi-scale attention features are obtained. The formula can be expressed as:
[0026] .
[0027] Preferably, in step 6, the material obtained in step 5 is... and Each input is a projection layer, which is a simple multilayer perceptron structure. Through these projection layers, the representational features of the face region in the original image are obtained. and enhance the representation features of the face region in the image. By representing the vector and The images are concatenated and then fed into a prediction head called Prediction to obtain true / false predictions of the input image. The entire process can be represented by the following formula:
[0028]
[0029]
[0030]
[0031] .
[0032] Preferably, the model has three losses: unsupervised contrastive learning loss. Supervised comparative learning loss and classification loss ;
[0033] Unsupervised contrastive learning loss It can be defined as: for a containing A small batch of the original images, Represents the index of all images (original image and enhanced image); if the index If it is an original image, then the index It is an index Data augmentation images, and vice versa; for unsupervised contrastive learning, the index... The corresponding sample is called the anchor point, index. The corresponding sample is called a positive sample; other indexes in the batch The samples that are negative are called negative samples, and the number of negative samples is ;index The representational features of a sample are denoted as Therefore, comparing learning loss It can be represented as:
[0034]
[0035] in Indicates the inner product. It is a temperature parameter;
[0036] Among them, supervised contrastive learning loss This can be defined as: all samples of the same category in a batch, including both original and enhanced images, are defined as positive samples, denoted as . , It is its cardinality; similarly, samples from different categories are considered negative samples, denoted as . Therefore, supervised contrastive learning loss It can be represented as:
[0037]
[0038] For a mini-batch of all the original images, where Indicates the index of the original image; for an index of The input, its label is The predicted probability is Cross-entropy is used to quantify the difference between predicted values and labels; cross-entropy loss. It can be defined as:
[0039]
[0040] Total loss of the model It can be defined as:
[0041] .
[0042] Compared with the prior art, the beneficial effects of the present invention are as follows: This deep forgery detection method based on balanced contrastive learning has the following advantages:
[0043] First, a deepfake detection model based on a balanced contrastive learning strategy is proposed. The proposed deepfake detection model is an end-to-end model that extracts facial features from images and videos and effectively distinguishes between fake and real faces by using contrastive learning. In particular, the model still exhibits excellent detection performance when dealing with unseen deepfake images and videos.
[0044] Secondly, a feature fusion method based on multi-scale attention interaction is proposed. The proposed balanced contrastive learning strategy, which combines supervised and unsupervised contrastive learning, has a simple structure and good performance. This strategy integrates the advantages of both supervised and unsupervised contrastive learning, enabling the model to more comprehensively learn the subtle differences between real and fake content during training. This method achieves accurate identification of fake content by extracting features from images and videos at different scales and using an attention mechanism for feature fusion.
[0045] Overall, this invention demonstrates excellent performance in terms of security, detection capabilities, and ease of use when dealing with deepfake content. With the continuous advancement of artificial intelligence technology, these solutions can play a crucial role in the future of computer vision. Attached Figure Description
[0046] Figure 1 This is a flowchart of the deep forgery detection model framework based on balanced contrastive learning of the present invention;
[0047] Figure 2 This is a flowchart of the multi-scale attention interaction module framework of the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] Please see Figure 1-2 The present invention provides the following technical solution: a deep forgery detection method based on balanced contrastive learning;
[0050] The deepfake detection model is a deepfake detection model based on a balanced contrastive learning strategy. This model combines unsupervised and supervised contrastive learning strategies, uses data augmentation methods to expand the data, and jointly enhances the deepfake classification effect through supervised and unsupervised contrastive loss.
[0051] The model comprises a Siamese encoder, two multi-scale attention interaction modules (MAI), two representation modules (Projection), and a prediction head (Prediction). Figure 1 As shown. The steps are as follows:
[0052] Step 1: For the input video, first divide it into video frames, and randomly select video frames for face region cropping;
[0053] Step 2: For the input image, crop the face region;
[0054] Step 3: For the cropped face region, data augmentation is used to obtain the enhanced face region image;
[0055] Step 4: Pass the face regions of the original image and the face regions of the enhanced image through a Siamese network encoder to obtain multi-scale features of the original image and the enhanced image respectively.
[0056] Step 5: Input the multi-scale features of the original image and the multi-scale features of the enhanced image into the multi-scale attention interaction module respectively to obtain the multi-scale attention fusion features of the face regions in the original image and the face regions in the enhanced image, such as... Figure 2 As shown;
[0057] Step 6: The fused features are passed through a representation layer and then concatenated into a prediction head to obtain the results of deepfake detection.
[0058] In step 1, for the input video, the OpenCV tool is first used to cut the video into frames, and a frame is randomly selected for face region cropping; the face region is obtained through dlib, and the face region is enlarged by 1.2 times and then cropped to obtain the face region to be detected.
[0059] In step 2, for the input image, the face region is obtained using dlib, and the face region is enlarged by 1.2 times and then cropped to obtain the face region to be detected.
[0060] In step 3, data augmentation uses random flipping (RF), random resizing and cropping (RRC), and random erasure (RE); the probability of RF is set to 0.5; RRC uses a scaling factor (1 / 1.3, 1) and an aspect ratio (0.9, 1.1); RE uses a scaling factor (0.02, 0.2), an aspect ratio (0.5, 2), and a probability (0.5).
[0061] In step 4, for the face region of the input original image... and enhance the face region of the image The encoders respectively obtain multi-scale features of the face region in the original image. , , )and( , , The formula is as follows:
[0062] (1)
[0063] (2)
[0064] In this network, the encoder is an Xception network, E represents the feature vector of the first stage EntryFlow of the Xception network, M represents the feature vector of the second stage Middle Flow of the Xception network, and O represents the feature vector of the third stage Exit Flow of the Xception network.
[0065] In step 5, the multi-scale attention interaction module MAI first uses an attention mechanism to extract attention from features at multiple scales, and obtains the relevant attention between different features by multiplying them in pairs. Finally, it performs feature fusion by adding the features together.
[0066] In the multi-scale attention interaction module, each input ( , , All features are processed through an SE Attention module, and then through an adaptive average pooling layer to reduce the size of the feature map to 7. 7 pixels; then, a 1x1 Conv2D layer adjusts the dimension of its features to 2048, followed by batch normalization to obtain a dimension of 7. 7 Attention features of 2048; input obtained in this structure , , attention characteristics , , Then, a pairwise method is used to calculate their cross-attention; and Pixel-wise multiplication to obtain cross-attention features Similarly, calculate ( , )and( , ) combination to obtain and Through the above steps, , and Integrated information at different scales; , and The features are concatenated together, then the feature dimension is adjusted to 2048 through a 1x1 Conv2D layer; finally, a batch normalization layer is used to obtain multi-scale attention features. And as the output of the multi-scale attention interaction module, the formula can be expressed as:
[0067] (3)
[0068] For the input ( , , Similarly, multi-scale attention features are obtained. The formula can be expressed as:
[0069] (4)
[0070] In step 6, the results obtained in step 5 are... and Each input is a projection layer, which is a simple multilayer perceptron structure. Through these projection layers, the representational features of the face region in the original image are obtained. and enhance the representation features of the face region in the image. By representing the vector and The images are concatenated and then fed into a prediction head called Prediction to obtain true / false predictions of the input image. The entire process can be represented by the following formula:
[0071] (5)
[0072] (6)
[0073] (7)
[0074] (8)
[0075] The model has three losses: unsupervised contrastive learning loss. Supervised comparative learning loss and classification loss ;
[0076] Unsupervised contrastive learning loss It can be defined as: for a containing A small batch of the original images, Represents the index of all images (original image and enhanced image); if the index If it is an original image, then the index It is an index Data augmentation images, and vice versa; for unsupervised contrastive learning, the index... The corresponding sample is called the anchor point, index. The corresponding sample is called a positive sample; other indexes in the batch The samples that are negative are called negative samples, and the number of negative samples is ;index The representational features of a sample are denoted as Therefore, comparing learning loss It can be represented as:
[0077]
[0078] in Indicates the inner product. It is a temperature parameter;
[0079] Among them, supervised contrastive learning loss This can be defined as: all samples of the same category in a batch, including both original and enhanced images, are defined as positive samples, denoted as . , It is its cardinality; similarly, samples from different categories are considered negative samples, denoted as . Therefore, supervised contrastive learning loss It can be represented as:
[0080]
[0081] For a mini-batch of all the original images, where Indicates the index of the original image; for an index of The input, its label is The predicted probability is Cross-entropy is used to quantify the difference between predicted values and labels; cross-entropy loss. It can be defined as:
[0082]
[0083] Total loss of the model It can be defined as:
[0084] .
[0085] Specific implementation examples are as follows:
[0086] The development language is Python, the development environment is Ubuntu, and the deep learning framework used is PyTorch. The datasets used are Faceforensics++ and Celeb-DF-V2. The Faceforensics++ dataset contains 5000 videos, including 1000 real videos and 4000 fake videos. The 4000 fake videos are divided into four types: NeuralTextures, DeepFakes, Face2Face, and FaceSwap, with 1000 fake videos from each type. The Celeb-DF-V2 dataset contains 6529 videos, including 890 real videos and 5639 fake videos.
[0087] The specific steps are as follows:
[0088] Step 1: Data Input
[0089] The input data can be video or images, and the output is extracted face region images. If the input is video, it is first converted into frames. OpenCV is used to segment the video into frames, and 30 frames are randomly selected from each video for face region cropping. The frames or images are then processed using DIB (Digital Embedded Boundaries) to obtain key facial regions, which are then enlarged by a factor of 1.2 for image cropping to obtain face region images. These face region images are then divided into training, validation, and test sets in an 8:1:1 ratio.
[0090] Step Two: Face Representation
[0091] Step two takes a face image as input and outputs an image representation vector of the face image. The input image size is then adjusted to 299. 299 3. The augmented image is obtained through data augmentation. The original image and the augmented image are distributed and input into the Xception network to obtain the three-level features of each image, as shown in formulas (1) and (2). By inputting the three-level feature distribution of the encoder of the original image and the augmented image into the multi-scale attention interaction module (MAI), as shown in formulas (3) and (4), the multi-scale attention features of the image are obtained. The multi-scale attention features are represented by the same representation layer (Projection) to obtain the representation vectors of the face in the original image and the face in the augmented image, as shown in formulas (5) and (6).
[0092] Step 3: Deepfake Image Prediction
[0093] Step 3 takes the face image representation vector as input and outputs the depth-spoofing probability value of the original image. The face representation vectors of the original image and the enhanced image are concatenated, and the depth-spoofing prediction probability value of the set of images is obtained through a prediction head, as shown in formulas (7) and (8). The range of the probability value is [0, 1]; where a probability value greater than or equal to 0.5 represents a real image, and a probability value less than 0.5 represents a depth-spoofing image.
[0094] Step 4: Model Training
[0095] Using the PyTorch framework, follow steps one, two, and three. Figure 1 The model was trained for 30 epochs with a batch size of 64, using the Adam optimizer and an initial learning rate of 2e. -4 In the first four epochs, the learning rate is warmed up using a warmup method. During training, the learning rate is halved every five epochs. The model uses AUC as the evaluation metric, and the validation set AUC and the current model parameters are recorded for each epoch.
[0096] Step 5: Model Testing
[0097] The model with the highest AUC in step four will be used as the test model. To test the robustness of the model, the model trained in Faceforensics++ will be tested on the Celeb-DF-V2 dataset. The predicted probability value for each image is in the range of [0, 1]; where a probability value greater than or equal to 0.5 indicates a real image, and a probability value less than 0.5 indicates a deepfake image.
[0098] Contents not described in detail in this specification are prior art known to those skilled in the art. Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A deep forgery detection method based on balanced contrastive learning, characterized in that: The method is based on a deep forgery detection model with a balanced contrastive learning strategy. The model combines unsupervised and supervised contrastive learning strategies, expands the data using data augmentation methods, and enhances the deep forgery classification effect by jointly using supervised and unsupervised contrastive loss. The model comprises a Siamese encoder, two multi-scale attention interaction modules (MAI), two representation modules (Projection), and a prediction head (Prediction). Its steps are as follows: Step 1: For the input video, first divide it into video frames, and randomly select video frames for face region cropping; Step 2: For the input image, crop the face region; Step 3: For the cropped face region, data augmentation is used to obtain the enhanced face region image; Step 4: Pass the face regions of the original image and the face regions of the enhanced image through a Siamese network encoder to obtain multi-scale features of the original image and the enhanced image respectively. Step 5: Input the multi-scale features of the original image and the multi-scale features of the enhanced image into the multi-scale attention interaction module to obtain the multi-scale attention fusion features of the face region of the original image and the face region of the enhanced image; In step 5, the multi-scale attention interaction module MAI first uses the SE Attention mechanism to extract attention from multiple scale features, and obtains the relevant attention between different features by multiplying them in pairs, and finally fuses the features by adding the features. Step 6: The fused features are passed through a representation layer and then concatenated into a prediction head to obtain the results of deepfake detection.
2. The deep forgery detection method based on balanced contrastive learning according to claim 1, characterized in that: In step 1, for the input video, the video is first segmented into frames using the OpenCV tool, and a frame is randomly selected for face region cropping; the face region is obtained through dlib, and after the face region is enlarged by 1.2 times, the image is cropped to obtain the face region to be detected.
3. The deep forgery detection method based on balanced contrastive learning according to claim 1, characterized in that: In step 2, for the input image, the face region is obtained using dlib, and the face region is enlarged by 1.2 times and then cropped to obtain the face region to be detected.
4. The deep forgery detection method based on balanced contrastive learning according to claim 1, characterized in that: In step 3, data augmentation employs random flipping (RF), random resizing and cropping (RRC), and random erasure (RE); the probability of the RF is set to 0.5; the RRC uses a scaling factor (1 / 1.3, 1) and an aspect ratio (0.9, 1.1); and the RE uses a scaling factor (0.02, 0.2), an aspect ratio (0.5, 2), and a probability of 0.
5.
5. The deep forgery detection method based on balanced contrastive learning according to claim 1, characterized in that: In step 4, for the face region of the input original image and enhance the face region of the image The encoders respectively obtain multi-scale features of the face region in the original image. , , )and( , , The formula is as follows: In this context, the encoder is an Xception network, E represents the feature vector of the first stage Entry Flow of the Xception network, M represents the feature vector of the second stage Middle Flow of the Xception network, and O represents the feature vector of the third stage Exit Flow of the Xception network.
6. The deep forgery detection method based on balanced contrastive learning according to claim 1, characterized in that: In the multi-scale attention interaction module, each input ( , , All features are processed through an SE Attention module, and then through an adaptive average pooling layer to reduce the size of the feature map to 7. 7 pixels; then, a 1x1 Conv2D layer adjusts the dimension of its features to 2048, followed by batch normalization to obtain a dimension of 7. 7 Attentional characteristics of 2048; The input was obtained in this structure. , , attention characteristics , , ; Then, a pairwise approach is used to calculate their cross-attention; and Pixel-wise multiplication to obtain cross-attention features ; Similarly, calculate ( , )and( , ) combination to obtain and Through the above steps, , and Integrated information at different scales; , and The features are concatenated together, then the feature dimension is adjusted to 2048 through a 1x1 Conv2D layer; finally, a batch normalization layer is used to obtain multi-scale attention features. And as the output of the multi-scale attention interaction module, the formula can be expressed as: For the input ( , , Similarly, multi-scale attention features are obtained. The formula can be expressed as: 。 7. The deep forgery detection method based on balanced contrastive learning according to claim 6, characterized in that: In step 6, the material obtained in step 5 will be... and Each input is a projection layer, which is a simple multilayer perceptron structure. Through these projection layers, the representational features of the face region in the original image are obtained. and enhance the representation features of the face region in the image. By representing the vector and The images are concatenated and then fed into a prediction head called Prediction to obtain true / false predictions of the input image. The entire process can be represented by the following formula: 。 8. The deep forgery detection method based on balanced contrastive learning according to claim 7, characterized in that: The model has three losses: unsupervised contrastive learning loss. Supervised comparative learning loss and classification loss ; Unsupervised contrastive learning loss It can be defined as: for a containing A small batch of the original images, This represents the index of all images, including both original and enhanced images; if the index... If it is an original image, then the index It is an index Data augmentation images, and vice versa; for unsupervised contrastive learning, the index... The corresponding sample is called the anchor point, index. The corresponding sample is called a positive sample; other indexes in the batch The samples that are negative are called negative samples, and the number of negative samples is ;index The representational features of a sample are denoted as Therefore, comparing learning loss It can be represented as: in Indicates the inner product. It is a temperature parameter; Among them, supervised contrastive learning loss This can be defined as: all samples of the same category in a batch, including both original and enhanced images, are defined as positive samples, denoted as . , It is its cardinality; Similarly, samples from different categories are considered negative samples, denoted as . Therefore, supervised contrastive learning loss It can be represented as: For a mini-batch of all the original images, where Indicates the index of the original image; for an index of The input, its label is The predicted probability is Cross-entropy is used to quantify the difference between predicted values and labels; cross-entropy loss. It can be defined as: Total loss of the model It can be defined as: 。