Deep fake detection method based on local-global self-supervised contrastive learning

Abstract

The present application relates to the technical field of deep fake detection, in particular to a deep fake detection method based on local-global self-supervised contrastive learning; the method comprises the following steps: firstly, collecting unlabelled face images and labelled real-fake data and uniformly preprocessing, including face detection, extension cropping and key point positioning; secondly, dividing two global perspectives and seven local perspectives according to the key points, and constructing multi-view contrastive learning samples; further, designing a self-supervised pre-training process based on a teacher-student framework, learning robust face representation unsupervisedly by comparing the consistency of local and global features and reconstructing mask image blocks; finally, supervisedly fine-tuning the pre-trained model, and optimizing the classification structure by using neural network architecture search. The present application realizes high-precision and strong-generalization deep fake detection.

Claims

1. A deep fake detection method based on local and global self-supervised contrastive learning, characterized in that, The specific implementation steps include the following: S1. Construct and preprocess the face data required for model training. Collect diverse unlabeled face images to form a pre-training set, and integrate real and fake images from multiple sources to form a labeled fine-tuning set. Then, perform face detection, extended cropping, and 68-point key point localization on all images to provide standardized input data for subsequent viewpoint segmentation and model training. S2. Based on the preprocessed face images and key points, several face viewpoints are systematically divided. By generating two global viewpoints containing the whole and random background sub-regions, as well as local viewpoints covering seven key regions including the whole face, forehead, eyes, nose, mouth and jaw, rich multi-view data pairs are constructed for self-supervised contrastive learning. S3. Design a self-supervised contrastive pre-training method based on a teacher-student framework. By having the student network learn to align local viewpoints with global viewpoints provided by the teacher network and reconstruct randomly masked global face blocks, the model learns robust face representations that combine local detail sensitivity with global consistency in an unsupervised manner. S4. Supervised fine-tuning of the pre-trained base model is performed, and a neural network architecture is used to search for and optimize the fine-tuning structure. By systematically searching for multi-scale feature fusion methods, feature token selection, pooling operations, and learning rate parameter combinations, the optimal fine-tuning configuration for deep pseudo-detection tasks is automatically found, thereby improving the accuracy and generalization ability of the final model.

2. The deep fake detection method based on local and global self-supervised contrastive learning according to claim 1, characterized in that, In step S1, face detection and extended cropping specifically involve: The YOLO-Face face detection model is used to detect and locate the face region in the input image. The bounding box of the region is then expanded to 2.5 times its original size before cropping to obtain the cropped face image. The key point localization specifically involves using the dlib face key point detector to accurately locate sixty-eight feature points in the cropped face image.

3. The deep fake detection method based on local and global self-supervised contrastive learning according to claim 2, characterized in that, In step S2, the two global perspectives include: The first global perspective is the cropped face image itself; The second global perspective is obtained by randomly cropping a sub-region from the cropped face image; The scaling factor during random cropping ranges from 1.5 to 2.5, ensuring that the sub-region includes part of the face and part of the random background.

4. The deep fake detection method based on local and global self-supervised contrastive learning according to claim 3, characterized in that, In step S2, the division of the local viewpoints of the seven key regions is based on the spatial coordinates of sixty-eight key points. The images of the whole face, forehead, left eye, right eye, nose, mouth and jaw are cropped out respectively. When cropping each region, the minimum bounding rectangle determined by the key points of the region is randomly expanded outward by a preset multiple to increase the diversity of data.

5. The deep fake detection method based on local and global self-supervised contrastive learning according to claim 4, characterized in that, In step S3, the teacher-student framework uses the Vision Transformer Base / 14 architecture as the feature encoder. Initially, the student encoder and the teacher encoder have the same structure and parameters. During training, the student encoder receives a sequence of image patches from local perspectives and a sequence of global perspectives from image patches that are randomly occluded by 50%, while the teacher encoder receives a complete sequence of global perspective image patches as input.

6. The deep fake detection method based on local and global self-supervised contrastive learning according to claim 5, characterized in that, In step S3, self-supervised contrastive pre-training is achieved by minimizing the joint loss function, which includes global-local contrastive loss and mask reconstruction loss. The global-local contrast loss is the similarity loss between the classification token features extracted by the student encoder from a local perspective and the classification token features extracted by the teacher encoder from a global perspective; the mask reconstruction loss is the feature difference loss between the predicted features of the student encoder for the occluded global image patch and the corresponding real image patch features provided by the teacher encoder.

7. A deep fake detection method based on local and global self-supervised contrastive learning according to claim 6, characterized in that, In step S3, the joint loss function is a weighted sum of the global-local contrast loss and the mask reconstruction loss, with weight coefficients of 0.6 and 0.4, respectively. During training, the student encoder parameters are updated using gradient descent, while the teacher encoder parameters are smoothly updated from the student encoder using an exponential moving average strategy.

8. The deep fake detection method based on local and global self-supervised contrastive learning according to claim 7, characterized in that, In step S4, the supervised fine-tuning specifically involves: freezing the pre-trained student encoder as a fixed feature extractor, and converting the image to be detected into a sequence of image patches to be input into the encoder; The search space for neural network architecture search includes four optional dimensions: multi-scale feature sources, type of feature tokens used, whether to add pooling layers, and the learning rate of the classification head.

9. A deep fake detection method based on local and global self-supervised contrastive learning according to claim 8, characterized in that, In step S4, the neural network architecture search adopts a sequential search strategy, trying different feature fusion schemes, structural configurations and learning rate combinations in turn. For each combination, only the classification head is trained and the performance is evaluated on the validation set. Finally, the configuration with the highest accuracy on the validation set is selected as the optimal model architecture, and the configuration is used to perform final training on complete labeled data to obtain the deep forgery detection model.

Images