Deep forgery detection method and system based on facial embedding difference guidance

Abstract

The invention belongs to an image processing technology, and particularly relates to a deep forgery detection method and system based on face embedded difference guidance, and the method comprises the steps: generating a forgery image according to an input real image, and calculating a difference image and a semantic difference vector of two images; learning the features of the real image and the forged image by using a first classifier, and learning the features of the real image and the difference image by using a second classifier; splicing and fusing the semantic difference vector and image features extracted from the difference image through a channel to obtain fused features, and learning the difference between the real face features and the fused features by using a third classifier; and calculating the distillation loss between the fusion feature and the image feature of the forged image, and training in combination with the loss of each classifier. According to the method, multi-level feature fusion, a three-classifier architecture and a knowledge distillation mechanism are adopted, the strong generalization ability across data sets is achieved, and the system can output a high-precision deep forgery judgment result.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a deep fake detection method and system based on facial embedding difference guidance. Background Art

[0002] With breakthroughs in AIGC technologies like generative adversarial networks (GANs) and diffusion models, deepfakes have generated images and videos that are indistinguishable from the real thing. This has positive implications for artistic creation, film and television entertainment, and other fields. However, these technological breakthroughs have also spawned new digital security threats: malicious actors can create deepfakes that generalize across datasets, leading to systemic risks such as the spread of false information and biometric spoofing, seriously threatening the security boundaries of digital identity authentication systems and the foundation of social trust. To address the threat posed by deepfakes, various deepfake detection methods have been proposed.

[0003] In the research field, existing deep fake detection methods can be mainly summarized into the following three categories:

[0004] (1) Forgery detection methods based on visual features: Early deep fake detectors mostly relied on learning some specific features, such as unnatural visual traces and subtle expressions that are lacking in human faces; time domain-based analysis methods mostly analyzed the temporal dynamics and spatial consistency between video frames, such as the FTCN network, which simultaneously analyzed the spatial features of a single frame and the temporal dynamics across frames to detect the incoherence of facial expressions and head movements, and the UMFC method introduced a multi-scale spatiotemporal attention module to focus on the abnormal movement patterns of key facial areas (such as eyes and mouth) on the time axis; frequency domain-based analysis methods used frequency domain transforms (DCT, wavelet transform, etc.) to reveal periodic noise or spectral anomalies in the frequency domain of forged images, such as the IAW algorithm that enhances the high-frequency noise differences in forged areas through stationary wavelet decomposition (SWD).

[0005] (2) Detection methods based on high-level semantic features: Detection is performed using the inconsistency of high-level semantic features such as identity and expression. For example, the ICT method models the identity difference between the inner and outer regions of the face. The ID-Reveal method uses metric learning and adversarial training strategies to help the model learn the unique temporal characteristics of facial movements during speech. Xu et al. introduced the R-MFDN method, which learns identity-aware features for each modality by comparing its features with those of non-matching identities and detects forgeries by exploiting the inconsistency between different modalities.

[0006] (3) Detection methods based on data synthesis: By synthesizing training data containing typical forgery artifacts, the model's generalization ability for unknown forgery types is improved. For example, the self-mixing image generation method (SBI) simulates various forgery artifacts such as edge blur and fiber optic unevenness by performing affine transformation and mixing on real images; Face X-ray generates forged images through boundary fusion, forcing the model to focus on artifacts at the edges of facial fusion.

[0007] In summary, current deepfake detection methods primarily rely on data augmentation and specific artifact features, but suffer from two key limitations: Detection methods that only learn specific visual artifacts struggle to capture rich forgery cues; while detection methods that rely on synthesizing existing images rely on full RGB images and are susceptible to background interference. Furthermore, deepfake manipulations induce feature discrepancies in the embedding space of facial recognition models, and these high-level semantic features are not effectively utilized by existing methods. Summary of the Invention

[0008] To improve the accuracy of identifying forged images, this paper proposes a deepfake detection method based on facial embedding difference guidance. A classifier is used to identify whether an input image is forged. Training the classifier includes the following steps:

[0009] The self-mixing image generator generates a fake image based on the input real image, calculates the difference between the two images, and obtains the differential image;

[0010] Using an image feature extractor to extract image features from the real image and the forged image respectively, using a first classifier to learn the features of the two images, and calculating the loss based on the classification results of the first classifier;

[0011] Extract image features from the real image and the difference image respectively, use the second classifier to learn the features of the two images, and calculate the loss based on the classification results of the second classifier;

[0012] Extract high-level semantic features from real images and fake images respectively, and use the difference between the two high-level semantic features as the semantic difference vector;

[0013] The semantic difference vector is fused with the image features extracted from the difference image through channel concatenation to obtain a fused feature. A third classifier is used to learn the difference between the real face features and the fused feature, and the loss is calculated based on the classification results.

[0014] Calculate the distillation loss between the fused features and the image features of the forged image;

[0015] All losses are used to train the image feature extractor and the first classifier, and the trained image feature extractor and the first classifier are used to determine whether the input real-time image is a forged image.

[0016] The present invention also proposes a deep fake detection system based on facial embedding difference guidance, which is used to implement a deep fake detection method based on facial embedding difference guidance. The system includes a real-time data processing subsystem and a training subsystem. The real-time data processing subsystem includes a pre-trained image feature extractor and a first classifier. The image feature extractor is used to extract features of the real-time image, and the first classifier is used to determine whether the input real-time image is a forged image. The training subsystem is used to train the image feature extractor and the first classifier. The training subsystem includes:

[0017] An image feature extractor, used to extract image features from an image;

[0018] Image semantic feature extractor, used to extract semantic features from images;

[0019] a first classifier for classifying the real image or the forged image based on image features extracted;

[0020] a second classifier for classifying based on image features extracted from the real image or the difference image;

[0021] a differentiator for calculating a difference image between a real image and a forged image, and calculating a semantic feature difference vector between the semantic feature vectors of the real image and the forged image;

[0022] A fusion module is used to fuse the semantic feature difference vector and the differential image to obtain a fusion feature;

[0023] The third classifier is used to classify based on the real image or fusion features;

[0024] a loss function calculation module, configured to calculate a total loss function based on the first classifier, the second classifier, the third classifier, and the distillation loss between the fusion feature and the image feature of the real image;

[0025] The optimizer is used to train the image feature extractor, the first classifier, the second classifier, and the third classifier according to the loss function.

[0026] The present invention constructs a deepfake detection framework by fusing multi-level features with knowledge distillation constraints, combines a pre-trained deep convolutional network to extract global image features and residual artifact features, and introduces a face recognition network to extract high-level semantic differences. Through cross-level feature fusion and multi-task joint optimization strategies, it achieves comprehensive capture of deepfake clues and improves cross-domain generalization capabilities. The present invention has the following beneficial effects:

[0027] 1. High Precision and Strong Generalization: By combining global feature differences, local artifact features, and high-level identity semantic differences, this method effectively integrates multi-dimensional forgery cues. In tests on cross-datasets such as FaceForensics++ and Celeb-DF, the average detection accuracy has been improved to 98.2%, significantly alleviating the overfitting problem caused by traditional methods' reliance on a single feature.

[0028] 2. Low computational overhead: Based on the feature reuse mechanism of the pre-trained EfficientNet-b4 network, multi-branch feature extraction can be completed with only a single forward propagation. The inference speed is improved compared to similar multi-model fusion solutions, and it can be deployed on edge devices for real-time detection.

[0029] 3. Strong anti-interference ability: Through residual artifact enhancement and feature distillation constraints, the model's robustness to interference factors such as illumination changes and compression artifacts is significantly enhanced. It can still maintain a detection accuracy of over 90% in strong noise scenes, making it suitable for complex real-world scenarios such as social media and video surveillance.

[0030] 4. Interpretability Optimization: Combining the visualization analysis of identity difference vectors and spatial artifact features, this technology supports the localization and interpretation of counterfeit areas, provides auxiliary decision-making basis for manual review, and promotes the development of detection technology from a "black box" to a trustworthy one. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the deep fake detection method based on facial embedding difference guidance of the present invention;

[0032] Figure 2 This is a flow chart of the deep fake detection method based on facial embedding difference guidance of the present invention;

[0033] Figure 3 This is a schematic diagram of the deep fake detection method guided by facial embedding differences in the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] This paper proposes a deep fake detection method based on facial embedding difference guidance, which uses a classifier to identify whether the input image is a fake image. Figures 1-2 , training the classifier includes the following steps:

[0036] The self-mixing image generator generates a fake image based on the input real image, calculates the difference between the two images, and obtains the differential image;

[0037] Using an image feature extractor to extract image features from the real image and the forged image respectively, using a first classifier to learn the features of the two images, and calculating the loss based on the classification results of the first classifier;

[0038] Extract image features from the real image and the difference image respectively, use the second classifier to learn the features of the two images, and calculate the loss based on the classification results of the second classifier;

[0039] Extract high-level semantic features from real images and fake images respectively, and use the difference between the two high-level semantic features as the semantic difference vector;

[0040] The semantic difference vector is fused with the image features extracted from the difference image through channel concatenation to obtain a fused feature. A third classifier is used to learn the difference between the real face features and the fused feature, and the loss is calculated based on the classification results.

[0041] Calculate the distillation loss between the fused features and the image features of the forged image;

[0042] All losses are used to train an image feature extractor, a first classifier, a second classifier, and a third classifier, and the trained image feature extractor and the first classifier are used to determine whether the input real-time image is a forged image.

[0043] As an optional manner, the method of the present invention includes generating a forged image based on an input real image, calculating a differential image and a semantic difference vector between the two images; using a first classifier to learn features of the real image and the forged image, and using a second classifier to learn features of the real image and the difference image; fusing the semantic difference vector with image features extracted from the differential image through channel splicing to obtain a fused feature, and using a third classifier to learn the difference between real face features and the fused feature; calculating the distillation loss between the fused feature and the image features of the forged image, and training in combination with the losses of each classifier.

[0044] The purpose of the present invention is to provide a deepfake detection algorithm guided by facial information differences, and to provide a deepfake detection method and system guided by facial information differences. The method generates a synthetic image containing common artifacts through self-mixing technology, and extracts low-level spatial artifacts present in the residual map of the real image and the synthetic image; at the same time, a pre-trained face recognition model is used to calculate the facial embedding difference between the real image and the synthetic image, capturing high-level semantic inconsistencies such as identity features. The innovation of the present invention lies in the design of a cross-modal distillation loss, which deeply fuses the local texture anomalies of the residual map with the global semantic deviation of the embedding difference, allowing the detector to simultaneously perceive the low-level artifact features and high-level semantic difference features of cross-domain forgeries. Then, through multi-classifier collaborative optimization, it is finally possible to achieve high deepfake detection accuracy with only a single image input.

[0045] In this embodiment, the EfficientNet-b4 deep convolutional network encoder is used to extract features from the real face image and the self-mixing forged image, and the image features f corresponding to the original face image are obtained. i r The image feature f corresponding to the self-mixing forged image i f Then, the difference image is constructed by performing a difference operation between the real image and the forged image, and the EfficientNet-b4 network with the same structure is used to extract the difference image to obtain its image feature f i Δ , the image feature has low-level spatial artifact features, and introduces cross entropy loss to optimize the local artifact classification of the real image and residual image pairs.

[0046] In this embodiment, the high-level semantic features of the real image and the forged image are extracted by pre-training the ArcFace network, and the difference is calculated to obtain the high-level semantic difference feature. The low-level spatial artifact feature and the high-level semantic difference feature are cascaded by combining the MLP connector to generate the fusion feature f i * , the cross entropy loss is introduced to optimize the artifact classification of real image features and fused features. At the same time, the distillation loss is introduced to constrain the distribution consistency of fused features and forged features through the L2 norm, prompting the model to learn more comprehensive forgery clues, so that the inference stage supports single image input.

[0047] The total loss function used in the present invention is expressed as:

[0048]

[0049] Among them, L total represents the total loss function of the self-mixing based image generator; represents the cross entropy loss between real and fake images; Represents the cross entropy loss between the real image and the residual image; represents the cross entropy loss between the real image and the fusion feature; L distill represents the distillation loss of the fusion feature and the corresponding feature of the forged image; λ1, λ2, and λ3 are L distill The weight coefficient of .

[0050] As an optional implementation, the cross entropy loss between real and forged images is Expressed as:

[0051]

[0052] Among them, N represents the number of samples during training; y i Represents the true value label of the i-th sample. If the input is a real image of the sample, the true value label value of the sample is 1. If the input is a fake image of the sample, the true value label value of the sample is 0. For the loss function Training with data pairs consisting of real and fake images; The classification probability of the image features representing the forged image or the real image features calculated by the first classifier; the loss function is used to optimize the classification of real and forged images so that the model can distinguish real images from forged images based on global features.

[0053] As an optional implementation, the cross entropy loss between the real image and the residual image is Expressed as:

[0054]

[0055] Among them, y i Represents the true value label of the i-th sample. If the input is the real image of the sample, the true value label value of the sample is 1. If the input is the differential image of the sample, the true value label value is 0. For the loss function Use data pairs consisting of real images and difference images for training; Represents the classification probability of the difference image or the real image calculated by the second classifier; loss Refine the classification of real images and difference images by optimizing The model is guided to recognize complete and pure forgery features, improving its ability to identify subtle deepfake artifacts.

[0056] As an optional implementation, the cross entropy loss between the real image and the fused features Expressed as:

[0057]

[0058] Among them, y i Represents the true value label of the i-th sample. If the input is the real image of the sample, the true value label value of the sample is 1. If the input is the fusion feature of the sample, the true value label value is 0. For the loss function Use data pairs consisting of real images and fusion features for training; Represents the classification probability of the fused feature vector or the real image calculated by the third classifier; by optimizing The model is guided to identify more comprehensive forgery features, improving the generalization ability of the model.

[0059] As an optional implementation, the distillation loss L between the fused features and the features corresponding to the forged image is distill Expressed as:

[0060]

[0061] Among them, f i * represents the fusion feature corresponding to the i-th training sample; f i f represents the image features of the forged image corresponding to the i-th training sample; ‖·‖2 represents the calculation of the L2 norm, which is used to constrain the similarity between the fusion artifact features and the features of the self-mixed forged image. The training is performed using data consisting of fused features and forged images; in the present invention, a sample includes an original face image and the forged image, differential image, and fused features corresponding to the face image.

[0062] like Figure 3 This embodiment describes the data processing process in the training phase and the real-time detection phase respectively. The training phase includes the following steps:

[0063] In the training phase, first, the real image of the i-th sample Self-mixing based image generator generates corresponding fake images

[0064] The real image of the i-th sample Subtract its fake image Get the corresponding difference image

[0065] The real image of the i-th sample is obtained by EfficientNet-b4 deep convolutional network encoder. Fake images Difference image The corresponding feature vectors f are extracted from ir (i.e., real face features), f i f (i.e. forging facial features), f i Δ (i.e. residual image features);

[0066] The first classifier W rf Based on the input real face features and forged face features, the first classifier is used to determine the probability that the input features belong to a real image or a forged image;

[0067] The second classifier W rp Learning is performed based on the input real face features and residual image features. The second classifier is used to determine the probability that the input features belong to the real image and the residual image.

[0068] The high-level semantic features of the real face and the forged face are extracted from the pre-trained ArcFace network respectively. The features obtained by subtracting the two high-level semantic features and the features of the residual image are spliced on the channel and then input into the MLP for cross-modal fusion to obtain the fused features.

[0069] The third classifier W rc Based on the input, the fusion features and the real face features are learned, and the third classifier is used to predict the probability that the input features belong to the real face and the fusion features;

[0070] Calculate the cross entropy between each classifier and the true label corresponding to the input feature as the loss of the classifier, and then calculate the L2 norm between the forged face feature and the fused feature as the distillation loss. Use all losses to train and optimize the classifier and image feature extractor;

[0071] Finally, the trained feature extractor and the first classifier are deployed on the device.

[0072] The feature extractor and the first classifier device are deployed for real-time detection, which includes the following steps:

[0073] Use the trained EfficientNet-b4 deep convolutional network encoder to extract image features from the image to be detected;

[0074] The image features of the image to be detected are input into the first classifier, and the first classifier outputs the probability that the image is a real image or a forged image. Generally, the classifier is used to judge the probability that the input sample is a real image. If the classifier judges that the probability that the sample is a real image is greater than the set threshold (generally, the threshold is set to 0.5), then the sample is a real image, otherwise the sample is a forged image.

[0075] The present invention also proposes a deep fake detection system based on facial embedding difference guidance, which is used to implement a deep fake detection method based on facial embedding difference guidance. The system includes a real-time data processing subsystem and a training subsystem. The real-time data processing subsystem includes a trained image feature extractor and a first classifier. The image feature extractor is used to extract features of the real-time image, and the first classifier is used to determine whether the input real-time image is a forged image. The training subsystem is used to train the image feature extractor and the first classifier. The training subsystem includes:

[0076] An image feature extractor, used to extract image features from an image;

[0077] Image semantic feature extractor, used to extract semantic features from images;

[0078] a first classifier for classifying the real image or the forged image based on image features extracted;

[0079] a second classifier for classifying based on image features extracted from the real image or the difference image;

[0080] a differentiator for calculating a difference image between a real image and a forged image, and calculating a semantic feature difference vector between the semantic feature vectors of the real image and the forged image;

[0081] A fusion module is used to fuse the semantic feature difference vector and the differential image to obtain a fusion feature;

[0082] The third classifier is used to classify based on the real image or fusion features;

[0083] a loss function calculation module, configured to calculate a total loss function based on the first classifier, the second classifier, the third classifier, and the distillation loss between the fusion feature and the image feature of the real image;

[0084] The optimizer is used to train the image feature extractor, the first classifier, the second classifier, and the third classifier according to the loss function.

[0085] Table 1

[0086]

[0087] In order to verify the effectiveness of the present invention, the present invention also compares the present invention with other existing detection methods, as shown in Table 1. Table 1 shows the detection accuracy AUC of existing methods such as Face X-ray, FTCN, SPSL and this method on the training set FF++ and multiple test sets (CDF2, DFD, DFDC, etc.). The closer this value is to 100%, the better the detection performance of the method. The AUC of the present application method on the FF++ dataset reached 99.68%, which is higher than most similar methods; in the cross-dataset scenario, the average AUC of the present application method reached 87.86%, which is higher than the current comparison method, indicating that the present application method can perform stably on different datasets and has good generalization ability and effectiveness.

[0088] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A deep fake detection method based on facial embedding difference guidance, which uses a classifier to identify whether the input image is a fake image, characterized by: Training the classifier consists of the following steps: The self-mixing image generator generates a fake image based on the input real image, calculates the difference between the two images, and obtains the differential image; Using an image feature extractor to extract image features from the real image and the forged image respectively, using a first classifier to learn the features of the two images, and calculating the loss based on the classification results of the first classifier; Extract image features from the real image and the difference image respectively, use the second classifier to learn the features of the two images, and calculate the loss based on the classification results of the second classifier; Extract high-level semantic features from real images and fake images respectively, and use the difference between the two high-level semantic features as the semantic difference vector; The semantic difference vector is fused with the image features extracted from the difference image through channel concatenation to obtain a fused feature. A third classifier is used to learn the difference between the real face features and the fused feature, and the loss is calculated based on the classification results. Calculate the distillation loss between the fused features and the image features of the forged image; All losses are used to train the image feature extractor, the first classifier, the second classifier, and the third classifier, and the trained image feature extractor and the first classifier are used to determine whether the input real-time image is a forged image.

2. The method according to claim 1, characterized in that The image feature extractor uses the EfficientNet-b4 deep convolutional network encoder.

3. The method according to claim 1, characterized in that A pre-trained ArcFace network is used to extract high-level semantic features from images.

4. The method according to claim 1, wherein The total loss of the self-mixing based image generator is expressed as: Among them, L total represents the total loss function of the self-mixing based image generator; represents the cross entropy loss between the original face image and the self-mixed forged image; Represents the cross entropy loss between the original face image and the residual image; represents the cross entropy loss between the original face image and the fusion feature; L distill represents the distillation loss of fusion features and forged features; λ1, λ2, and λ3 are L distill The weight coefficient of .

5. The method according to claim 1 or 4, characterized in that Cross entropy loss of original face images and self-mixed forged images Expressed as: Among them, N represents the number of samples during training; y i Represents the true value label of the i-th sample; The classification probability of the image feature of the forged image or the image feature of the real image calculated by the first classifier.

6. The method according to claim 1 or 4, characterized in that Cross entropy loss between original face image and residual image Expressed as: Among them, N represents the number of samples during training; y i Represents the true value label of the i-th sample; Indicates the classification probability of the features of the residual image or the features of the true image calculated by the second classifier.

7. The method according to claim 1 or 4, characterized in that Cross entropy loss between the original face image and the fused features Expressed as: Among them, N represents the number of samples during training; y i Represents the true value label of the i-th sample; Indicates the classification probability of the fused features or the features of the real image calculated by the third classifier.

8. The method according to claim 1 or 4, characterized in that Distillation loss L of fused features and forged features distill Expressed as: Where N is the number of samples during training; f i * represents the fused forged feature corresponding to the i-th training sample; f i f represents the image features of the self-mixing forged image corresponding to the i-th training sample; ‖·‖2 represents the calculation of the L2 norm.

9. A deep fake detection system guided by facial embedding differences, characterized by: A method for detecting deep fakes based on facial embedding difference guidance according to claim 1, comprising a real-time data processing subsystem and a training subsystem for completing training, wherein the real-time data processing subsystem comprises an image feature extractor and a first classifier, wherein the image feature extractor is used to extract features of a real-time image, and the first classifier is used to determine whether an input real-time image is a fake image; The training subsystem is used to train the image feature extractor and the first classifier. The training subsystem includes: An image feature extractor, used to extract image features from an image; Image semantic feature extractor, used to extract semantic features from images; a first classifier for classifying the real image or the forged image based on image features extracted; a second classifier for classifying based on image features extracted from the real image or the difference image; a differentiator for calculating a difference image between a real image and a forged image, and calculating a semantic feature difference vector between the semantic feature vectors of the real image and the forged image; A fusion module is used to fuse the semantic feature difference vector and the differential image to obtain a fusion feature; The third classifier is used to classify based on the real image or fusion features; a loss function calculation module, configured to calculate a total loss function based on the first classifier, the second classifier, the third classifier, and the distillation loss between the fusion feature and the image feature of the real image; The optimizer is used to train the image feature extractor, the first classifier, the second classifier, and the third classifier according to the loss function.

Images