Face deepfake detection method and system based on feature decoupling and contrastive learning

Abstract

The application provides a face deep fake detection method and system based on feature decoupling and contrast learning, which is independent of the forgery method. The method applies random augmentation to face images to destroy specific fake traces; a semantic visual encoder based on CLIP is used to extract semantic features and project them into a contrast space; global true and false sample contrast loss and different forgery method cohesion loss are calculated by combining a difficult sample mining mechanism; the global true and false separation and intra-class aggregation are balanced through multi-objective optimization, forcing the model to strip method-specific features and learn essential fake attributes. The application uses the semantic prior of CLIP and the feature decoupling mechanism to effectively solve the problem of overfitting to specific fake patterns in the prior art, significantly improving the generalization ability and robustness of the model to unknown fake methods.

Description

Technical Field

[0001] This invention relates to the field of face forgery detection technology, and more specifically, to a face deep forgery detection method and system based on feature decoupling and contrastive learning, independent of forgery methods. Background Technology

[0002] In recent years, with the rapid development of generative technologies (such as GANs (Generative Adversarial Networks), diffusion models, autoencoder models, etc.), digital image generation and editing technologies have achieved revolutionary breakthroughs. These technologies can generate visually highly realistic fake images or videos that are almost indistinguishable from real ones by the naked eye.

[0003] Existing methods for detecting face spoofing can be mainly divided into the following four categories:

[0004] Detection methods based on physiological and physical inconsistencies: These methods mainly focus on detecting anomalies that violate natural laws, such as unreasonable reflection of ambient light in the eyes, abnormal blinking, inconsistent lip movements, and facial lighting that violates physical laws. However, the latest generation of forgery technology can now simulate these unreasonable details very well.

[0005] Detection methods based on frequency domain analysis and texture features: detection is performed from the perspective of frequency domain artifacts or local texture anomalies; however, these high-frequency features are easily destroyed by post-processing, and different generation models will produce different frequency domain and texture features, resulting in weak generalization of this type of detection method.

[0006] Data-driven deep learning-based detection methods: This is currently a mainstream approach. It eliminates the need for manual feature design, utilizing large-scale labeled datasets to train deep convolutional neural networks end-to-end. This method can achieve extremely high accuracy on data from the same source. However, it also suffers from poor generalization performance, easily overfitting to forged features left by specific forgery methods in the training data. When faced with unknown forgery methods, its performance drops significantly.

[0007] In summary, the core flaw of existing technologies lies in the fact that the models fail to learn the universal and essential "forgery attributes" of forged samples. Instead, they overfit specific fingerprints generated by specific forgery methods in the training data, which have "forgery method patterns." For example, a certain GAN architecture may produce specific grid artifacts or frequency domain features. This results in the forgery detection model essentially being a "forgery method classifier" that works on a narrow data distribution, rather than a robust forgery detector that can cope with various threats, including unknown threats. Summary of the Invention

[0008] The present invention aims to solve at least one of the aforementioned technical problems existing in the prior art.

[0009] To this end, the first aspect of the present invention provides a face deepfake detection method that is independent of forgery methods based on feature decoupling and contrastive learning.

[0010] A second aspect of the present invention provides a face deepfake detection system that is independent of forgery methods based on feature decoupling and contrastive learning.

[0011] This invention provides a face deepfake detection method based on feature decoupling and contrastive learning, independent of forgery methods, comprising:

[0012] S1. Obtain the original dataset containing real face images and fake face images generated by various fake methods; perform face detection and alignment processing on the images in the original dataset, and apply random enhancement processing to the aligned face images to destroy specific fake traces, generating two different enhanced views for each image;

[0013] S2. Input two different augmented views corresponding to the same image into the semantic visual encoder module based on CLIP (Contrastive Language-Image Pre-trained Model) to extract visual feature representations rich in semantic information; the semantic visual encoder module is built based on the pre-trained CLIP model and trained using a progressive fine-tuning strategy;

[0014] S3. The visual feature representation is mapped to a low-dimensional contrast space using the feature projection module, and L2 normalization is performed to obtain a normalized feature vector.

[0015] S4. Calculate the multi-level contrast loss based on the normalized feature vector; the multi-level contrast loss is composed of a weighted average of the global real-fake sample contrast loss and the cohesion loss of different forgery methods; wherein, the global real-fake sample contrast loss incorporates a hard sample mining mechanism to bring homogeneous samples closer together and push away the distance between real samples and forged samples in the feature space, ignoring the specific differences in forgery methods; the cohesion loss of different forgery methods measures and minimizes the distance from the centroid of samples within the same forgery category to its centroid by calculating the feature centroid of each category, thereby promoting the clustering of samples of the same forgery method in the feature space.

[0016] S5. Based on the multi-level contrast loss, construct a multi-objective optimization problem, and use the gradient descent method to find the optimal update direction that balances the global real and fake sample contrast loss and the cohesion loss of different forgery methods. Update the model parameters until convergence, so that the model learns forgery attribute features that are independent of specific forgery methods.

[0017] S6. Obtain the face image to be detected, extract features through the trained semantic visual encoder module and the feature projection module, and output the true / false detection result using the classifier.

[0018] The deep face forgery detection method based on feature decoupling and contrastive learning, which is independent of the forgery method described above according to the present invention, may further have the following additional technical features:

[0019] In the above technical solution, the random enhancement processing applied to the aligned face image to destroy specific forgery traces includes:

[0020] Perform at least one of the following operations on a face image: color space perturbation, spatial domain transformation, information loss simulation, compression simulation, and noise injection;

[0021] The color space perturbation includes adjustments to brightness, contrast, saturation, or hue; the spatial domain transformation includes random horizontal flipping, rotation, scaling, cropping, Gaussian blur, motion blur, or median filtering; the information loss simulation includes random block occlusion or random erasure; the compression simulation includes random JPEG compression; and the noise injection includes Gaussian noise injection.

[0022] In the above technical solution, the step of inputting two different enhanced views corresponding to the same image into a CLIP-based semantic visual encoder module to extract visual feature representations rich in semantic information includes:

[0023] The input image is segmented into fixed-size patches, and the patches are linearly projected into an embedding vector sequence. After adding positional encoding, the vector is input into a multi-layer Transformer encoder, and a high-dimensional feature vector is output by aggregation through CLS tokens.

[0024] The progressive fine-tuning strategy includes: freezing the underlying parameters of the CLIP model in the early stage of training and only fine-tuning the parameters of the top layer network; then gradually unfreezing more layer parameters for further optimization.

[0025] In the above technical solution, the feature projection module maps the high-dimensional feature space to the low-dimensional vector space through a multilayer perceptron;

[0026] The method for calculating the normalized feature vector includes:

[0027]

[0028]

[0029]

[0030] in, This represents the high-dimensional feature vector output by the semantic visual encoder module; This represents the low-dimensional feature vector after projection. Representing vectors The i-th element; This represents the first learnable weight matrix; This represents the second learnable weight matrix; This represents the first learnable bias term; This represents the second learnable bias term; Indicates the activation function; This represents the L2 norm.

[0031] In the above technical solution, the calculation of the global real / fake sample comparison loss involves the construction of positive and negative samples, and the construction process includes:

[0032] For each original sample image, its two corresponding different augmented views are used to form a positive sample pair. For images containing... The training batch of original sample images, the positive sample set is defined as:

[0033]

[0034] in, Represents the set of positive samples; This represents the L2-normalized feature vector representation of an augmented view corresponding to the i-th original sample image; This represents the L2-normalized feature vector representation of the other augmented view corresponding to the i-th original sample image;

[0035] Construct a complete set of negative samples that includes cross-class negative sample sets, cross-method negative sample sets, and cross-sample negative sample sets. The cross-category negative sample set includes samples whose true / false categories are different from those of the anchor sample; the cross-method negative sample set includes samples whose true / false categories are the same as those of the anchor sample but whose true / false methods are different; and the cross-sample negative sample set includes other samples from the same batch that are not from the same source as the anchor sample.

[0036] In the above technical solution, the hard sample mining mechanism includes a hard negative sample selection strategy and a hard positive sample weighting strategy;

[0037] The difficult negative sample selection strategy includes: selecting from the entire negative sample set. The set of difficult negative samples is constructed by selecting the top K samples with the highest similarity to the anchor sample. , is represented as:

[0038]

[0039] in, It's a hyperparameter; This represents the function that filters based on the highest similarity. Indicates anchor point sample; This represents the other samples that are not anchor samples in the entire set of negative samples, where k is the index of the other samples that are not anchor samples in the entire set of negative samples; This represents the function for calculating cosine similarity.

[0040] The weighted strategy for difficult positive samples includes: for each pair of positive samples, calculating an adaptive weight based on their feature similarity, the calculation method including:

[0041]

[0042] in, These are hyperparameters used to control the weighting intensity; This represents the adaptive weight corresponding to the i-th positive sample pair.

[0043] In the above technical solution, the calculation method for the global real / fake sample comparison loss includes:

[0044]

[0045] in, This represents the global loss between real and fake samples; This represents the InfoNCE loss incorporating a hard sample mining mechanism; This represents the similarity score between positive sample pairs; This represents a temperature parameter used to control the sharpness of the similarity distribution. Indicates sample With sample Similarity score, Select as or .

[0046] In the above technical solution, the method for calculating the cohesion loss of different forgery methods includes:

[0047]

[0048]

[0049] in, Indicates the cohesion loss of different forgery methods; This represents the total number of categories, where c is the category index; Represents the L2 norm; It is a category All sample sets, It is a category The number of samples; Indicates anchor point sample; The centroid of the feature space represents category c.

[0050] In the above technical solution, the method for calculating the multi-level contrast loss includes:

[0051]

[0052] in, Indicates multi-level contrast loss; This represents the global weighting of the loss between real and fake samples; Indicates the cohesion loss weights for different forgery methods; and All are adaptive weight parameters;

[0053] The multi-objective optimization problem aims to minimize the multi-level contrast loss.

[0054] This invention provides a face deepfake detection system based on feature decoupling and contrastive learning, independent of forgery methods, for implementing the method described in any of the above technical solutions. The system includes:

[0055] The data preprocessing and strong data augmentation module is used to standardize the original image and apply strong random augmentation to destroy specific forgery traces, generating training sample pairs.

[0056] The CLIP-based semantic visual encoder module is used to extract semantic visual features from augmented images based on a pre-trained CLIP model.

[0057] The feature projection module, acting as a feature adapter, is used to transform high-dimensional features to a low-dimensional contrast space and normalize them.

[0058] The multi-level contrastive learning and hard sample mining module is used to construct a set of positive and negative samples, calculate the multi-level contrastive loss which includes the global real and fake sample contrast loss and the cohesion loss of different forgery methods, and perform hard sample mining.

[0059] The model optimization and inference module is used to optimize model parameters based on multi-objective loss, and to perform true / false discrimination using trained feature extractors and classifiers during the inference phase.

[0060] In summary, due to the adoption of the above-mentioned technical features, the beneficial effects of the present invention are:

[0061] This invention constructs a detection framework that autonomously isolates method-specific features and focuses on learning the essential "forgery attributes" by introducing feature decoupling and multi-level contrastive learning mechanisms. This method does not rely on any specific tool traces. Through the feature decoupling mechanism, it actively isolates specific fingerprint features associated with specific generation methods (such as specific GAN architectures or diffusion models), guiding the model to learn the common, deep-seated forgery-related features of all forged content. This mechanism fundamentally alleviates the core problem of existing technologies being prone to overfitting to specific forgery patterns, leading to performance degradation when facing unknown attacks.

[0062] This invention utilizes multi-level contrastive learning and hard sample mining mechanisms to significantly improve the model's generalization ability. By creating a "controllable contradiction" between the global real / fake sample contrast loss and the cohesion loss of different forgery methods, the model is guided to find the optimal balance between macroscopic separation (separation of real and fake samples) and microscopic structure (aggregation of similar forgery samples). This design forces the model to ignore non-essential method differences, ultimately extracting robust and universal discriminative features, enabling the model to maintain high detection accuracy and robustness even when facing unknown forgery methods.

[0063] This invention employs the CLIP model, pre-trained on massive internet data, as the backbone for feature extraction, significantly improving the upper limit of model performance. Benefiting from the rich semantic priors learned by the CLIP model on massive image-text pairs, this invention enhances the model's sensitivity to high-level semantic forgery traces such as identity inconsistencies and lighting inconsistencies, rather than being limited to pixel-level texture artifacts. Simultaneously, the introduction of the pre-trained model enables the model to achieve a better convergence state with less data and shorter training time, significantly improving training efficiency.

[0064] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0065] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0066] Figure 1 This is a flowchart illustrating a face deepfake detection method based on feature decoupling and contrastive learning, which is independent of the forgery method, according to an embodiment of the present invention.

[0067] Figure 2 This is a schematic diagram of the network architecture of a face deepfake detection system based on feature decoupling and contrastive learning, which is independent of the forgery method, according to an embodiment of the present invention. Detailed Implementation

[0068] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0069] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0070] The following reference Figure 1 and Figure 2 This describes a face deepfake detection method and system based on feature decoupling and contrastive learning, independent of the forgery method, provided according to some embodiments of the present invention.

[0071] Some embodiments of this application provide a face deepfake detection method that is independent of forgery methods based on feature decoupling and contrastive learning.

[0072] like Figure 1 As shown, the first embodiment of the present invention proposes a face deepfake detection method based on feature decoupling and contrastive learning that is independent of the forgery method, including the following steps S1 to S6.

[0073] S1. Obtain an original dataset containing real face images and fake face images generated by various fake methods; perform face detection and alignment processing on the images in the original dataset, and apply random enhancement processing to the aligned face images to destroy specific fake traces, generating two different enhanced views for each image.

[0074] Specifically, the face image data collected in the original dataset is used for model training and testing. The forged face images include datasets constructed using one or more of the following forgery methods: DeepFakes, Face2Face, FaceSwap, NeuralTextures, StyleGAN, Midjourney, Stable Diffusion, ADM, DDPM, iDDPM, Pseudo-numerical Diffusion Model (PNDM), GLIDE, LDM, SD-v1, SD-v2, VQ-Diffusion, DALL-E 2, VQDM, and BigGAN. Also included are datasets of genuine and fake faces using unknown forgery methods collected from the internet, such as WildDeepfake.

[0075] The data preprocessing performed on the original dataset in step S1 mainly involves face detection, extracting standard face regions from the original images, and eliminating the influence of variations in pose and scale.

[0076] In one specific embodiment, the face detection model of InsightFace (an open-source "2D & 3D depth face analysis toolkit") is first initialized. Then, face detection is performed on the original input image to obtain face bounding boxes and key points such as the left and right eyes, nose, and left and right corners of the mouth. Based on the obtained key points, a transformation matrix for face alignment is calculated, and an affine transformation is performed on the face region to align its key points with a standard template. Finally, the face region is cropped based on the face bounding boxes and resized to a uniform size of 224. 224.

[0077] The purpose of constructing the training batch dataset is to ensure the diversity of data within each batch. N face image samples are randomly sampled in each training batch, and each batch is required to contain real face samples and face samples generated by various different forgery methods.

[0078] In some implementations, the random enhancement process applied to the aligned face image to disrupt specific forgery traces includes:

[0079] Perform at least one of the following operations on a face image: color space perturbation, spatial domain transformation, information loss simulation, compression simulation, and noise injection, to generate two distinct views. By actively disrupting, interfering with, or erasing fragile surface statistical features that are strongly correlated with a specific generative model through random transformations, the model is forced to learn the essential features of real and fake images.

[0080] The color space perturbation includes adjustments to brightness, contrast, saturation, or hue; the spatial domain transformation includes random horizontal flipping, rotation, scaling, cropping, Gaussian blur, motion blur, or median filtering; the information loss simulation includes random block occlusion or random erasure; the compression simulation includes random JPEG compression; and the noise injection includes Gaussian noise injection.

[0081] S2. Input two different enhanced views corresponding to the same image into the CLIP-based semantic visual encoder module to extract visual feature representations rich in semantic information; the semantic visual encoder module is built based on the pre-trained CLIP model and trained using a progressive fine-tuning strategy.

[0082] The CLIP-based semantic visual encoder module is responsible for converting preprocessed images into visual feature representations rich in high-level semantic information. It doesn't learn from scratch but is built upon a CLIP model pre-trained on massive amounts of internet data. The CLIP model possesses a unique advantage due to its training on vast amounts of image-text pairs; the features generated by its visual encoder are inherently aligned semantically with linguistic concepts such as "real," "artificial," and "face." This provides a high-level starting point for deepfake detection, enabling the model to examine images from the perspective of semantic consistency rather than just pixel-level texture.

[0083] In some embodiments, inputting two different enhanced views corresponding to the same image into a CLIP-based semantic visual encoder module to extract visual feature representations rich in semantic information includes:

[0084] The input image is segmented into fixed-size patches (e.g., 16). 16), and linearly project the tiles into an embedding vector sequence, add position encoding and input it into a multi-layer Transformer encoder. The self-attention mechanism in each layer enables the model to capture the long-distance dependencies between different regions in the image, thereby globally understanding the facial structure and lighting logic; finally, a high-dimensional feature vector is output by aggregating CLS tokens. Specifically, CLS tokens aggregate global information and output a high-dimensional feature vector that can represent the high-level semantics of the entire image.

[0085] The progressive fine-tuning strategy includes: freezing the underlying parameters of the CLIP model in the early stage of training and only fine-tuning the parameters of the top layer network; then gradually unfreezing more layer parameters for further optimization.

[0086] Specifically, in the initial training phase, the low-level parameters are frozen to preserve the general knowledge of the CLIP model, and only the top-level network is fine-tuned to better focus on learning the discriminative features of real and fake faces. Subsequently, more layers are gradually unfrozen for more refined optimization. This training strategy allows the module to not only possess the powerful generalization ability and semantic understanding foundation of the CLIP model, but also to be finely tuned to meet the specific needs of deepfake detection, providing high-quality, discriminative feature representations for subsequent contrastive learning.

[0087] S3. The visual feature representation is mapped to a low-dimensional contrast space using the feature projection module, and L2 normalization is performed to obtain a normalized feature vector.

[0088] The feature projection module, located after the CLIP semantic visual encoder, acts as a feature adapter. Its purpose is to perform a "task-specific" transformation on the high-quality, discriminative feature representation output by the CLIP encoder, mapping it to a low-dimensional space relevant to the forgery detection task.

[0089] Specifically, the feature projection module first uses a designed multilayer perceptron to perform a nonlinear transformation on the features output by the CLIP-based semantic visual encoder module in S2, transforming them from the original 768-dimensional space of the CLIP output. Project onto a low-dimensional, compact vector space of dimension 128. This process allows the model to retain information crucial for distinguishing real and fake faces while discarding a large amount of redundant, task-irrelevant general semantic information. Then, L2 normalization is performed on the projected features, which sets the magnitude of all feature vectors to 1, thus constraining them to a unit hypersphere. The normalized output vector is denoted as... The formula for the above process is shown below:

[0090]

[0091]

[0092]

[0093] in, This represents the high-dimensional feature vector output by the semantic visual encoder module; This represents the low-dimensional feature vector after projection. Representing vectors The summation operation iterates through all dimensions of the vector, starting from the i-th element. This represents the first learnable weight matrix; This represents the second learnable weight matrix; This represents the first learnable bias term; This represents the second learnable bias term; This represents the L2 norm.

[0094] After this processing, the similarity measurement between feature vectors can be completely simplified to calculating their cosine similarity. This provides great convenience and numerical stability for subsequent comparisonal learning loss calculation.

[0095] S4. Calculate multi-level contrast loss based on the normalized feature vector; the multi-level contrast loss is composed of a weighted average of the global real / fake sample contrast loss and the cohesion loss of different forgery methods; wherein, the global real / fake sample contrast loss incorporates a hard sample mining mechanism to bring homologous samples closer together and push away the distance between real samples and forged samples in the feature space, ignoring the specific differences in forgery methods; the cohesion loss of different forgery methods measures and minimizes the distance from the centroid of samples within the same forgery category to its centroid by calculating the feature centroid of each category, thereby promoting the clustering of samples of the same forgery method in the feature space.

[0096] Specifically, the calculation of the global true / false sample comparison loss involves the construction of positive and negative samples, and the construction process includes:

[0097] Regarding the construction of positive samples, specifically, for each sample, whether it's a real face or a fake face, a positive sample pair is formed by two different augmented views. The model needs to learn that samples from the same source but with different appearances are essentially the same. In a specific embodiment, for each original sample image, its corresponding two different augmented views are used to form a positive sample pair. The training batch of original sample images, the positive sample set is defined as:

[0098]

[0099] in, Represents the set of positive samples; This represents the L2-normalized feature vector representation of an augmented view corresponding to the i-th original sample image; This represents the L2-normalized feature vector representation of the other augmented view corresponding to the i-th original sample image;

[0100] The negative sample construction employs a multi-level, comprehensive comparison strategy, specifically constructing a complete negative sample set that includes cross-class negative sample sets, cross-method negative sample sets, and cross-sample negative sample sets. The cross-category negative sample set includes samples whose true / false categories are different from those of the anchor sample; the cross-method negative sample set includes samples whose true / false categories are the same as those of the anchor sample but whose true / false methods are different; and the cross-sample negative sample set includes other samples from the same batch that are not from the same source as the anchor sample.

[0101] In a specific embodiment, the sets of negative samples are represented as follows:

[0102]

[0103]

[0104]

[0105] in, Let represent the set of cross-class negative samples corresponding to the i-th sample (anchor sample); Indicates the category of the anchor sample; This represents an element in the set, i.e., a specific sample; Indicates sample Category; This indicates that the current sample category is the fake category; Represents the set of negative samples across methods; This indicates a method for forging anchor point samples; Indicates sample Forgery methods; Represents the set of negative samples across samples; This indicates other samples that are not homologous to the anchor sample.

[0106] negative sample set It is the union of the three negative sample sets mentioned above. That is:

[0107]

[0108] in, The union symbol represents the set of sets.

[0109] After constructing positive and negative samples through the above process, global real and fake sample comparison learning is performed by constructing a global real and fake sample comparison loss.

[0110] In the conventional global real-fake sample comparison loss, a binary separation feature space is established to clearly separate all real samples from all fake samples, without considering the differences in specific fake methods. Specifically, InfoNCE loss can be adopted.

[0111] Minimizing the InfoNCE loss is equivalent to maximizing the proportion of positive sample similarity. It can bring positive samples closer together and push away negative samples, thereby clearly separating all real samples from all fake samples, while maintaining the maximum possible distance between real samples and fake sample clusters.

[0112] During training, the contributions of different samples to the model's learning are uneven. In some embodiments, to avoid wasting computational resources on simple samples already mastered, and to encourage the model to focus on optimizing the most difficult sample pairs to distinguish, thereby obtaining sharper decision boundaries and improving the model's classification performance, this system introduces a difficult sample mining mechanism. A difficult negative sample selection strategy is adopted for negative samples, and a difficult positive sample weighting strategy is adopted for positive samples.

[0113] Specifically, difficult negative samples refer to samples that are very similar in features to anchor samples but belong to different categories. These samples can easily cause model confusion. The difficult negative sample selection strategy includes: selecting from the entire set of negative samples. The set of difficult negative samples is constructed by selecting the top K samples with the highest similarity to the anchor sample. , is represented as:

[0114]

[0115] in, It is a hyperparameter that can be set to a fixed value, such as 5; This represents the function that filters based on the highest similarity. Indicates anchor point sample; This represents the other samples that are not anchor samples in the entire negative sample set, where k is the index of the other samples in the entire negative sample set that are not homologous to the anchor samples; This represents the function for calculating cosine similarity.

[0116] Hard positive samples refer to different views of the same original sample, but which the model struggles to identify their similarity. These samples are crucial for learning view invariance. The weighting strategy for hard positive samples includes: for each pair of positive samples, calculating adaptive weights based on their feature similarity, the calculation method of which includes:

[0117]

[0118] in, The hyperparameter used to control the weighting strength can be set to 0.5. This controls the weighting strength. Difficult positive samples have lower similarity, and the weights corresponding to the above formula will automatically increase. This represents the adaptive weight corresponding to the i-th positive sample pair.

[0119] In one specific embodiment, the calculation method for the global real / fake sample comparison loss, which incorporates the aforementioned hard sample mining mechanism, includes:

[0120]

[0121] in, This represents the global loss between real and fake samples; This represents the InfoNCE loss incorporating a hard sample mining mechanism; This represents the similarity score between positive sample pairs; This represents a temperature parameter used to control the sharpness of the similarity distribution. Indicates sample With sample Similarity score, Select as or .

[0122] Subsequently, cohesive learning of different forgery methods is performed. Unlike the global level focus in the global real and fake sample comparison loss, this step focuses on the micro level. Cohesive learning of different forgery methods is a solution to the problem of poor generalization in forgery detection. Although different forgery methods differ significantly in surface features, they all share some essential "forgery attributes".

[0123] The cohesive learning mechanism of different forgery methods requires samples from the same forgery method to cluster relatively in the feature space, forming sub-clusters and promoting feature decoupling. This forces the model to maintain macroscopic separation between real and fake samples while building fine-grained structures within the forged samples, thereby compelling the model to strip away method-specific features and retain only common forgery attributes.

[0124] Specifically, the centroids of the categories are first calculated. For each category (including the real category and other fake categories), the centroids of the feature space are calculated as follows:

[0125]

[0126]

[0127] Then, based on the above category centroids, the cohesion loss function is calculated. The cohesion loss function mainly measures the compactness of samples within each category. The calculation methods for the cohesion loss of different forgery methods include:

[0128]

[0129]

[0130] in, Indicates the cohesion loss of different forgery methods; This represents the total number of categories, where c is the category index; Represents the L2 norm; It is a category All sample sets, It is a category The number of samples; Indicates anchor point sample; The centroid of the feature space represents category c.

[0131] Global comparison attempts to mix all fake samples, while cohesive learning requires them to maintain their internal order. This contradictory optimization objective forces the model to find a balance, eventually learning to strip away strong method-specific features, ignoring method differences, and retaining only the common, essential "fake attributes" of various fake methods. By using a multi-objective loss function, a "controllable contradiction" is created, allowing the model to discover that focusing on common attributes is more "cost-effective" than focusing on method differences, guiding the model to spontaneously find the optimal feature representation.

[0132] Multi-objective optimization requires simultaneously optimizing two seemingly contradictory objectives: global real / fake sample comparison loss and cohesion loss. Global real / fake sample comparison loss requires all fake samples to be tightly clustered, while cohesion loss requires samples from different fake methods to be relatively separable. The overall loss function consists of the following two main objectives:

[0133]

[0134] in, Indicates multi-level contrast loss; This represents the global weighting of the loss between real and fake samples; Indicates the cohesion loss weights for different forgery methods; and All are adaptive weight parameters.

[0135] S5. Based on the multi-level contrast loss, construct a multi-objective optimization problem. The multi-objective optimization problem aims to minimize the multi-level contrast loss. The optimal update direction that balances the global real and fake sample contrast loss and the cohesion loss of different forgery methods is found by using gradient descent. The model parameters are updated until convergence, so that the model learns forgery attribute features that are independent of the specific forgery method.

[0136] Specifically, the model optimization problem is expressed as:

[0137]

[0138]

[0139] in, It's about updating the direction. Represents the n-dimensional real space, Denotes the norm of d; Represents the gradient operator; Indicates limiting conditions; This indicates a preset threshold used for step size limitation;

[0140] In deepfake detection tasks, the gradients of the two loss functions inherently conflict. This encourages all fake samples to have similar feature vectors, while This encourages denser feature aggregation of samples using the same forgery method. Multi-objective optimization problems require minimizing both losses simultaneously, so the model must find a balance. At this balance, the model cannot use too many method-specific features (which would increase the global comparison loss between real and fake samples), nor can it completely ignore method differences (which would increase the cohesion loss). Therefore, the model is forced to use features that both aggregate all forged samples (satisfying the global comparison loss) and aggregate samples using the same forgery method (satisfying the cohesion loss), i.e., shared attribute features.

[0141] S6. Obtain the face image to be detected, extract features through the trained semantic visual encoder module and the feature projection module, and output the true / false detection result using the classifier.

[0142] Other embodiments of the present invention provide a face deepfake detection system based on feature decoupling and contrastive learning, independent of forgery methods, for implementing the methods described in any of the above embodiments, such as... Figure 2 As shown, the system includes: a data preprocessing and strong data augmentation module, a CLIP-based semantic visual encoder module, a feature projection module, a multi-level contrastive learning and hard sample mining module, and a model optimization and inference module.

[0143] The data preprocessing and strong data augmentation module is used to standardize the original image and apply strong random augmentation to destroy specific forgery traces, generating training sample pairs.

[0144] The CLIP-based semantic visual encoder module extracts semantic visual features from augmented images based on a pre-trained CLIP model; the feature projection module acts as a feature adapter, transforming high-dimensional features to a low-dimensional contrast space and normalizing them; the multi-level contrast learning and hard sample mining module constructs a set of positive and negative samples, calculates a multi-level contrast loss including global true / false sample contrast loss and cohesion loss of different forgery methods, and performs hard sample mining; the model optimization and inference module optimizes model parameters based on multi-objective loss and performs true / false discrimination using trained feature extractors and classifiers during the inference phase.

[0145] In this specification, the illustrative expressions of the terms used do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0146] Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention shall be included within the scope of protection of this invention.

Claims

1. A face deepfake detection method based on feature decoupling and contrastive learning, independent of forgery methods, characterized in that, include: S1. Obtain the original dataset containing real face images and fake face images generated by various forgery methods; Face detection and alignment are performed on the images in the original dataset, and random enhancement processing that destroys specific forgery traces is applied to the aligned face images. Two different enhanced views are generated for each image. S2. Input two different enhanced views corresponding to the same image into the CLIP-based semantic visual encoder module to extract visual feature representations rich in semantic information; The semantic visual encoder module is built based on a pre-trained CLIP model and trained using a progressive fine-tuning strategy. S3. The visual feature representation is mapped to a low-dimensional contrast space using the feature projection module, and L2 normalization is performed to obtain a normalized feature vector. S4. Calculate the multi-level contrast loss based on the normalized feature vector; the multi-level contrast loss is composed of a weighted average of the global real-fake sample contrast loss and the cohesion loss of different forgery methods; wherein, the global real-fake sample contrast loss incorporates a hard sample mining mechanism to bring homogeneous samples closer together and push away the distance between real samples and forged samples in the feature space, ignoring the specific differences in forgery methods; the cohesion loss of different forgery methods measures and minimizes the distance from the centroid of samples within the same forgery category to its centroid by calculating the feature centroid of each category, thereby promoting the clustering of samples of the same forgery method in the feature space. S5. Based on the multi-level contrast loss, construct a multi-objective optimization problem, and use the gradient descent method to find the optimal update direction that balances the global real and fake sample contrast loss and the cohesion loss of different forgery methods. Update the model parameters until convergence, so that the model learns forgery attribute features that are independent of specific forgery methods. S6. Obtain the face image to be detected, extract features through the trained semantic visual encoder module and the feature projection module, and output the true / false detection result using the classifier.

2. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 1, characterized in that, The random enhancement process applied to the aligned face image to destroy specific forgery traces includes: Perform at least one of the following operations on a face image: color space perturbation, spatial domain transformation, information loss simulation, compression simulation, and noise injection; The color space perturbation includes adjustments to brightness, contrast, saturation, or hue; the spatial domain transformation includes random horizontal flipping, rotation, scaling, cropping, Gaussian blur, motion blur, or median filtering; the information loss simulation includes random block occlusion or random erasure; the compression simulation includes random JPEG compression; and the noise injection includes Gaussian noise injection.

3. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 1, characterized in that, The step of inputting two different enhanced views corresponding to the same image into a CLIP-based semantic visual encoder module to extract visual feature representations rich in semantic information includes: The input image is segmented into fixed-size patches, and the patches are linearly projected into an embedding vector sequence. After adding positional encoding, the vector is input into a multi-layer Transformer encoder, and a high-dimensional feature vector is output by aggregation through CLS tokens. The progressive fine-tuning strategy includes: freezing the underlying parameters of the CLIP model in the early stage of training and only fine-tuning the parameters of the top layer network; then gradually unfreezing more layer parameters for further optimization.

4. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 1, characterized in that, The feature projection module maps the high-dimensional feature space to the low-dimensional vector space through a multilayer perceptron. The method for calculating the normalized feature vector includes: in, This represents the high-dimensional feature vector output by the semantic visual encoder module; This represents the low-dimensional feature vector after projection. Representing vectors The i-th element; This represents the first learnable weight matrix; This represents the second learnable weight matrix; This represents the first learnable bias term; This represents the second learnable bias term; Indicates the activation function; This represents the L2 norm.

5. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 1, characterized in that, The calculation of the global true / false sample comparison loss involves the construction of positive and negative samples, and the construction process includes: For each original sample image, its two corresponding different augmented views are used to form a positive sample pair. For images containing... The training batch of original sample images, the positive sample set is defined as: in, Represents the set of positive samples; This represents the L2-normalized feature vector representation of an augmented view corresponding to the i-th original sample image; This represents the L2-normalized feature vector representation of the other augmented view corresponding to the i-th original sample image; Construct a complete set of negative samples that includes cross-class negative sample sets, cross-method negative sample sets, and cross-sample negative sample sets. The cross-category negative sample set includes samples whose true / false categories are different from those of the anchor sample; the cross-method negative sample set includes samples whose true / false categories are the same as those of the anchor sample but whose true / false methods are different; and the cross-sample negative sample set includes other samples from the same batch that are not from the same source as the anchor sample.

6. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 5, characterized in that, The hard sample mining mechanism includes a hard negative sample selection strategy and a hard positive sample weighting strategy; The difficult negative sample selection strategy includes: selecting from the entire negative sample set. The set of difficult negative samples is constructed by selecting the top K samples with the highest similarity to the anchor sample. , is represented as: in, It's a hyperparameter; This represents the function that filters based on the highest similarity. Indicates anchor point sample; This represents the other samples that are not anchor samples in the entire set of negative samples, where k is the index of the other samples that are not anchor samples in the entire set of negative samples; This represents the function for calculating cosine similarity. The weighted strategy for difficult positive samples includes: for each pair of positive samples, calculating an adaptive weight based on their feature similarity, the calculation method including: in, These are hyperparameters used to control the weighting intensity; This represents the adaptive weight corresponding to the i-th positive sample pair.

7. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 6, characterized in that, The calculation method for the global real / fake sample comparison loss includes: in, This represents the global loss between real and fake samples; This represents the InfoNCE loss incorporating a hard sample mining mechanism; This represents the similarity score between positive sample pairs; This represents a temperature parameter used to control the sharpness of the similarity distribution. Indicates sample With sample Similarity score, Select as or .

8. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 7, characterized in that, The methods for calculating the cohesion loss of different forgery methods include: in, Indicates the cohesion loss of different forgery methods; This represents the total number of categories, where c is the category index; Represents the L2 norm; It is a category All sample sets, It is a category The number of samples; Indicates anchor point sample; The centroid of the feature space represents category c.

9. The face deepfake detection method based on feature decoupling and contrastive learning independent of forgery methods as described in claim 8, characterized in that, The calculation method for the multi-level contrast loss includes: in, Indicates multi-level contrast loss; This represents the global weighting of the loss between real and fake samples; Indicates the cohesion loss weights for different forgery methods; and All are adaptive weight parameters; The multi-objective optimization problem aims to minimize the multi-level contrast loss.

10. A face deepfake detection system based on feature decoupling and contrastive learning, independent of forgery methods, characterized in that, The system for implementing the method as described in any one of claims 1 to 9 comprises: The data preprocessing and strong data augmentation module is used to standardize the original image and apply strong random augmentation to destroy specific forgery traces, generating training sample pairs. The CLIP-based semantic visual encoder module is used to extract semantic visual features from augmented images based on a pre-trained CLIP model. The feature projection module, acting as a feature adapter, is used to transform high-dimensional features to a low-dimensional contrast space and normalize them. The multi-level contrastive learning and hard sample mining module is used to construct a set of positive and negative samples, calculate the multi-level contrastive loss which includes the global real and fake sample contrast loss and the cohesion loss of different forgery methods, and perform hard sample mining. The model optimization and inference module is used to optimize model parameters based on multi-objective loss, and to perform true / false discrimination using trained feature extractors and classifiers during the inference phase.

Images