A dual-branch perceptual CLIP forensics approach for scalable deepfake detection

Abstract

The present invention discloses a scalable dual-branch perceptual CLIP forensics method for deepfake detection, relating to the field of image processing technology. The method comprises the following steps: S1: constructing a dual-path multimodal feature fusion framework, namely, a perceptual contrastive language-image pre-trained forensics network; S2: constructing a dynamic feature fusion module that combines feature quality assessment, similarity analysis, and adaptive temperature scaling, establishing an explicit link mechanism between the contrastive language-image pre-training and trace networks, enabling the perceptual contrastive language-image pre-training forensics network to adaptively adjust fusion weights and activate forgery-related priors in the perceptual contrastive language-image pre-training; and S3: constructing an objective function. The technical problem to be solved by the present invention is to provide a scalable dual-branch perceptual CLIP forensics method for deepfake detection that significantly improves the generalization capability of deepfake detection across different scenarios and forgery techniques.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a dual-branch perceptual CLIP forensics method for generalizable deep fake detection. Background Art

[0002] In the fields of computer vision and deep learning, face forgery techniques typically follow a series of core steps. First, facial regions in the input image are detected and aligned using a facial detector such as MTCNN or RetinaFace. Both the source and target images are then encoded into a shared latent space, where compact feature vectors are extracted. These vectors are then manipulated within the latent space—either swapped or fused—through a conditional generative network or an attention-based module. The manipulated features are then decoded to synthesize the manipulated image. During training, various methods, including loss functions, are used.

[0003] Reconstruction loss, adversarial loss, perceptual loss, and identity preservation loss are jointly optimized to improve synthesis quality, lighting consistency, and identity fidelity. Finally, post-processing techniques such as edge smoothing and Poisson blending are applied to seamlessly merge the synthesized face into the background of the original video frame.

[0004] Despite these sophisticated pipelines, deepfake content often exhibits observable artifacts in noise distribution, edge fusion, facial expressions, lighting, and subtle physiological cues. These flaws—such as high-frequency texture distortion, fusion artifacts, temporal jitter, and signal inconsistencies—provide critical evidence for forgery detection algorithms. Detection models leverage spectral analysis, facial landmark tracking, and multimodal verification to identify and localize such forgeries.

[0005] Facial forgery detection typically relies on convolutional neural networks (CNNs) or pre-trained models based on visual Transformers to extract multi-scale texture, frequency, and semantic features from facial images. These features are then separated in a latent space using attention mechanisms or metric learning to distinguish between "real" and "forged" instances, ultimately resulting in a binary classifier making predictions. The core goal is to capture forgery characteristics unique to DeepFake technology, such as high-frequency noise inconsistencies, edge feathering, lighting and geometric distortions, and facial landmark misalignment.

[0006] However, most existing methods tend to adopt a single-path feature extraction pipeline, focusing on either low-level visual artifacts (such as CNN-based methods) or high-level semantic representations (such as Transformer-based methods). These models often rely on fixed fusion strategies and static temperature parameters, which makes it difficult to simultaneously capture global and local forgery cues or adaptively balance feature weights across different inputs. As a result, they often struggle to generalize across domains and are difficult to combat novel facial forgeries.

[0007] Recently, several studies have attempted to incorporate CLIP into facial forgery detection. For example, C2PCLIP injects category-level cues into CLIP to learn shared representations between different forgery types, significantly improving generalization across datasets. M2F2-Det leverages CLIP's multimodal representation and cue learning to maintain strong detection performance even in unseen forgery scenes. Wavelet-CLIP fuses frequency domain tracks extracted by wavelet transform with CLIPViT features, simultaneously modeling spatial and frequency-level cues for robust forgery detection. ReCLIP introduces universal perturbations into the CLIP visual encoder, allowing it to be "reprogrammed" for forgery detection without fine-tuning.

[0008] Although these CLIP-based methods are innovative, they share a common limitation: they regard CLIP as merely a single feature extractor, lack adaptability to specific tasks, and fail to fully utilize the complementary advantages of low-level texture analysis and high-level semantic understanding. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a dual-branch perception CLIP forensics method for generalizable deep fake detection, which significantly improves the generalization ability of deep fake detection in different scenarios and forgery technologies.

[0010] The present invention adopts the following technical solutions to achieve the invention objectives:

[0011] A dual-branch perceptual CLIP forensics method for scalable deepfake detection, characterized by comprising the following steps:

[0012] S1: Construct a dual-path multimodal feature fusion framework, namely, a perceptual contrast language-image pre-trained forensic network;

[0013] The framework adopts a dual-path architecture: one path uses a trace network branch based on depthwise separable convolution to extract local high-frequency forgery cues, while the other path uses a frozen contrastive language-image pre-trained visual transformer backbone network to encode global semantic features;

[0014] S2: Construct a dynamic feature fusion module that combines feature quality assessment, similarity analysis, and adaptive temperature scaling. It also establishes an explicit link mechanism between the contrastive language-image pre-training and the trace network, enabling the perceptual contrastive language-image pre-trained forensics network to adaptively adjust fusion weights and activate forgery-related priors in the perceptual contrastive language-image pre-training.

[0015] S3: Construct the objective function.

[0016] As a further limitation of this technical solution, the specific content of S1 is:

[0017] S11: First, we connect the structural and semantic features extracted by the trace network and the contrastive language-image pre-training;

[0018] S12: Then reconstructed and integrated through the fully connected mapping layer;

[0019] S13: The model calculates adaptive fusion weights based on feature quality and semantic consistency, dynamically balancing the contributions of global semantic information and local structural details, and fully leveraging the complementary advantages of the two features;

[0020] S14: The fused features are passed through a multi-layer perceptron classification head consisting of rectified linear units, random dropout, and batch normalization layers to perform binary classification between forged and real samples.

[0021] As a further limitation of this technical solution, in the trace network path, 12 depth-wise separable convolutional modules are introduced, followed by a global average pooling layer and a linear projection layer to extract a 2048-dimensional global image representation as the input for the final fusion with the contrastive language-image pre-trained features;

[0022] In order to further enhance the discriminative ability of the intermediate representation, the intermediate feature maps are extracted from the 4th, 5th and 6th layers, denoted as , where index , is a collection of feature maps extracted from the 4th, 5th and 6th layers of the network, is the batch size, Indicates the number of channels, and Respectively represent the spatial resolution of the feature map, index corresponds to the selected middle layer;

[0023] Divide the feature map into non-overlapping spatial blocks, the formula is as follows:

[0024] (1);

[0025] in: Indicates that through The first layer obtains the spatial segmentation of the feature map patch area;

[0026] Indicates the spatial size of each patch;

[0027] Indicates the total quantity;

[0028] The number of patches per feature map satisfies ;

[0029] Each patch is then processed by spatial average pooling to obtain a local representation, calculated as , AvgPool () represents the average pooling operation, and in the layer The set of all such patch-level embeddings of forms the patch label set, defined as , Represents the average pooling after The characteristics of the patch, Indicates that from The set of all patch labels extracted by the layer.

[0030] As a further limitation of this technical solution, the specific steps of S2 are:

[0031] S21: The output feature representation of the trace network and the contrastive language-image pre-training branch is:

[0032] (2);

[0033] In order to achieve dynamic fusion of semantic and structural features, a learnable feature fusion module is introduced to fuse weights It is generated by a lightweight multilayer perceptron network that takes the following input:

[0034] (3);

[0035] in: The quality factor for comparing language-image pre-training features;

[0036] Represents the similarity score between the trace network and the contrastive language-image pre-trained features;

[0037] represents a learnable temperature term;

[0038] S22: Next, the module calculates the structural features in the trace network and contrasting semantic features in language-image pre-training to obtain the intermediate representation:

[0039] (4);

[0040] S23: In order to further improve the distinguishability of the fusion representation, the intermediate representation , structural characteristics and semantic features Concatenate them and feed the result into a fully connected layer followed by a nonlinear projection:

[0041] (5);

[0042] in: represents fusion features;

[0043] Indicates that nonlinear transformation is performed on the results after batch normalization and linear transformation;

[0044] represents batch normalization;

[0045] represents the weight matrix;

[0046] represents the bias vector;

[0047] S24: Finally, fusion features Make binary predictions using a multilayer perceptron classifier:

[0048] (6);

[0049] in: is a multi-layer perceptron.

[0050] As a further limitation of this technical solution, the specific steps of S3 are:

[0051] S31: Cross entropy loss, in order to optimize the distinguishing ability of fusion features in the binary classification task, the output of the classifier The cross entropy loss is applied on represents the set of positive and negative samples, and the predicted probability of the false category is defined as:

[0052] (7);

[0053] Indicates the sample number;

[0054] in: represents the real sample, represents a fake sample;

[0055] S32: Binary cross entropy loss is defined as:

[0056] (8);

[0057] Represents a sample set

[0058] S33: Contrastive learning loss, defining the training batch as ,in and represent the sets of real samples and fake samples respectively, and the contrast loss is defined as:

[0059] (9);

[0060] in: and yes or A pair of positive samples in ;

[0061] Represents the feature representation of negative samples;

[0062] Represents the corresponding negative sample set;

[0063] represents the temperature parameter;

[0064] S34: Overall loss. During the training phase, the optimization objectives include cross entropy loss. and contrastive learning loss , the overall loss is defined as:

[0065] (10);

[0066] in: and is the weight coefficient used to balance each loss term.

[0067] Compared with the prior art, the advantages and positive effects of the present invention are:

[0068] 1. This paper proposes a lightweight and decoupled dual-pathway framework for facial forgery detection. Although CLIP demonstrates excellent general visual understanding capabilities, it suffers from distribution shift in the context of forgery detection and lacks the ability to capture fine-grained manipulation traces when used as a static feature extractor. To address these limitations, we introduce the Perceptual CLIP Forensics Network (PCFN), a dual-branch architecture consisting of a local structure modeling path and a frozen semantic encoding path. These two branches are designed to perceive local texture anomalies and global semantic consistency, respectively, and are fused in the final stage to perform unified forgery prediction. Despite its highly simplified structure, PCFN achieves stable and significant cross-dataset performance improvements on multiple benchmark datasets, demonstrating strong detection capabilities and generalization potential. Our framework provides a solid foundation for building efficient and robust facial forgery detectors in open-world scenarios.

[0069] 2. Our proposed framework, PCFN, employs a parallel dual-branch architecture—combining a frozen CLIP encoder and a lightweight TraceNet branch based on depthwise separable convolutions. This architecture is further enhanced by a multi-scale dynamic feature fusion mechanism, enabling precise and adaptive integration of global semantics and local forgery cues. Experimental results demonstrate that PCFN significantly improves the generalization of deepfake detection across diverse scenarios and forgery techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0071] A specific embodiment of the present invention is described in detail below with reference to the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific embodiment.

[0072] The present invention comprises the following steps:

[0073] S1: We construct a dual-path multimodal feature fusion framework, namely the Perceptual Contrastive Language–Image Pre-training (CLIP) Forensic Network (PCFN), which effectively combines the global semantic understanding of CLIP with the local high-frequency sensitivity of TraceNet, addressing the limitations of single-stream models in complex forgery scenarios and greatly improving versatility.

[0074] The framework adopts a dual-path architecture: one path uses a trace network branch based on depthwise separable convolution to extract local high-frequency forgery cues, while the other path uses a frozen Contrastive Language - Image Pretraining Vision Transformer (CLIP ViT) backbone network to encode global semantic features; to effectively combine these two modalities, we further introduce a dynamic feature fusion module, which integrates quality-aware weighting, similarity-based interaction, and an adaptive temperature mechanism to enhance the fusion effect and representation ability, thereby improving the accuracy and robustness of forgery detection.

[0075] The specific content of S1 is:

[0076] PCFN integrates a frozen CLIP ViT backbone network in parallel with the CLIP branch, which is responsible for extracting global semantic features from the image. Simultaneously, the TraceNet branch employs a staged stacking of multiple layers of depthwise separable convolutional blocks, progressively capturing local high-frequency artifacts from shallow to deep layers, focusing on fine-grained detection of transition boundaries and subtle noise artifacts. To achieve deep fusion of the two feature streams at different spatial and semantic scales, a dynamic fusion module with a learnable weight mechanism is constructed.

[0077] S11: First, we connect the structural and semantic features extracted by the trace network and the contrastive language-image pre-training;

[0078] S12: Then reconstructed and integrated through the fully connected mapping layer;

[0079] S13: The model calculates adaptive fusion weights based on feature quality and semantic consistency, dynamically balancing the contributions of global semantic information and local structural details, and fully leveraging the complementary advantages of the two features;

[0080] S14: The fused features are passed through a multi-layer perceptron classification head, which includes a rectified linear unit (ReLU), random dropout (Dropout) and batch normalization layer to perform binary classification between forged samples and real samples.

[0081] This dual-path design ensures generalization ability and sensitivity to subtle details, and is verified by large-scale cross-dataset experiments to effectively improve the generalization performance of forgery detection. Figure 1 shown.

[0082] In the trace network path, 12 depth-wise separable convolutional modules are introduced, followed by a global average pooling layer and a linear projection layer to extract a 2048-dimensional global image representation as the input for the final fusion with the contrastive language-image pre-trained features;

[0083] In order to further enhance the discriminative ability of the intermediate representation, the intermediate feature maps are extracted from the 4th, 5th and 6th layers, denoted as , where index , is a collection of feature maps extracted from the 4th, 5th and 6th layers of the network, is the batch size, Indicates the number of channels, and Respectively represent the spatial resolution of the feature map, index corresponds to the selected middle layer;

[0084] Divide the feature map into non-overlapping spatial blocks, the formula is as follows:

[0085] (1);

[0086] in: Indicates that through The first layer obtains the spatial segmentation of the feature map patch area;

[0087] Indicates the spatial size of each patch;

[0088] Indicates the total quantity;

[0089] The number of patches per feature map satisfies ;

[0090] Each patch is then processed by spatial average pooling to obtain a local representation, calculated as , AvgPool () represents the average pooling operation, and in the layer The set of all such patch-level embeddings of forms the patch label set, defined as , Represents the average pooling after The characteristics of the patch, Indicates that from The set of all patch labels extracted by the layer.

[0091] S2: Construct a dynamic feature fusion module that combines feature quality assessment, similarity analysis, and adaptive temperature scaling. It establishes an explicit link mechanism between contrastive language-image pre-training and the trace network, enabling the Perceptual CLIP Forensics Network to adaptively adjust fusion weights and activate forgery-related priors in contrastive language-image pre-training.

[0092] The specific steps of S2 are:

[0093] To effectively integrate the heterogeneous features of the trace network and the contrastive language-image pre-training branch, a dynamic feature fusion module is designed to explicitly model the complementary relationship between their representations and enhance the overall discriminative ability.

[0094] S21: The output feature representation of the trace network and the contrastive language-image pre-training branch is:

[0095] (2);

[0096] In order to achieve dynamic fusion of semantic and structural features, a learnable feature fusion module is introduced to fuse weights It is generated by a lightweight Multilayer Perceptron (MLP) network, which is based on the following input:

[0097] (3);

[0098] in: The quality factor for comparing language-image pre-training features;

[0099] Represents the similarity score between the trace network and the contrastive language-image pre-trained features;

[0100] represents a learnable temperature term;

[0101] S22: Next, the module calculates the structural features in the trace network and contrasting semantic features in language-image pre-training to obtain the intermediate representation:

[0102] (4);

[0103] S23: In order to further improve the distinguishability of the fusion representation, the intermediate representation , structural characteristics and semantic features Concatenate them and feed the result into a fully connected layer followed by a nonlinear projection:

[0104] (5);

[0105] in: represents fusion features;

[0106] Indicates that nonlinear transformation is performed on the results after batch normalization and linear transformation;

[0107] represents batch normalization;

[0108] represents the weight matrix;

[0109] represents the bias vector;

[0110] S24: Finally, fusion features Make binary predictions using a multilayer perceptron classifier:

[0111] (6);

[0112] in: is a multi-layer perceptron.

[0113] This design effectively enhances the integration of global semantic information and local forgery traces, and activates the forgery-related knowledge embedded in CLIP through a dynamic guidance strategy, thereby improving the generalization ability of cross-domain forgery detection.

[0114] S3: Construct the objective function.

[0115] The specific steps of S3 are:

[0116] S31: Cross entropy loss, in order to optimize the distinguishing ability of fusion features in the binary classification task, the output of the classifier The cross entropy loss is applied on represents the set of positive and negative samples, and the predicted probability of the false category is defined as:

[0117] (7);

[0118] Indicates the sample number;

[0119] in: represents the real sample, represents a fake sample;

[0120] S32: Binary cross entropy loss is defined as:

[0121] (8);

[0122] represents a sample set;

[0123] Since the CLIP path is frozen during training, this loss is only applied to the TraceNet path to guide the model to learn to distinguish the real / fake boundaries;

[0124] S33: Contrastive learning loss, since in this context, fake samples may belong to different categories (e.g., DF, NT), the sample similarity between them is not considered. Instead, it focuses on clustering real samples together while increasing the distance between real samples and fake samples. The training batch is defined as ,in and represent the sets of real samples and fake samples respectively, and the contrast loss is defined as:

[0125] (9);

[0126] in: and yes or A pair of positive samples in ;

[0127] Represents the feature representation of negative samples;

[0128] Represents the corresponding negative sample set;

[0129] represents the temperature parameter;

[0130] Since the CLIP pathway is frozen, this contrastive learning objective is only applied to the TraceNet pathway and its attention module, encouraging stronger discrimination of local boundary tracks and global semantics.

[0131] S34: Overall loss. During the training phase, the optimization objectives include cross entropy loss. and contrastive learning loss , the overall loss is defined as:

[0132] (10);

[0133] in: and is the weight coefficient used to balance each loss term.

[0134] The above disclosure is only a specific embodiment of the present invention, but the present invention is not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A dual-branch perceptual CLIP forensics approach for generalizable deepfake detection, characterized by: The following steps are involved: S1: Construct a dual-path multimodal feature fusion framework, namely, a perceptual contrast language-image pre-trained forensic network; The framework adopts a dual-path architecture: one path uses a trace network branch based on depthwise separable convolution to extract local high-frequency forgery cues, while the other path uses a frozen contrastive language-image pre-trained visual transformer backbone network to encode global semantic features; S2: Construct a dynamic feature fusion module that combines feature quality assessment, similarity analysis, and adaptive temperature scaling. It also establishes an explicit link mechanism between the contrastive language-image pre-training and the trace network, enabling the perceptual contrastive language-image pre-trained forensics network to adaptively adjust fusion weights and activate forgery-related priors in the perceptual contrastive language-image pre-training. S3: constructing an objective function, wherein the objective function includes a cross entropy loss and a contrastive learning loss; The specific steps of S2 are: S21: The output feature representation of the trace network and the contrastive language-image pre-training branch is: （2）； In order to achieve dynamic fusion of semantic and structural features, a learnable feature fusion module is introduced to fuse weights It is generated by a lightweight multilayer perceptron network that takes the following input: （3）； in: The quality factor for comparing language-image pre-training features; Represents the similarity score between the trace network and the contrastive language-image pre-trained features; represents a learnable temperature term; S22: Next, the module calculates the structural features in the trace network and contrasting semantic features in language-image pre-training to obtain the intermediate representation: （4）； S23: In order to further improve the distinguishability of the fusion representation, the intermediate representation , structural characteristics and semantic features Concatenate them and feed the result into a fully connected layer followed by a nonlinear projection: （5）； in: represents fusion features; Indicates that nonlinear transformation is performed on the results after batch normalization and linear transformation; represents batch normalization; represents the weight matrix; represents the bias vector; S24: Finally, fusion features Make binary predictions using a multilayer perceptron classifier: （6）； in: is a multi-layer perceptron.

2. The dual-branch perceptual CLIP forensics method for scalable deepfake detection according to claim 1, characterized in that: The specific content of S1 is: S11: First, we connect the structural and semantic features extracted by the trace network and the contrastive language-image pre-training; S12: Then reconstructed and integrated through the fully connected mapping layer; S13: The model calculates adaptive fusion weights based on feature quality and semantic consistency, dynamically balancing the contributions of global semantic information and local structural details, and fully leveraging the complementary advantages of the two features; S14: The fused features are passed through a multi-layer perceptron classification head consisting of rectified linear units, random dropout, and batch normalization layers to perform binary classification between forged and real samples.

3. The dual-branch perceptual CLIP forensics method for scalable deepfake detection according to claim 2, characterized in that: In the trace network path, 12 depth-wise separable convolutional modules are introduced, followed by a global average pooling layer and a linear projection layer to extract a 2048-dimensional global image representation as the input for the final fusion with the contrastive language-image pre-trained features; In order to further enhance the discriminative ability of the intermediate representation, the intermediate feature maps are extracted from the 4th, 5th and 6th layers, denoted as , where index , is a collection of feature maps extracted from the 4th, 5th and 6th layers of the network, is the batch size, Indicates the number of channels, and Respectively represent the spatial resolution of the feature map, index corresponds to the selected middle layer; Divide the feature map into non-overlapping spatial blocks, the formula is as follows: （1）； in: Indicates that through The first layer obtains the spatial segmentation of the feature map patch area; Indicates the spatial size of each patch; Indicates the total quantity; The number of patches per feature map satisfies ; Each patch is then processed by spatial average pooling to obtain a local representation, calculated as , AvgPool () represents the average pooling operation, and in the layer The set of all such patch-level embeddings of forms the patch label set, defined as , Represents the average pooling after The characteristics of the patch, Indicates that from The set of all patch labels extracted by the layer.

4. The dual-branch perceptual CLIP forensics method for scalable deepfake detection according to claim 3, characterized in that: The specific steps of S3 are: S31: Cross entropy loss, in order to optimize the distinguishing ability of fusion features in the binary classification task, the output of the classifier The cross entropy loss is applied on represents the set of positive and negative samples, and the predicted probability of the false category is defined as: （7）； Indicates the sample number; in: represents the real sample, represents a fake sample; S32: Binary cross entropy loss is defined as: （8）； Represents a sample set S33: Contrastive learning loss, defining the training batch as ,in and represent the sets of real samples and fake samples respectively, and the contrast loss is defined as: （9）； in: and yes or A pair of positive samples in ; Represents the feature representation of negative samples; Represents the corresponding negative sample set; represents the temperature parameter; S34: Overall loss. During the training phase, the optimization objectives include cross entropy loss. and contrastive learning loss , the overall loss is defined as: （10）； in: and is the weight coefficient used to balance each loss term.

Images