High-robust face deepfake detection method based on CLIP semantic guidance

Abstract

The application discloses a high-robust face deepfake detection method based on CLIP semantic guidance, comprising the following steps: constructing a deepfake image detection model, wherein the model comprises a prompt word alignment semantic mapping module and a disturbance response invariant modeling module, obtaining a prompt word response tensor of an input image through the prompt word alignment semantic mapping module, and obtaining a detection result according to the prompt word response tensor through the disturbance response invariant modeling module. Through multi-view detection under fixed disturbance, prompt word guided semantic alignment mapping, cross-disturbance response statistical modeling and disturbance invariance learning, the detection stability of the model under various post-processing conditions is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of deepfake image detection, and in particular to a highly robust face deepfake detection method based on CLIP semantic guidance. Background Technology

[0002] In recent years, deep generative technologies, represented by Generative Adversarial Networks (GANs) and Diffusion Models, have made groundbreaking progress, enabling deepfake facial images to achieve unprecedented levels of visual realism in tasks such as identity replacement, facial expression replay, and local redrawing. This high-fidelity forgery poses significant risks in scenarios such as online misinformation, identity theft, and the spread of malicious misinformation, seriously threatening the credibility of digital media and social information security. Therefore, how to conduct high-precision and robust detection of such forged images has become a key scientific problem and technological challenge that urgently needs to be addressed in the field of digital image forensics.

[0003] Currently, mainstream deepfake detection methods can be mainly classified into three categories: detection based on spatial domain subtle artifacts, detection based on frequency domain statistical distortion, and detection by fusing temporal or cross-modal information. These methods typically rely on capturing low-level clues left behind during the forgery process, such as texture anomalies, boundary fusion defects, spectral bias, or inconsistencies between video frames. Although these methods perform well in ideal environments where the training and test sets are evenly distributed, they face severe challenges in real-world, complex application scenarios. Post-processing operations commonly performed on images during propagation, such as JPEG compression, resolution scaling, Gaussian blurring, noise interference, and color space adjustments, can easily mask or destroy these low-level forgery traces, leading to a sharp degradation in the performance and a significant decrease in robustness of existing detectors. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a robust deepfake face detection method based on CLIP semantic guidance, which solves the technical problem that existing technologies suffer from reduced robustness when faced with common post-processing perturbations such as compression, blurring, and scaling in real propagation links.

[0005] This invention provides a robust deepfake face detection method based on CLIP semantic guidance, comprising:

[0006] A deepfake image detection model is constructed, comprising: a cue word alignment semantic mapping module and a perturbation response invariant modeling module. The cue word alignment semantic mapping module obtains the cue word response tensor of the input image, and the perturbation response invariant modeling module obtains the detection result based on the cue word response tensor.

[0007] The specific process of obtaining the prompt word response tensor of the input image through the prompt word alignment semantic mapping module includes the following steps:

[0008] Step A1: Perform pixel-domain mapping on the input image;

[0009] Step A2: Construct a fixed perturbation ladder to generate several perturbation views for the pixel-domain mapped image;

[0010] Step A3: Extract visual features using the frozen visual language model;

[0011] Step A4: Project the visual features onto the text feature dimension through a shared linear projection matrix, calculate the cosine similarity with several prompt word embeddings, obtain the prompt word response map, and construct the prompt word response tensor;

[0012] The specific process of obtaining the detection results based on the cue word response tensor through the perturbation response invariant modeling module includes the following steps:

[0013] Step B1: Obtain the cross-perturbation statistical representation of the cue word response tensor;

[0014] Step B2: Calculate the prompt word score based on the prompt word response tensor, and obtain the prompt word route weight through Stable Prompt Top-K Routing (SPTKR) selection and normalization;

[0015] Step B3: Based on the cross-perturbation statistical representation and the prompt word routing weights, obtain the local discrimination results and global discrimination results through local branches and global branches respectively, and obtain the detection results after weighted fusion.

[0016] Furthermore, in step A1, the pixel domain mapping function in the pixel domain mapping process includes: fixed decimal quantization, or random affine transformation, or a combination of fixed decimal quantization and random affine transformation.

[0017] Furthermore, in step A2, the perturbation ladder uses the same set of perturbation types in both the training and inference phases. In the training phase, the intensity of each perturbation is randomly sampled and set, while in the inference phase, the intensity of each perturbation is set deterministically. The perturbation types include original view, JPEG compression, blur, scaling, noise, and color perturbation.

[0018] The formula for generating the perturbation view is:

[0019] ;

[0020] In the formula, I (k) Perturbation view; P k(I0) represents the k-th perturbation operator; I0 represents the image after pixel-domain mapping; and K represents the total number of perturbation operators.

[0021] Furthermore, step A3 also includes optimizing the frozen visual language model, specifically as follows:

[0022] Insert a perturbation conditional adapter into the residual block of the frozen visual language model.

[0023] First, the perturbation conditional adapter performs a bottleneck residual transformation on the input parameters. The specific formula is as follows:

[0024] ;

[0025] In the formula, Visual token features are input to the perturbation condition adapter; and These are the lower projection matrix and the upper projection matrix, respectively, used to map the features to the bottleneck dimension and then restore them to the original feature dimension; The activation function for the Gaussian error linear unit; The disturbance compensation characteristics are obtained from the bottleneck residual transformation.

[0026] Then, based on the conditional vector formed by the disturbance type and disturbance intensity, modulation parameters are generated, with the specific formula as follows:

[0027] ;

[0028] In the formula, For the first The condition vector of each perturbation view is obtained by concatenating the one-hot encoding of the perturbation type with the scalar of the perturbation intensity; For conditionally modulated multilayer perceptrons; and These are the conditional scaling and conditional offset terms, which are consistent with the visual token dimension.

[0029] Finally, the output is generated based on the modulation parameters, and the specific formula is as follows:

[0030] ;

[0031] In the formula, y is the output of the disturbance condition adapter; γ and β are the condition scaling term and condition offset term in the modulation parameters, respectively; This is element-wise multiplication.

[0032] Furthermore, in step A4, the specific formula for the prompt word response graph is as follows:

[0033] ;

[0034] In the formula, A(l,k,m) (p) is the prompt word response graph; s is the similarity scaling factor; t m Embed for the m-th cue word; For the lth layer A perturbation view in spatial location The visual token at the location; To share a linear projection matrix, which is used to map visual features to the text embedding dimension; It is an L2 norm; Indicates transpose;

[0035] The constructed prompt response tensor is: Where L is the number of layers extracted by the visual encoder; K is the number of views of the fixed perturbation ladder; M is the number of cue words in the cue word library; and H and W are the spatial location dimensions.

[0036] Furthermore, the specific process of step B1 is as follows:

[0037] The specific formula for the cross-disturbance response variance plot is as follows:

[0038] First, obtain the cross-perturbation response variance plot of the cue word response tensor. The specific formula is as follows:

[0039] ;

[0040] Then, the cross-disturbance response variance plot is log-scaled, using the following formula:

[0041] ;

[0042] Finally, after instance normalization of the log-scaling results, the cross-perturbation statistical characterization is obtained.

[0043] In the formula, V (l,m) (p) is the variance plot of the response across the disturbance; A (l,k,m) (p) is the cue word response graph; α is the scaling factor; This is the result of logarithmic scaling; Indicates the dimension along the perturbation view Calculate the variance above.

[0044] Furthermore, in step B2, the formula for calculating the prompt word score is:

[0045] ;

[0046] In the formula, s m Score points for the prompt words; A is a nonnegative function; (l,k,m) (p) is the prompt word response diagram;

[0047] The stable prompt word Top-K routing uses the same Top-K mechanism in both the training and inference phases. Prompt words not selected for the Top-K set are assigned a base weight, and a minimum guarantee mechanism of retaining one prompt word is implemented. The specific formula for obtaining the prompt word route weight is as follows:

[0048] ;

[0049] In the formula, Assign route weights to prompt words; For indicator functions; It is a stable term; The normalized soft weight for the m-th cue word; Prior bias of basic prompt words; Score for prompt words; T is the temperature parameter; M is the total number of prompt words; This is the set of Top-K cue words.

[0050] Furthermore, the training phase of the deepfake image detection model also includes: introducing leave-of-sight. Figure 1 Perturbation invariant learning is performed using consistency constraints and domain adversarial perturbation confusion, and parameters are optimized for the deepfake image detection model through backpropagation.

[0051] Furthermore, the aforementioned retention of view Figure 1 The loss function for consistency constraints is:

[0052] ;

[0053] The loss function for domain adversarial perturbation confusion is:

[0054] ;

[0055] In the formula, D(·) is the perturbation domain classifier; d is the perturbation domain label; g is the global representation used for perturbation domain classification, which is obtained by statistically stitching together the spatial mean of the response maps of each layer; GRL(·) is the gradient reversal layer, which keeps the input unchanged during forward propagation and multiplies the gradient by a negative coefficient during backward propagation; CE(·,·) is the cross-entropy loss.

[0056] Furthermore, the binary cross-entropy loss function of the deepfake image detection model during the training phase is:

[0057] ;

[0058] The objective function is:

[0059] ;

[0060] In the formula, and To balance the weights for positive and negative samples; The function is the Sigmoid function; N is the batch size. ∈{0,1} represents the true label; For the first The predicted logit for each sample; For the regularization term of the adapter modulation parameters under disturbance conditions; To leave a view Figure 1 Induced loss; For domain-specific perturbation confusion loss; For the prior stable terms of the prompt words; For prompt word entropy regularization; , , , , These are the corresponding weighting coefficients.

[0061] The beneficial effects of this invention are:

[0062] This invention explicitly models realistic post-processing perturbations such as compression, blurring, and scaling as fixed perturbation domains, maintaining consistency in perturbation types across multiple views during training and inference phases, thereby effectively reducing performance degradation caused by perturbation type mismatch. Simultaneously, by leveraging a frozen visual language model and a pre-defined artifact cue lexicon, a multi-layered cue response graph is constructed, unifying various scattered cues such as compression artifacts, boundary anomalies, frequency anomalies, color distortion, and geometric inconsistencies into an interpretable forensic semantic space, effectively improving the semantic consistency and model interpretability of the detection process.

[0063] This invention utilizes statistical modeling of cross-disturbance responses, Top-K routing selection based on stable cue words, and look-ahead techniques. Figure 1 Mechanisms such as consistency constraints and domain adversarial perturbation confusion suppress the model's over-reliance on single perturbation fingerprints, enabling it to maintain high detection performance under both known and unknown post-processing perturbation conditions.

[0064] The model of this invention introduces a local and global dual-branch fusion structure, which takes into account both spatial fine-grained artifact capture and overall statistical consistency modeling, and achieves a balance between intra-domain detection accuracy and robustness to real-world scenarios. Attached Figure Description

[0065] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0066] Figure 1 This is a schematic diagram of the detection framework according to a specific embodiment of the present invention;

[0067] Figure 2This is a multi-view prompt word response diagram under a fixed disturbance ladder in a specific embodiment of the present invention;

[0068] Figure 3 This is a visualization of the Top-K routing weights of stable prompt words in a specific embodiment of the present invention;

[0069] Figure 4 This is a visualization of the statistical characterization of the cross-disturbance response in a specific embodiment of the present invention;

[0070] Figure 5 This is a visualization of the robust detection results in a specific embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] The present invention will be further illustrated below with reference to specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Modifications to the present invention in various equivalent forms all fall within the scope defined by the appended claims.

[0073] This invention aims to address the issue of decreased robustness of existing deepfake detection methods in real-world propagation paths when faced with common post-processing perturbations such as compression, blurring, and scaling. It proposes a highly robust face deepfake detection method based on CLIP semantic guidance. This method significantly improves the model's detection stability under various post-processing conditions through multi-view detection under fixed perturbations, semantic alignment mapping guided by cue words, statistical modeling of cross-perturbation responses, and perturbation invariance learning.

[0074] like Figure 1 As shown in the figure, a specific embodiment of the present invention provides a highly robust deepfake face detection method based on CLIP semantic guidance, including:

[0075] A deepfake image detection model (PRISM) is constructed. The PRISM model includes a Prompt-Aligned Semantic Mapping (PASM) module and a Perturbation-Invariant Response Modeling (PRIM) module. The PASM module obtains the prompt response tensor of the input image, and the PRIM module uses the PASM module to obtain the detection result based on the prompt response tensor.

[0076] The specific process of obtaining the prompt word response tensor of the input image through the prompt word alignment semantic mapping module includes the following steps:

[0077] Step A1: Perform pixel-domain mapping on the input image;

[0078] To further simulate the non-ideal processing in the real propagation path, such as platform quantization, brightness drift, and slight color shift, pixel-domain mapping is performed on the input image to obtain the pixel-domain mapped image. The process is represented as follows:

[0079] ;

[0080] In the formula, denoted as the pixel-domain mapping function; I is the input image.

[0081] The pixel-domain mapping function can be fixed-decimal-place quantization, random affine transformation, or a combination of both. By performing a lightweight mapping of the input image in the pixel domain, the distributional discrepancy between the training data and the processing of the real platform can be reduced.

[0082] Step A2: Construct a fixed perturbation ladder to generate several perturbation views for the pixel-domain mapped image;

[0083] Construct a fixed perturbation ladder to generate the same input image. A perturbation view;

[0084] The perturbation ladder uses a consistent set of perturbation types in both the training and inference phases. During the training phase, the intensity of each perturbation is randomly sampled and set, while during the inference phase, the intensity of each perturbation is set deterministically. The perturbation types include original view, JPEG compression, blur, scaling, noise, and color perturbation.

[0085] The formula for generating the perturbation view is:

[0086] ;

[0087] In the formula, I (k) Perturbation view; P k(I0) represents the k-th perturbation operator; I0 represents the image after pixel-domain mapping; and K represents the total number of perturbation operators.

[0088] Step A3: Extract visual features from the frozen visual language model. The visual encoder in the frozen visual language model extracts multi-layer image patch tokens. Lightweight perturbation conditional adapters are inserted into the later residual blocks to compensate for feature drift between different perturbation domains. For the input token 𝑥, the perturbation conditional adapter first performs a bottleneck residual transformation, with the specific formula as follows:

[0089] ;

[0090] In the formula, Visual token features are input to the perturbation condition adapter; and These are the lower projection matrix and the upper projection matrix, respectively, used to map the features to the bottleneck dimension and then restore them to the original feature dimension; The activation function for the Gaussian error linear unit; The disturbance compensation characteristics are obtained from the bottleneck residual transformation.

[0091] Then, based on the condition vector c formed by the disturbance type and disturbance intensity... (k) The modulation parameters are generated using the following formula:

[0092] ;

[0093] In the formula, For the first The condition vector of each perturbation view is obtained by concatenating the one-hot encoding of the perturbation type with the scalar of the perturbation intensity; For conditionally modulated multilayer perceptrons; and These are the conditional scaling and conditional offset terms, which are consistent with the visual token dimension.

[0094] Finally, the output is generated based on the modulation parameters, and the specific formula is as follows:

[0095] ;

[0096] In the formula, y is the output of the disturbance condition adapter; γ and β are the condition scaling term and condition offset term in the modulation parameters, respectively; This is element-wise multiplication.

[0097] This perturbation condition adapter allows mid-to-high-level features to better adapt to different perturbation domains while keeping the visual encoder backbone frozen.

[0098] Step A4: Project the visual features onto the text feature dimension through a shared linear projection matrix, calculate the cosine similarity with several prompt word embeddings, obtain the prompt word response map, and construct the prompt word response tensor;

[0099] To map visual features to a forensic semantic space defined by artifact cue words, this invention pre-constructs a cue word library containing semantic concepts such as compression artifacts, color anomalies, boundary anomalies, frequency anomalies, blur anomalies, noise anomalies, generation artifacts, and geometric distortion. For the visual token f at position p of the l-th layer and k-th view... (l,k) (p) After projecting the text feature dimension using the shared linear projection matrix 𝑊, the cosine similarity is calculated with the embedding t of the 𝑚th prompt word to obtain the prompt word response map. The specific formula is as follows:

[0100] ;

[0101] In the formula, A (l,k,m) (p) is the prompt word response graph; s is the similarity scaling factor; t m Embed for the m-th cue word; For the lth layer A perturbation view in spatial location The visual token at the location; To share a linear projection matrix, which is used to map visual features to the text embedding dimension; It is an L2 norm; Indicates transpose;

[0102] This allows the construction of a dense cue word response tensor with multiple layers, multiple views, and multiple cue words: This serves as the semantic basis for subsequent statistical modeling across perturbations.

[0103] The specific process of obtaining the detection results based on the cue word response tensor through the perturbation response invariant modeling module includes the following steps:

[0104] Step B1: For the response changes of the same prompt word under different perturbation views, perform statistical modeling on the view dimension and calculate the cross-perturbation response variance plot to characterize the dynamic changes of evidence semantics under linear compression, ambiguity, scaling, noise, and color shift. The specific formula for the cross-perturbation response variance plot is as follows:

[0105] ;

[0106] To enhance numerical stability and suppress the influence of extreme values, logarithmic scaling is further performed, with the specific formula as follows:

[0107] ;

[0108] Finally, after instance normalization of the logarithmic scaling results for all layers and all prompt words, cross-perturbation statistical representations are obtained.

[0109] In the formula, V (l,m) (p) is the variance plot of the response across the disturbance; A (l,k,m) (p) is the cue word response graph; α is the scaling factor; This is the result of logarithmic scaling; Indicates the dimension along the perturbation view Calculate the variance above.

[0110] Step B2: Calculate the prompt word score based on the prompt word response tensor, and obtain the prompt word route weight through Stable Prompt Top-K Routing (SPTKR) selection and normalization;

[0111] Because the cue word database is large, the response strength of different samples to cue words varies under different perturbations. To reduce the interference of noisy cue words, this invention dynamically calculates the cue word score for each sample and performs stable cue word Top-K routing. The formula for calculating the cue word score of the m-th cue word is as follows:

[0112] ;

[0113] In the formula, s m Score points for the prompt words; A is a nonnegative function; (l,k,m) (p) is the prompt word response diagram;

[0114] Further increase the prior bias of the basic prompt words With temperature parameters Introducing route weight calculation and in the Top-K prompt word set The above is normalized to obtain the prompt word route weight, and the specific formula is as follows:

[0115] ;

[0116] In the formula, Assign route weights to prompt words; For indicator functions; It is a stable term; The normalized soft weight for the m-th cue word; Prior bias of basic prompt words; Score for prompt words; T is the temperature parameter; M is the total number of prompt words; This is the set of Top-K cue words.

[0117] The stable prompt word Top-K routing uses the same Top-K mechanism in both the training and inference phases. It assigns a base weight to prompt words that are not selected into the Top-K set and sets a minimum guarantee mechanism to retain one prompt word.

[0118] Step B3: Based on the cross-perturbation statistical representation and the prompt word routing weights, obtain the local discrimination results and global discrimination results through local branches and global branches respectively, and obtain the detection results after weighted fusion;

[0119] The local branch takes as input dense statistical maps across layers and across prompt words, the total response map of prompt words, and the weighted response map of prompt words. It extracts fine-grained clues such as local artifacts, boundary anomalies, and texture inconsistencies through a convolutional residual structure. The global branch performs spatial mean pooling and max pooling on the statistical representation across perturbations and constructs global statistical features by combining prompt word routing weights to learn a more stable overall discrimination pattern. Finally, it outputs a fusion result of the local and global discrimination results, with the specific formula as follows:

[0120] ;

[0121] In the formula, This is the local discrimination result output for the local branch; The global discrimination result output by the global branch; and These are the corresponding fusion weights.

[0122] To suppress the dependence of deepfake image detection models on specific perturbation types, a leave-of-sight approach is introduced. Figure 1 Perturbation invariant learning is performed using consistency constraints and domain adversarial perturbation confusion, and parameters are optimized for the deepfake image detection model through backpropagation.

[0123] Leave a glance Figure 1 The consistency constraint is that, for any view k, after removing that view, the cross-perturbation statistical representation is recalculated and the prediction is obtained. The loss function is:

[0124] ;

[0125] Domain adversarial perturbation and obfuscation is a method of representing the global representation. The perturbation domain classifier is input and subjected to domain adversarial training via a gradient inversion layer (GRL) to weaken the discriminative perturbation domain information in the representation. The loss function is:

[0126] ;

[0127] In the formula, D(·) is the perturbation domain classifier; d is the perturbation domain label; g is the global representation used for perturbation domain classification, which is obtained by statistically stitching together the spatial mean of the response maps of each layer; GRL(·) is the gradient reversal layer, which keeps the input unchanged during forward propagation and multiplies the gradient by a negative coefficient during backward propagation; CE(·,·) is the cross-entropy loss.

[0128] The binary cross-entropy loss function for the deepfake image detection model during the training phase is:

[0129] ;

[0130] The objective function is:

[0131] ;

[0132] In the formula, and To balance the weights for positive and negative samples; The function is the Sigmoid function; N is the batch size. ∈{0,1} represents the true label; For the first The predicted logit for each sample; For the regularization term of the adapter modulation parameters under disturbance conditions; To leave a view Figure 1 Induced loss; For domain-specific perturbation confusion loss; For the prior stable terms of the prompt words; For prompt word entropy regularization; , , , , These are the corresponding weighting coefficients.

[0133] The process of the present invention will be described below through an embodiment:

[0134] Step S1: Input image preprocessing and pixel domain mapping:

[0135] Face detection and region cropping are performed on the input face image, and the cropped face regions are uniformly scaled to the input resolution of the visual encoder. Then, image normalization is performed, and pixel-domain mapping is executed with preset probabilities during the training phase. Pixel-domain mapping adopts a mixed mode, consisting of a combination of fixed-point quantization with two decimal places and lightweight random affine transformations. The random translation range is [-0.05, 0.05], and the random scaling range is [0.98, 1.02]. The image after pixel-domain mapping is represented as follows:

[0136] ;

[0137] Step S2: Generate multi-view samples by fixing the perturbation ladder:

[0138] In this embodiment, the fixed perturbation step K is set to 6 views. During the training phase, the intensity of each perturbation is randomly sampled, while deterministic intensity is used during the inference phase. The JPEG compression quality factor ranges from [60, 95], the blur standard deviation ranges from [0.05, 0.8], the scaling factor ranges from [0.65, 0.95], the noise standard deviation ranges from [0.001, 0.02], and the color perturbation parameters are: brightness = 0.2, contrast = 0.2, saturation = 0.15, hue = 0.05. The k-th perturbation view is calculated as follows:

[0139] ;

[0140] Simultaneously, a condition vector is constructed for each view, consisting of a one-hot encoding of the perturbation type and a scalar representation of the perturbation intensity. This is for use by subsequent disturbance condition adapters.

[0141] Step S3: Multi-layer feature extraction and perturbation condition adaptation:

[0142] A frozen CLIP ViT-L / 14 visual encoder is used as the backbone network, and PatchTokens are extracted from layers 6, 9, and 12 as multi-layer visual representations. A perturbation conditional adapter with a bottleneck dimension of 64 is inserted at the 10th, 11th, and 12th residual block positions to perform perturbation domain modulation on the mid-to-high-level features. The calculation process of the perturbation conditional adapter is as follows:

[0143] ;

[0144] ;

[0145] ;

[0146] By adapting the above conditions, performance fluctuations caused by feature drift in different perturbation domains can be mitigated without fine-tuning the individual visual encoder.

[0147] Step S4: Construct the cue word response graphs A-maps:

[0148] In this embodiment, an artifact cue word library containing 40 cue words is constructed, covering semantic concepts such as JPEG artifacts, quantization noise, color inconsistency, shadow inconsistency, fusion boundary anomaly, frequency anomaly, blur, noise, generation traces, interpolation error, and geometric distortion.

[0149] After mapping the Patch Token of each level and each view to the text feature dimension through a shared linear projection, the cosine similarity is calculated with the cue word embedding to obtain the cue word response map:

[0150] ;

[0151] This leads to the construction of a multi-layer, multi-view prompt response tensor. This serves as the input for subsequent statistical modeling across disturbances.

[0152] like Figure 2 As shown, under fixed perturbation views such as the original image, JPEG compression, Gaussian blur, scaling resampling, additive noise, and color perturbation, multiple artifact prompts form a spatially consistent response distribution in the key facial regions; for example... Figure 3 As shown, the stable cue word Top-K routing assigns higher weights to high-response cue words such as Gaussian blur, high-frequency missing, color inconsistency, and motion blur, while retaining only the base weight for non-critical cue words, thereby reducing the interference of noisy cue words on sample-level discrimination.

[0153] Step S5: Construct a statistical representation of the response across disturbances:

[0154] response tensor to prompt words Variance statistics are performed on the view dimensions to obtain a cross-perturbation response variance plot. Then, logarithmic scaling and instance normalization are performed to enhance stability and suppress extreme values. The specific process is as follows:

[0155] ;

[0156] ;

[0157] like Figure 4 As shown, the statistical representation of the cross-perturbation response effectively highlights robust semantic regions such as Gaussian blur, texture inconsistency, high-frequency missing data, and upsampling artifacts at layers 6, 9, and 12, demonstrating the complementary characterization of local artifacts and overall statistical changes at different levels; for example... Figure 5 As shown, the detection probabilities for the local branch, global branch, and fusion result are given respectively: the probability of the local branch is 0.467, the probability of the global branch is 0.437, and the probability of the fusion result is 0.454. The fusion judgment combines local fine-grained evidence and global statistical evidence to obtain a more stable judgment of the tendency to be true or false. Based on the obtained probabilities, the overall judgment result is biased towards the truth.

[0158] In this embodiment, all Instance normalization is performed in the spatial dimension to obtain standardized cross-perturbation statistical representations V-maps.

[0159] Step S6: Build a stable Top-K route for the prompt words:

[0160] For each sample, the cue word score is calculated based on the responses across all layers, all views, and all spatial locations, using the following formula:

[0161] ;

[0162] In this embodiment, the number of Top-K words is 12, the temperature parameter T is 0.7, the basic prior scale of the cue words is 1.0, and the base scale of non-Top-K cue words is 0.01. The cue word scores are processed by softmax and Top-K selection to obtain the sample-level cue word weights, and the specific formula is as follows:

[0163] ;

[0164] During the training phase, a random drop probability of 0.25 is applied to the Top-K prompts, and a safety net mechanism of retaining at least one prompt is adopted; the same Top-K routing mechanism is still used during the inference phase to ensure consistency between training and testing.

[0165] Step S7: Local and global dual-branch fusion discrimination:

[0166] The local branch takes cross-layer, cross-word V-maps, the total response map of the prompts, and the weighted response map of the prompts as input. It extracts local spatial evidence using convolution and residual structures, with a local head hidden dimension of 256. The global branch performs spatial mean pooling and max pooling on the V-maps and combines them with weighted statistical features of the prompts. It then outputs the global discrimination result through a multi-layer perceptron (MLP), with a global MLP hidden dimension of 512. The local and global branches are finally fused into the total discrimination output, with the specific formula as follows:

[0167] ;

[0168] Training objective construction and model optimization:

[0169] The fused output is subjected to a class-balanced binary cross-entropy loss, with a label smoothing coefficient set to 0.1. Simultaneously, an adapter regularization term and a leave-one-look term are introduced. Figure 1 The overall training objective is as follows: Consistency loss, domain adversarial perturbation confusion loss, cue word stability term, and cue word entropy regularization term.

[0170] ;

[0171] ;

[0172] ;

[0173] ;

[0174] In this embodiment, , , , , .

[0175] This invention was trained on the FaceForensics++ (C23) training set and validated on Celeb-DF-v2, DFDC, DFDCP, DeeperForensics-1.0 (DF1.0), and FaceForensics++ high and low quality settings. The evaluation metric used was video-level AUC (%), and the video score was obtained by averaging 32 randomly selected frames.

[0176] Table 1. Cross-dataset evaluation results (Test Set AUC, %)

[0177]

[0178] As shown in Table 1, the present invention achieves AUCs of 90.11%, 78.82%, and 86.46% on the three cross-dataset benchmarks of CD2, DFDC, and DFDCP, respectively, all of which are higher than other comparative methods. This indicates that the present invention can more effectively improve the cross-domain robust detection capability under real propagation link perturbations by using fixed perturbation multi-view and semantic response statistical modeling.

[0179] Table 2. Robustness evaluation results on the DF1.0 std / rand dataset (Test Set AUC, %)

[0180]

[0181] As shown in Table 2, on the DF1.0 std / rand test set containing real perturbations of random type and random strength, the present invention achieves an AUC of 96.63%, which is higher than the other comparative methods, indicating that the present invention has more stable detection performance for perturbations of unknown strength and complex combinations.

[0182] Table 3. FF++ Cross-Quality Evaluation Results (Test Set AUC, %)

[0183]

[0184] As shown in Table 3, in the FF++ cross-quality evaluation, the present invention achieved AUC of 99.26% and 91.58% on the HQ and LQ test sets, respectively. The advantage was more obvious under the LQ setting, indicating that the present invention can still maintain high detection accuracy under severe compression and quality degradation scenarios.

[0185] In summary, this invention effectively improves the robustness of deepfake image detection under real post-processing perturbation conditions by using fixed perturbation multi-view, cue word aligned semantic mapping, cross-perturbation response statistics, and perturbation invariant learning, while maintaining interpretable semantic representation.

[0186] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A robust deepfake face detection method based on CLIP semantic guidance, characterized in that, include: A deepfake image detection model is constructed, comprising: a cue word alignment semantic mapping module and a perturbation response invariant modeling module. The cue word alignment semantic mapping module obtains the cue word response tensor of the input image, and the perturbation response invariant modeling module obtains the detection result based on the cue word response tensor. The specific process of obtaining the prompt word response tensor of the input image through the prompt word alignment semantic mapping module includes the following steps: Step A1: Perform pixel-domain mapping on the input image; Step A2: Construct a fixed perturbation ladder to generate several perturbation views for the pixel-domain mapped image; Step A3: Extract visual features using the frozen visual language model; Step A4: Project the visual features onto the text feature dimension through a shared linear projection matrix, calculate the cosine similarity with several prompt word embeddings, obtain the prompt word response map, and construct the prompt word response tensor; The specific process of obtaining the detection results based on the cue word response tensor through the perturbation response invariant modeling module includes the following steps: Step B1: Obtain the cross-perturbation statistical representation of the cue word response tensor; Step B2: Calculate the prompt word score based on the prompt word response tensor, and obtain the prompt word route weight through stable prompt word Top-K routing and normalization; Step B3: Based on the cross-perturbation statistical representation and the prompt word routing weights, obtain the local discrimination results and global discrimination results through local branches and global branches respectively, and obtain the detection results after weighted fusion.

2. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 1, characterized in that, In step A1, the pixel domain mapping function in the pixel domain mapping process includes: fixed decimal quantization, or random affine transformation, or a combination of fixed decimal quantization and random affine transformation.

3. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 1, characterized in that, In step A2, the perturbation ladder uses the same set of perturbation types in both the training and inference phases. During the training phase, the intensity of each perturbation is randomly sampled and set, while during the inference phase, the intensity of each perturbation is set deterministically. The perturbation types include original view, JPEG compression, blur, scaling, noise, and color perturbation. The formula for generating the perturbation view is: ； In the formula, I (k) Perturbation view; P k (I0) represents the k-th perturbation operator; I0 represents the image after pixel-domain mapping; and K represents the total number of perturbation operators.

4. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 1, characterized in that, Step A3 further includes optimizing the frozen visual language model, specifically as follows: Insert a perturbation conditional adapter into the residual block of the frozen visual language model. First, the perturbation conditional adapter performs a bottleneck residual transformation on the input parameters. The specific formula is as follows: ； In the formula, Visual label features are input to the perturbation condition adapter; and These are the lower projection matrix and the upper projection matrix, respectively, used to map the features to the bottleneck dimension and then restore them to the original feature dimension; The activation function for the Gaussian error linear unit; The disturbance compensation characteristics are obtained from the bottleneck residual transformation. Then, based on the conditional vector formed by the disturbance type and disturbance intensity, modulation parameters are generated, with the specific formula as follows: ； In the formula, For the first The condition vector of each perturbation view is obtained by concatenating the one-hot encoding of the perturbation type with the scalar of the perturbation intensity; For conditionally modulated multilayer perceptrons; and These are conditional scaling and conditional offset terms that are consistent with the visual mark dimensions, respectively; Finally, the output is generated based on the modulation parameters, and the specific formula is as follows: ； In the formula, y is the output of the disturbance condition adapter; γ and β are the condition scaling term and condition offset term in the modulation parameters, respectively; This is element-wise multiplication.

5. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 1, characterized in that, In step A4, the specific formula for the prompt word response graph is as follows: ； In the formula, A (l,k,m) (p) is the prompt word response graph; s is the similarity scaling factor; t m Embed for the m-th cue word; For the lth layer A perturbation view in spatial location Visual markers at the location; To share a linear projection matrix, which is used to map visual features to the text embedding dimension; It is an L2 norm; Indicates transpose; The constructed prompt response tensor is: Where L is the number of layers extracted by the visual encoder; K is the number of views of the fixed perturbation ladder; M is the number of cue words in the cue word library; and H and W are the spatial location dimensions.

6. The robust face deepfake detection method based on CLIP semantic guidance as described in claim 1, characterized in that, The specific process of step B1 is as follows: The specific formula for the cross-disturbance response variance plot is as follows: First, obtain the cross-perturbation response variance plot of the cue word response tensor. The specific formula is as follows: ； Then, the cross-disturbance response variance plot is log-scaled, using the following formula: ； Finally, after instance normalization of the log-scaling results, the cross-perturbation statistical characterization is obtained. In the formula, V (l,m) (p) is the variance plot of the response across the disturbance; A (l,k,m) (p) is the cue word response graph; α is the scaling factor; This is the result of logarithmic scaling; Indicates the dimension along the perturbation view Calculate the variance above.

7. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 1 or 6, characterized in that, In step B2, the formula for calculating the prompt word score is as follows: ； In the formula, s m Score points for the prompt words; A is a nonnegative function; (l,k,m) (p) is the prompt word response diagram; The stable prompt word Top-K routing uses the same Top-K mechanism in both the training and inference phases. Prompt words not selected for the Top-K set are assigned a base weight, and a minimum guarantee mechanism of retaining one prompt word is implemented. The specific formula for obtaining the prompt word route weight is as follows: ； In the formula, w m Assign route weights to prompt keywords; For indicator functions; It is a stable term; The normalized soft weight for the m-th cue word; b m Prior bias of basic prompt words; Score points for the prompt words; T represents the temperature parameter; M represents the total number of prompt words. This is the set of Top-K cue words.

8. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 1, characterized in that, The training phase of the deepfake image detection model also includes: introducing leave-one-view consistency constraints and domain adversarial perturbation confusion to perform perturbation invariant learning, and optimizing the parameters of the deepfake image detection model through backpropagation.

9. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 8, characterized in that, The loss function for the leave-one-view consistency constraint is: ； The loss function for domain adversarial perturbation confusion is: ； In the formula, D(·) is the perturbation domain classifier; d is the perturbation domain label; g is the global representation used for perturbation domain classification, which is obtained by statistically stitching together the spatial mean of the response maps of each layer; GRL(·) is the gradient reversal layer, which keeps the input unchanged during forward propagation and multiplies the gradient by a negative coefficient during backward propagation; CE(·,·) is the cross-entropy loss.

10. The robust deepfake face detection method based on CLIP semantic guidance as described in claim 8, characterized in that, The binary cross-entropy loss function of the deepfake image detection model during the training phase is: ； The objective function is: ； In the formula, and To balance the weights for positive and negative samples; The function is the Sigmoid function; N is the batch size. ∈{0,1} represents the true label; For the first The predicted logit for each sample; For the regularization term of the adapter modulation parameters under disturbance conditions; To avoid loss of view consistency; For domain-specific perturbation confusion loss; For the prior stable terms of the prompt words; For prompt word entropy regularization; , , , , These are the corresponding weighting coefficients.

Images