Deepfake detection method and device based on visual language model, and medium

Abstract

The present application relates to a kind of deep fake detection method, equipment and medium based on visual language model, the method of present application obtains pre-trained visual-language model, including visual encoder and text encoder;Through single-hop perturbation, original text prompt is disturbed on sensitive attribute dimension, and generates counterfactual text prompt;Freeze text encoder and extract text feature, while extracting the visual feature of input image by visual encoder and calculating the similarity between each text feature;According to the difference between visual feature and the similarity of original and counterfactual prompt, adjust the affine parameter of normalization layer in visual encoder middle layer, obtain adjusted visual encoder;Finally, the image to be detected is input into the encoder, and the visual feature based on extraction is output Deep fake detection result.The present application remodels feature geometry, eliminates demographic bias while maintaining the highest detection accuracy.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, specifically to a method, device, and medium for deepfake detection based on a visual language model. Background Technology

[0002] Deepfake technology utilizes artificial intelligence methods such as generative adversarial networks and diffusion models to tamper with or synthesize faces in images and videos, including face replacement, expression replay, and attribute editing. Therefore, developing efficient and accurate deepfake detection technology has become an important research direction in the fields of digital forensics and information security.

[0003] Early deepfake detection methods primarily relied on hand-designed features or traditional convolutional neural networks, distinguishing real content from forgeries by analyzing clues such as spatial texture, frequency domain artifacts, and temporal inconsistencies in images. However, these methods were often designed for specific forgery types and had limited generalization ability when faced with unknown generative models or cross-dataset scenarios.

[0004] In recent years, visual-language models (VLMs) such as CLIP (Contrastive Language-Image Pre-training) have demonstrated powerful representation capabilities in multimodal learning. Research shows that deepfake detection methods based on visual-language models can achieve superior performance compared to traditional dedicated detectors with only a small number of labeled samples, making them a cutting-edge technology in the field. However, as research has deepened, scholars have discovered serious fairness issues with these models: different demographic groups exhibit systematic differences in detection results, with some groups having a significantly higher probability of misclassifying forged content than others. For example, existing research shows that advanced detectors exhibit asymmetric accuracy across different groups; similarly, models produce inconsistent predictions for male and female samples. This detection bias, dependent on identity features, means that the model focuses more on "who appears in the content" than "whether the content is forged," violating the fundamental principle that deepfake detection should be independent of identity information.

[0005] To address the aforementioned fairness issues, existing technologies have proposed a series of mitigation methods. One type of method attempts to balance the error rate across different subgroups by introducing demographic labels for data reweighting or split-bulk optimization. Another type employs feature decoupling techniques to separate forgery-related features from demographic features, or explicitly incorporates fairness constraints into the loss function. However, these methods all treat fairness as an externally imposed objective, intervening post-hocly through feature manipulation or regularization, failing to reveal the internal mechanisms of bias generation, and thus struggling to fundamentally eliminate bias. Furthermore, existing debiasing techniques for vision-language models are primarily geared towards general vision tasks, such as cue editing, representation projection, or adapter-based fine-tuning. These methods do not consider the complex interactions between forgery cues and demographic attributes in multimodal space during deep forgery detection, nor can they identify the origin of bias within the model.

[0006] Therefore, how to locate and correct the structural root causes of demographic bias in visual-language models from the perspective of the model's internal mechanisms, and achieve cross-group fairness while maintaining high detection accuracy, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] This invention provides a method, device, and medium for deepfake detection based on a visual language model. Its purpose is to solve the fairness collapse problem caused by the amplification of demographic bias in existing deepfake detection methods based on visual language models, so as to achieve identity-independent, fair, and accurate detection.

[0008] To achieve the above objectives, the first aspect of the present invention provides a deepfake detection method based on a visual language model, comprising the following steps:

[0009] Obtain a pre-trained visual-language model, wherein the visual-language model includes a visual encoder and a text encoder;

[0010] For an input image, construct its original text prompt and at least one counterfactual text prompt, which is generated by single-hop perturbation of the original text prompt on the sensitive attribute dimension;

[0011] The text encoder is kept frozen, and text features of the original text prompt and counterfactual text prompt are extracted;

[0012] The visual encoder extracts the visual features of the input image and calculates the similarity between them and each text feature.

[0013] Based on the difference in similarity between the visual features and the original text prompts and counterfactual text prompts, the affine parameters of the normalization layer in the visual encoder are adjusted to obtain the adjusted visual encoder.

[0014] The image to be detected is input into the adjusted visual encoder to extract visual features, and the depth forgery detection result is output based on the extracted visual features.

[0015] Furthermore, the method for constructing the original text prompt and the counterfactual text prompt includes:

[0016] For the input image, based on its real label and corresponding original sensitive attributes, generate original real prompts and original fake prompts containing sensitive attribute descriptions, as the original text prompts;

[0017] Based on the original sensitive attributes, at least one new set of sensitive attributes is obtained by performing single-hop perturbation on the sensitive attribute dimension. The single-hop perturbation is to change the value of only one sensitive attribute dimension in a single operation.

[0018] Based on the new sensitive attributes and the real tags, counterfactual truth prompts and counterfactual false prompts are generated as counterfactual text prompts;

[0019] The original sensitive attributes include at least a first sensitive attribute and a second sensitive attribute, and the sensitive attribute dimensions include at least a first sensitive attribute dimension and a second sensitive attribute dimension.

[0020] Furthermore, methods for extracting textual features from original text prompts and counterfactual text prompts include:

[0021] The original real prompts and the original fake prompts are respectively input into the frozen text encoder to obtain the original real text features and the original fake text features;

[0022] The counterfactual and counterfactual false prompts are input into the frozen text encoder to obtain counterfactual text features and counterfactual false text features.

[0023] Furthermore, the method for calculating its similarity with each text feature includes:

[0024] Calculate the first similarity between the visual feature and the original real text feature, the second similarity between the visual feature and the original fake text feature, the third similarity between the visual feature and the counterfactual real text feature, and the fourth similarity between the visual feature and the counterfactual fake text feature;

[0025] Each similarity is obtained by multiplying the inner product of the visual feature and the corresponding text feature by a learnable scaling parameter.

[0026] Furthermore, the method for adjusting the affine parameters of the normalization layer in the visual encoder based on the difference in similarity between the visual features and the original text prompt and the counterfactual text prompt includes:

[0027] The original similarity difference is calculated based on the first similarity and the second similarity, wherein the original similarity difference is the difference between the second similarity and the first similarity:

[0028]

[0029] in, For the original similarity difference, This represents the first similarity between visual features and the original, real-text features. The second similarity between visual features and features of the original fake text;

[0030] The counterfactual similarity difference is calculated based on the third and fourth similarities, and the counterfactual similarity difference is the difference between the fourth and third similarities:

[0031]

[0032] in, To counterfactual similarity differences, The third similarity score represents the visual features and the counterfactual text features. The fourth similarity score represents the visual features and the features of counterfactual and false text.

[0033] Construct a counterfactual isometric alignment loss based on the mean squared error between the original similarity difference and the counterfactual similarity difference:

[0034]

[0035] in, For counterfactual isometric alignment loss, For batch size, For sample index within a batch, This represents the total number of samples within the batch.

[0036] The total loss function is constructed by combining the classification loss and the counterfactual isometric alignment loss:

[0037]

[0038] in, For the total loss function, For classifying losses, The weighting coefficients are used to balance the strength of the counterfactual equidistant alignment constraint;

[0039] Based on the total loss function, the affine parameters of the normalized layer in the visual encoder are updated using the gradient descent method.

[0040] Furthermore, the layer normalization layer operation is as follows:

[0041]

[0042] in, Indicates the first Layer normalization operation; For the first Layer input; For input The mean; For input Standard deviation; This indicates element-wise multiplication; and The first The layer can learn scaling and translation parameters;

[0043] During the update process, all parameters of the visual encoder, except for the layer normalization layer affine parameter, and all parameters of the text encoder are kept frozen.

[0044] Furthermore, the method of inputting the image to be detected into the adjusted visual encoder, extracting visual features, and outputting depth forgery detection results based on the extracted visual features includes:

[0045] The image to be detected is input into the adjusted visual encoder to obtain the adjusted visual features;

[0046] The adjusted visual features are input into the linear classification head to obtain the forgery probability prediction result;

[0047] The detection method determines whether the image to be detected is a deepfake image based on the forgery probability prediction result.

[0048] Furthermore, the sensitive attribute dimension includes a first sensitive attribute dimension and a second sensitive attribute dimension, and the single-hop perturbation is to randomly select one dimension from the first sensitive attribute dimension and the second sensitive attribute dimension to change, and only the attribute value of one dimension is changed each time.

[0049] To achieve the above objectives, a second aspect of the present invention provides an electronic device including a memory and a processor, the memory being used to store a program that supports the processor in executing the deepfake detection method based on a visual language model, and the processor being configured to execute the program stored in the memory.

[0050] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the deepfake detection method based on a visual language model.

[0051] The beneficial effects of this invention are:

[0052] Compared with existing technologies, this invention provides a deepfake detection method, device, and medium based on a visual-language model. Through in-depth analysis of the internal mechanism of the visual-language model in deepfake detection, it discovers and locates for the first time the structural root of demographic bias in the layer normalization layer of the visual encoder. This module anisotropically amplifies identity cues such as gender and demographic attributes during feature calibration, while suppressing genuine signals related to forgery. To solve this technical challenge, this invention employs a counterfactual text prompt construction strategy, generating and modifying the original prompt by performing single-hop perturbation on the sensitive attribute dimension. The model employs a semantically dual counterfactual description and keeps the text encoder completely frozen as a stable semantic anchor. It then calculates the similarity difference between visual features and original and counterfactual text features. Based on this difference, a counterfactual isometric alignment loss is constructed to guide the fine-tuning of the affine parameters of the normalized layer. This allows the feature geometry to be reshaped from identity clustering-dominated to real-fake axis alignment with only a few parameters adjusted. Ultimately, the model completely eliminates its dependence on demographic attributes during detection and achieves the best detection accuracy and group fairness metrics in experiments across multiple deepfake datasets. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0054] Figure 1 This is a flowchart of a deepfake detection method based on a visual language model disclosed in an embodiment of the present invention.

[0055] Figure 2 This is a t-SNE visualization of the LN layer representation of CLIP trained on FF++ disclosed in an embodiment of the present invention, wherein (a) is a gender-dominated clustering graph formed by CLIP, and (b) is a graph showing that the geometry is mainly aligned along the real-fake axis after using CIA.

[0056] Figure 3 This is an IsoCLIP framework diagram disclosed in an embodiment of the present invention. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0058] According to embodiments of the present invention, it should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the following methods, in some cases the steps shown or described may be executed in a different order than that shown here.

[0059] This invention relates to the field of digital image forensics and information security, specifically to the detection of deepfakes in facial images widely circulated on internet platforms, social media, or in forensic settings. The method uses a facial image as input, extracts visual features using an adjusted visual encoder, and ultimately outputs the detection result of whether the image is a deepfake. This provides reliable technical support for content moderation systems on social media platforms and authenticity verification systems for news organizations.

[0060] like Figure 1 As shown, this invention provides a deepfake detection method based on a visual language model, the method comprising:

[0061] Step S100: Obtain a pre-trained visual-language model, wherein the visual-language model includes a visual encoder and a text encoder;

[0062] A visual-language model is a multimodal model trained through comparative learning on massive image-text pairs, capable of mapping images and text to the same semantic space for similarity calculation. In this embodiment, the CLIP model is preferably used as the pre-trained visual-language model, specifically the CLIP ViT-L / 14 variant. This model uses a VisionTransformer (ViT) as the visual encoder and a Transformer as the text encoder, enabling it to encode the input image and text separately. Dimensional embedding vector.

[0063] The visual encoder is denoted as The text encoder is denoted as The embedding vectors output by both are located in a shared space. In the dimensional embedding space, that is, for the input image and text description ,have:

[0064]

[0065]

[0066] in, and express 3D image and text embedding; for The model uses a real vector space to represent the dimension of the embedded vectors. This pre-trained model achieves powerful cross-modal alignment capabilities by maximizing the similarity of matching image-text pairs and minimizing the similarity of non-matching pairs.

[0067] It should be noted that the visual encoder in the CLIP model contains multiple Transformer blocks, each with a Layer Normalization (LN) layer. Layer normalization is a technique used to stabilize deep network training. It is achieved by normalizing the mean and variance of the layer inputs and applying a learnable affine transformation. Its mathematical expression will be explained in detail in subsequent steps. The pre-trained model parameters obtained in this step will serve as the basis for subsequent fine-tuning. Except for the affine parameters of the layer normalization layer of the visual encoder, all other parameters remain in their initial pre-trained state.

[0068] Step S200: For the input image, construct its original text prompt and at least one counterfactual text prompt, wherein the counterfactual text prompt is generated by single-hop perturbation of the original text prompt on the sensitive attribute dimension;

[0069] When constructing text prompts, two sets of text descriptions need to be generated for each input image: one set is the original text prompt that retains the original demographic semantics, and the other set is the counterfactual text prompt obtained by changing the demographic attributes.

[0070] First, for the input image It is necessary to obtain its real tags. ,in Represents a real image. This indicates a forged image; Assign a sample index number. Simultaneously, it is necessary to obtain the original sensitive attributes corresponding to the image; this embodiment includes at least the first sensitive attribute. Second sensitive attribute ,For example You can choose either the "first attribute value (male)" or the "second attribute value (female)". The values ​​can be categorized as "third attribute value (e.g., population in region a)," "fourth attribute value (e.g., population in region b)," "fifth attribute value (e.g., population in region c)," or "sixth attribute value (e.g., population in region d)." Based on this information, an original text prompt pair is constructed. The original true prompt Use a template format of "a real image of a person with [gender attribute] [demographic attribute]", such as "a real image of a person with the first attribute of a certain region a"; original false prompts Use the template format of "a fake image of a person with [first attribute] [demographic attribute]", such as "a fake image of a woman or man in region b".

[0071] Next, counterfactual text prompts are constructed. The generation of counterfactual text prompts is based on a "single-hop perturbation" strategy, that is, within the original sensitive attributes... Based on this, only the value of one sensitive attribute dimension is changed, while the other dimension remains unchanged. Define a single-hop neighborhood. For the set of all attribute pairs that differ from the original attribute in exactly one dimension:

[0072]

[0073] in, This indicates that only the gender attribute is changed (from Become Meanwhile, demographic attributes remain unchanged. ; This indicates that only demographic attributes have been changed (from Become While gender attributes remain unchanged. ; The gender attribute after perturbation; These are the demographic attributes after the disturbance.

[0074] Randomly select a new attribute pair from this single-hop neighborhood. As a perturbation-sensitive attribute, for example, "the second attribute value of the first attribute value" can be perturbed to "the second attribute value of the third attribute value" (changing only the second sensitive attribute) or "the first attribute value of the first attribute value" (changing only the first sensitive attribute). Based on the perturbation-sensitive attribute. and the same real label Constructing counterfactual text prompts .

[0075] Among them, the counterfactual truth is revealed. Using a template of "a real image of a [perturbed sex] [perturbed population] person", counterfactual falsehoods are presented. The template used is “a fake image of a person from a [perturbed post-population] with [perturbed gender]. In this way, the original cue and the counterfactual cue differ only in demographic attributes, while the semantic structure (real or fake) remains consistent, providing training samples for subsequent forced model alignment that is invariant to identity.

[0076] It should be noted that the "hop" in "single-hop perturbation" refers to a single attribute modification operation, which changes only one sensitive attribute dimension at a time. This local perturbation method can generate semantically controllable counterfactual samples, avoiding the introduction of excessive semantic shifts due to simultaneous changes in multiple attributes, thereby more accurately guiding the model to focus on the impact of changes in demographic attributes on the prediction results.

[0077] Step S300: Keep the text encoder frozen and extract the text features of the original text prompt and the counterfactual text prompt;

[0078] After constructing the text prompts, this step performs forward computation on the text encoder to extract text features. The text encoder uses the pre-trained CLIP text encoder. This encoder, based on the Transformer architecture, is capable of encoding input text sequences into... The embedding vector is dimensional. In this implementation, all parameters of the text encoder are frozen, meaning they do not participate in gradient updates during training and are used only as a fixed feature extractor. The purpose of freezing the text encoder is to preserve the stable cross-modal alignment capability learned by the CLIP model during the pre-training phase, making it an invariant semantic anchor and avoiding the introduction of additional semantic drift due to changes in text features.

[0079] Specifically, for the original text prompts constructed in step S200 and counterfactual text prompts Input them into the frozen text encoder respectively. This yields the corresponding text feature vector:

[0080]

[0081]

[0082] in, Indicates the first The original real text features of each sample Indicates the characteristics of the original fake text. Indicates textual features that contradict the facts. Characteristics of counterfactual and false text; This is a fake text prompt; This is the original, real text prompt. All text features reside in the same shared embedding space as the visual features, with dimensions... Determined by the pre-trained model, for example, for the CLIP ViT-L / 14 model. These text features will serve as fixed anchors for subsequent image-text similarity calculations, measuring the alignment of visual features under different semantic descriptions.

[0083] Step S400: Extract the visual features of the input image through the visual encoder and calculate the similarity between them and each text feature;

[0084] For the input image via visual encoder Extract its visual features. The visual encoder uses CLIP's visual Transformer architecture, which can encode images into... The embedding vector is dimensional. During model training, the main parameters of the visual encoder (including convolutional layers, self-attention layers, etc.) remain frozen, while only the affine parameters of the layer normalization (LayerNorm) layers are frozen. It can be updated (the update process is performed in step S500). Therefore, the visual encoder at the current moment is denoted as... This indicates that the layer normalization parameters are in the state of the current iteration step. For the input image The extracted visual features are represented as follows:

[0085]

[0086] in, yes 3D visual embedding vector.

[0087] Next, the similarity between this visual feature and the four text features obtained in step S300 is calculated. Image-text similarity is defined as:

[0088]

[0089] in, Visual features Text features The similarity between them; Represents the vector dot product. It is a learnable logit scaling parameter used to adjust the scale of similarity; These are underlying learnable scalar parameters, which are ensured through exponential mapping. It is a positive number; For text feature vectors, Possible values , , , This scaling parameter is optimized along with the layer normalized affine parameters during training to adaptively adjust the contrast.

[0090] Based on the above definition, the similarity between the visual features and the four text features is calculated respectively:

[0091] Features of the original real text Similarity (first similarity): ;

[0092] Features of the original fake text Similarity (second similarity): ;

[0093] Features of Counterfactual Real Text Similarity (third similarity): ;

[0094] Features of counterfactual false texts Similarity (fourth similarity): .

[0095] These four sets of similarity measures quantify the degree of alignment of current visual features under different semantic descriptions (original / counterfactual, real / false).

[0096] It should be noted that the inner product operation measures the directional consistency between vectors in the shared embedding space; higher similarity indicates a better match between visual features and textual descriptions. Learnable scaling parameters... It can dynamically adjust the numerical range of similarity, enabling the model to better balance the contrast intensity of different samples during training.

[0097] Step S500: Based on the difference between the similarity between the visual features and the original text prompts and counterfactual text prompts, adjust the affine parameters of the normalization layer in the visual encoder to obtain the adjusted visual encoder.

[0098] In this embodiment, demographic bias is corrected by constructing a counterfactual isometric alignment loss and adjusting the affine parameters of the normalization layer in the visual encoder. First, based on the four sets of similarities calculated in step S400, the similarity difference under the original prompt and the similarity difference under the counterfactual prompt are calculated respectively. The original similarity difference is defined as the difference between the original false prompt similarity and the original true prompt similarity, used to quantify the model's tendency to judge the semantics of "false" and "true" under the current visual features.

[0099]

[0100] in, For the original similarity difference, This represents the first similarity between visual features and the original, real-text features. This represents the second similarity between visual features and features of the original fake text.

[0101] Similarly, a counterfactual similarity difference is calculated based on the third and fourth similarities, whereby the counterfactual similarity difference is the difference between the fourth and third similarities:

[0102]

[0103] in, To counterfactual similarity differences, The third similarity score represents the visual features and the counterfactual text features. The fourth similarity score represents the visual features and the features of counterfactual false text.

[0104] These two differences reflect the gap in the model's judgment of image authenticity under raw demographic semantics and counterfactual demographic semantics.

[0105] To ensure consistent judgments across different demographic descriptions, this invention constructs a counterfactual isometric alignment loss, forcing the original similarity difference to be as close as possible to the counterfactual similarity difference. This loss is defined based on mean squared error (MSE):

[0106]

[0107] in, For counterfactual isometric alignment loss, This is the batch size (i.e., the number of samples participating in the current iteration). For sample index within a batch, This represents the total number of samples within the batch. It is the square of the L2 norm, that is, the square of the Euclidean distance.

[0108] The purpose of this loss function is to generate a large loss value when the model makes inconsistent true-false judgments of the same image under different demographic descriptions, thereby driving the model parameters to be updated in the direction of reducing this inconsistency.

[0109] At the same time, the model also needs to maintain accuracy for the deepfake detection task itself. Therefore, visual features are used... Input linear classification head The predicted forgery probability is obtained. And calculate the classification loss. The classification loss uses the standard binary cross-entropy loss:

[0110]

[0111] in, The true label for the image (0 indicates real, 1 indicates fake); For the first The true label for each sample is 0 for a real image and 1 for a fake image. For the first The predicted forgery probability of each sample; The classification head function is a linear classification function. Combining the classification loss with the counterfactual isometric alignment loss yields the total loss function:

[0112]

[0113] in, For the total loss function, For classifying losses, The weighting coefficients are used to balance the strength of the counterfactual equidistant alignment constraint; This is the counterfactual isometric alignment loss.

[0114] Based on the total loss function The model parameters are updated using gradient descent. During the update process, it's necessary to determine which parameters participate in the gradient update: the text encoder. All parameters of the visual encoder remain frozen and are not updated; In addition to the affine parameters of the normalized layer, In addition, all other parameters (including those of convolutional layers, self-attention layers, and multilayer perceptron layers) remain frozen. Therefore, only the scaling parameters of the normalization layers in the visual encoder are frozen. Translation parameters Participate in gradient updates.

[0115] The layer normalization operation is defined as follows:

[0116]

[0117] in, Indicates the first Layer normalization operation; For the first Layer input; For input The mean; For input Standard deviation; This indicates element-wise multiplication; and The first The layer can learn scaling and translation parameters.

[0118] Counterfactual isometric alignment loss during backpropagation right and The gradient has an explicit form:

[0119]

[0120]

[0121] in, For counterfactual isometric alignment loss, adjust the scaling parameter The gradient; For counterfactual isometric alignment loss, the translation parameters are adjusted accordingly. The gradient; and These represent the similarity differences between the original image and the counterfactual image (i.e., and ), For learnable logit scaling parameters, The features are normalized. The text features differ. These gradient expressions show that the counterfactual isometric alignment loss can directly pass the bias correction signal to the affine parameters of the layer normalization, while the influence on other encoder layers is suppressed due to the inherent properties of layer normalization.

[0122] Layer normalization operation has a suppressive effect on upstream parameter perturbations, that is, it satisfies... and This causes the impact of the previous layer parameter perturbation on the final output to be attenuated to Magnitude, of which For layer normalization projection operators; The vector to be projected; The compression factor is much less than 1, indicating that layer normalization strongly suppresses upstream disturbances; however, disturbances to the layer-normalized affine parameters completely bypass normalization, and their effect is... The magnitude is much larger than the impact of previous layer perturbations. This theory ensures that the direction of deviation in the feature space can be efficiently controlled by normalizing the affine parameters of the microlayer alone, without adjusting a large number of other parameters.

[0123] In this manner, during each iteration, the model receives a batch of data, calculates similarity, difference, and loss through forward propagation, calculates gradients through backpropagation, and updates only the normalized layers of all layers in the visual encoder. and Parameters. The training process lasts for multiple epochs (10 epochs in this implementation), using the AdamW optimizer, with an initial learning rate set to... The batch size is set to 16. After training, the adjusted visual encoder is obtained. Its layer-normalized affine parameters have been optimized to suppress demographic bias and maintain the identity of the real-fake decision boundary. This fine-tuning method only adjusts 0.03% of the total model parameters, but achieves effective reshaping of the feature geometry, transforming the original identity attribute-driven clustering into a distribution aligned with the real-fake axis, thereby fundamentally solving the fairness collapse problem of visual-language models in deepfake detection.

[0124] Step S600: Input the image to be detected into the adjusted visual encoder, extract visual features, and output the depth forgery detection result based on the extracted visual features.

[0125] After completing model training and obtaining the adjusted visual encoder, this step applies the trained model to a practical deepfake detection task. During training, step S500 fine-tunes only the affine parameters of the normalization layer in the visual encoder, while keeping the text encoder completely frozen and other main parameters of the visual encoder unchanged. The adjusted visual encoder obtained after training is denoted as [insert denoting here]. Its layer normalization parameters have been optimized to suppress demographic bias and maintain the identity of the true-false decision boundary. During the inference phase, forward propagation only requires the adjusted visual encoder and linear classification head, without needing to call the text encoder again or construct counterfactual cues.

[0126] For any image to be detected First, it is input into the adjusted visual encoder to extract visual features. This process is represented as:

[0127]

[0128] in, For the adjusted visual encoder output 3D visual embedding vector; This is the adjusted visual encoder obtained after training. It should be noted that the main architecture of the visual encoder remains the same as during pre-training, with only the affine parameters of the normalization layers being fine-tuned. Therefore, the extracted visual features have been incorporated into the debiased geometric structure, enabling them to more accurately reflect the realism of the image rather than demographic attributes.

[0129] Next, the extracted visual features are input into a linear classification head to obtain the forgery probability prediction result. This process is represented as:

[0130]

[0131] in, This is a linear classification head, typically composed of a single-layer fully connected network. Its parameters are optimized along with the layer-normalized affine parameters during training in step S500. Output is a scalar representing the probability that the image to be detected is predicted to be a fake image. The closer the value is to 1, the higher the confidence level of the model in judging it as fake. The closer the value is to 0, the higher the confidence level of the model in judging it as real.

[0132] Finally, based on the forgery probability prediction results, it is determined whether the image to be detected is a deepfake image. A decision threshold is typically set. (For example ),like If so, the image is determined to be a depth fake image; if If the image is deemed to be real, then it is determined to be a real image. This decision threshold can be adjusted according to the specific application scenario's requirements for precision and recall.

[0133] Figure 2 The t-SNE visualization results of the CLIP model trained on the FF++ dataset without the processing method of this invention are shown. Figure 2 As shown in (a), the model forms gender-dominated clusters, with the separation between male and female videos even exceeding the separation between real and fake videos, indicating that demographic clues dominate the feature space. This qualitative observation reveals the inherent fairness collapse of visual-language models in deepfake detection, namely, that the model focuses more on "who appears in the video" than on "whether the video is faked". Figure 2 (b) shows that after using CIA, the geometry is mainly aligned along the real-fake axis.

[0134] Figure 3 The overall framework of the IsoCLIP method of this invention is illustrated in the figure. As shown, the method comprises three main parts: a visual encoder based on layer normalization fine-tuning, a frozen text encoder as semantic anchors, and a counterfactual isometric alignment module. The histogram in the figure reveals through uniform parameter perturbation analysis that the layer normalization layer is the most bias-sensitive component, with a perturbation response much higher than other layers. The flowchart in the figure illustrates the complete process from the input image and counterfactual text prompts, through feature extraction, similarity calculation, loss construction, to parameter update.

[0135] To verify the effectiveness of the method of the present invention (hereinafter referred to as IsoCLIP), a systematic experimental evaluation was conducted on several publicly available deepfake detection datasets.

[0136] Five widely used face forgery detection benchmarks were selected. FaceForensics++ (FF++) was used as the in-domain training set, containing five typical tampering types: DeepFakes, Face2Face, FaceSwap, NeuralTextures, and FaceShifter. Out-of-domain test sets included DFD, DFDC, Celeb-DF, and FairFD, specifically designed for fairness analysis. The FairFD dataset covers multiple diverse demographic groups, containing approximately 11,000 identities, over 52,000 real images, and corresponding 11 forgery variants, totaling over 600,000 samples, making it an ideal benchmark for evaluating group fairness. Demographic annotations used data provided by Ju et al., dividing the dataset into eight subgroups for fine-grained fairness analysis.

[0137] Accuracy was measured using the area under the receiver operating characteristic (AUC). Fairness was assessed using six complementary measures: demographic equality difference (DPD), equal opportunity difference (DEOdds), equal opportunity difference (DEO), false positive rate fairness (FFPR), overall accuracy equality fairness (FOAE), and population accuracy standard deviation (STD). All fairness measures were evaluated with lower values ​​as better, and AUC with higher values ​​as better.

[0138] Comparisons were made with three representative methods: spatial detectors (RECCE, UCF, CapsuleNet, FFD, CORE); frequency-based detectors (F3Net, SRM); fairness enhancement detectors (DAG, DAW, PFGDFD and its pruned variant BPFA); and a VLM-based debiasing baseline (DebiasCLIP). All comparison models were either retrained or used with the officially provided optimal weights.

[0139] All experiments were implemented using PyTorch and a single NVIDIA GeForce RTX 5090 GPU. The optimizer used was AdamW with an initial learning rate of 2×10⁻⁶. -4 The batch size was 16, and the training lasted for 10 epochs. The counterfactual isometric alignment loss weights were set to 1.2, and mixed precision training was used. The visual encoder was initialized with CLIP ViT-L / 14 pre-trained weights, and the input images were pre-processed using CLIP normalization.

[0140] Table 1 presents a detailed comparison with various detectors on the FairFD benchmark. The IsoCLIP method of this invention achieves the lowest values ​​across all fairness metrics, while maintaining an AUC of 94.56%, significantly outperforming existing methods. Specifically, compared to spatially based detectors, IsoCLIP's DPD (0.79) is nearly 9 points lower than the optimal Capsule (9.68); compared to the fairness-enhanced PFGDFD, the DEO decreases significantly from 9.51 to 1.65; and compared to DebiasCLIP, which is also CLIP-based, the AUC is improved by more than 21 percentage points, while all fairness metrics are comprehensively optimized. This indicates that this invention effectively shifts the model's focus from demographic attributes to forgery traces by correcting layer normalization bias.

[0141] Table 1 Comparison of different detector series on the FairFD benchmark

[0142]

[0143] To further verify its generalization ability, cross-domain tests were conducted on four datasets: FF++, CDF, DFD, and DFDC. The results are summarized in Table 2. IsoCLIP achieved AUC leadership on most datasets while maintaining the lowest fairness bias. For example, on the highly challenging DFDC dataset, IsoCLIP achieved an AUC of 74.54%, far exceeding F3Net (59.73%) and PFGDFD (63.52%), while significantly reducing metrics such as DPD and DEO. This demonstrates that the method of this invention is not only effective in correcting bias but also generalizes well to unknown forgery types and distributions.

[0144] Table 2. Main results on FF++, CDF, DFD, and DFDC

[0145]

[0146] To verify the necessity of each core component, an ablation study was conducted on the FairFD dataset, and the results are shown in Table 3. When the counterfactual isometric alignment loss (w / o CIA) was removed, the DPD increased from 0.79 to 1.27, the DEO increased from 1.65 to 2.74, the FFPR surged to 30.98, and the AUC decreased by 1.67. Further removal of layer normalization tuning (w / o LN tuning) caused a complete performance collapse, with the AUC plummeting to 85.14, severely deteriorating the fairness metric. This indicates that a frozen backbone network alone cannot handle demographic bias, and that robust bias correction requires the combined use of CIA loss and LN tuning.

[0147] Table 3 Ablation of CIA and LN fine-tuning components on FairFD

[0148]

[0149] A comparison of different fine-tuning strategies (Table 4) shows that Vanilla CLIP, with full-parameter fine-tuning, exhibits extremely poor fairness (DPD 7.81, DEO 9.57) and an AUC of only 67.03. While Adapter and LoRA can improve AUC, their fairness metrics fluctuate drastically; for example, Adapter's FFPR is as high as 33.49, and LoRA's DEO reaches 12.56. Only Layer Normalization Tuning (LN Tuning) maintains an AUC of 94.56 while minimizing all fairness metrics, verifying that restricting updates to the normalized layer is the optimal choice.

[0150] Table 4 Ablation using different fine-tuning strategies on FairFD

[0151]

[0152] The form of counterfactual isometric regularization is also crucial. Table 5 compares the effects of sample-level MSE constraints with distribution-level KL divergence and Wasserstein distance. MSE loss achieves the best balance between fairness and accuracy, while KL divergence causes DEO to rise to 2.28 and AUC to drop to 92.05; although Wasserstein distance has a slightly lower DPD, its FFPR is as high as 20.70. This indicates that hard sample-level geometric constraints are better at guaranteeing local counterfactual consistency than distribution-level alignment.

[0153] Table 5 Ablation using different regularization methods on FairFD

[0154]

[0155] The construction strategy of counterfactual text prompts also affects performance (Table 6). Jointly perturbing the population and gender (two hops) increases the FFPR to 11.56 and decreases the AUC to 92.87; while perturbing only the population or gender has its advantages and disadvantages, the fairness index is worse than random single-hop perturbation. The random single-hop strategy (randomly selecting the population or gender for a single attribute change each time) achieves the lowest DPD, DEO, and FFPR while maintaining a high AUC (94.56), proving that mild local perturbation is most beneficial for learning identity-invariant representations.

[0156] Table 6. Ablation of different counterfactual text prompting strategies on FairFD

[0157]

[0158] Furthermore, sensitivity analysis of the counterfactual alignment weights shows that, The optimal balance is achieved across multiple datasets; being too large or too small will compromise fairness or accuracy. For example, on FairFD, As the value increases from 0.6 to 1.2, the AUC gradually improves from 93.68 to 94.56, while the DPD decreases from 0.96 to 0.79; further increases to 1.4 cause fairness to rebound. Therefore, this invention is preferred. As the default setting.

[0159] In summary, this invention, through in-depth analysis of the structural origins of bias within the visual-language model, discovers for the first time that the layer normalization layer is key to amplifying demographic bias, and proposes the IsoCLIP framework based on this. This framework, by fine-tuning only a small number of layer normalization affine parameters (accounting for 0.03% of the total parameters) and combining them with counterfactual isometric alignment loss, can reshape the feature geometry from identity-dominated to authenticity-dominated. Extensive experiments demonstrate that IsoCLIP achieves state-of-the-art detection accuracy and group fairness on five mainstream deepfake datasets, significantly outperforming existing dedicated detectors and fairness enhancement methods. This invention provides a simple yet effective approach to bias reduction, laying a new foundation for building fair and reliable multimodal forgery detection systems.

[0160] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0161] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0163] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0165] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A deepfake detection method based on a visual language model, characterized in that, Includes the following steps: Obtain a pre-trained visual-language model, wherein the visual-language model includes a visual encoder and a text encoder; For an input image, construct its original text prompt and at least one counterfactual text prompt, which is generated by single-hop perturbation of the original text prompt on the sensitive attribute dimension; The text encoder is kept frozen, and text features of the original text prompt and counterfactual text prompt are extracted; The visual encoder extracts the visual features of the input image and calculates the similarity between them and each text feature. Based on the difference in similarity between the visual features and the original text prompts and counterfactual text prompts, the affine parameters of the normalization layer in the visual encoder are adjusted to obtain the adjusted visual encoder. The image to be detected is input into the adjusted visual encoder to extract visual features, and the depth forgery detection result is output based on the extracted visual features.

2. The deepfake detection method based on a visual language model as described in claim 1, characterized in that, The methods for constructing the original text prompt and the counterfactual text prompt include: For the input image, based on its real label and corresponding original sensitive attributes, generate original real prompts and original fake prompts containing sensitive attribute descriptions, as the original text prompts; Based on the original sensitive attributes, at least one new set of sensitive attributes is obtained by performing single-hop perturbation on the sensitive attribute dimension. The single-hop perturbation is to change the value of only one sensitive attribute dimension in a single operation. Based on the new sensitive attributes and the real tags, counterfactual truth prompts and counterfactual false prompts are generated as counterfactual text prompts; The original sensitive attributes include at least a first sensitive attribute and a second sensitive attribute, and the sensitive attribute dimensions include at least a first sensitive attribute dimension and a second sensitive attribute dimension.

3. The deepfake detection method based on a visual language model as described in claim 2, characterized in that, Methods for extracting text features from original text prompts and counterfactual text prompts include: The original real prompts and the original fake prompts are respectively input into the frozen text encoder to obtain the original real text features and the original fake text features; The counterfactual and counterfactual false prompts are input into the frozen text encoder to obtain counterfactual text features and counterfactual false text features.

4. The deepfake detection method based on a visual language model as described in claim 3, characterized in that, The method for calculating its similarity to each text feature includes: Calculate the first similarity between the visual feature and the original real text feature, the second similarity between the visual feature and the original fake text feature, the third similarity between the visual feature and the counterfactual real text feature, and the fourth similarity between the visual feature and the counterfactual fake text feature; Each similarity is obtained by multiplying the inner product of the visual feature and the corresponding text feature by a learnable scaling parameter.

5. The deepfake detection method based on a visual language model as described in claim 4, characterized in that, The method for adjusting the affine parameters of the normalization layer in the visual encoder based on the difference in similarity between the visual features and the original text prompt and the counterfactual text prompt includes: The original similarity difference is calculated based on the first similarity and the second similarity, wherein the original similarity difference is the difference between the second similarity and the first similarity: in, For the original similarity difference, This represents the first similarity between visual features and the original, real-text features. The second similarity between visual features and features of the original fake text; The counterfactual similarity difference is calculated based on the third and fourth similarities, and the counterfactual similarity difference is the difference between the fourth and third similarities: in, To counterfactual similarity differences, The third similarity score represents the visual features and the counterfactual text features. The fourth similarity score represents the visual features and the features of counterfactual and false text. Construct a counterfactual isometric alignment loss based on the mean squared error between the original similarity difference and the counterfactual similarity difference: in, For counterfactual isometric alignment loss, For batch size, For sample index within a batch, This represents the total number of samples within the batch. The total loss function is constructed by combining the classification loss and the counterfactual isometric alignment loss: in, For the total loss function, For classifying losses, To balance the weighting coefficients of the counterfactual equidistant alignment constraint strength; Based on the total loss function, the affine parameters of the normalized layer in the visual encoder are updated using the gradient descent method.

6. The deepfake detection method based on a visual language model as described in claim 5, characterized in that, The layer normalization layer operation is as follows: in, Indicates the first Layer normalization operation; For the first Layer input; For input The mean; For input Standard deviation; This indicates element-wise multiplication; and The first The layer can learn scaling and translation parameters; During the update process, all parameters of the visual encoder, except for the layer normalization layer affine parameter, and all parameters of the text encoder are kept frozen.

7. The deepfake detection method based on a visual language model as described in claim 1, characterized in that, The method of inputting the image to be detected into the adjusted visual encoder, extracting visual features, and outputting depth forgery detection results based on the extracted visual features includes: The image to be detected is input into the adjusted visual encoder to obtain the adjusted visual features; The adjusted visual features are input into the linear classification head to obtain the forgery probability prediction result; The detection method determines whether the image to be detected is a deepfake image based on the forgery probability prediction result.

8. The deepfake detection method based on a visual language model as described in claim 2, characterized in that, The sensitive attribute dimension includes a first sensitive attribute dimension and a second sensitive attribute dimension. The single-hop perturbation is to randomly select one dimension from the first sensitive attribute dimension and the second sensitive attribute dimension and change it, and only the attribute value of one dimension is changed each time.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing the deepfake detection method based on any one of claims 1-8, the processor being configured to execute the programs stored in the memory.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the deepfake detection method based on the visual language model as described in any one of claims 1-8.

Images