Method and device for carrying out deep forgery detection by utilizing visual language model and medium

Abstract

The invention relates to a method and equipment for carrying out deep forgery detection by utilizing a visual language model and a medium. The method comprises the following steps of: constructing a text prompt set containing a plurality of forgery related text prompts; processing an input image by using a visual encoder to obtain an original visual mark sequence; performing uncertainty sampling on the original visual mark sequence to obtain a plurality of uncertainty perceived visual mark sequences; based on the text prompt set, calculating the similarity between the visual marks in the visual mark sequences and the text prompts in the text prompt set; selecting at least one key visual mark from the visual mark sequence according to the similarity; replacing a global semantic mark of a specific layer in the visual encoder with the selected key visual mark to obtain enhanced visual representation; and outputting a counterfeit detection result of the input image based on the enhanced visual representation. According to the method, pre-training knowledge is reserved, meanwhile, the perception ability of the model to a forged area is enhanced, and the detection accuracy and robustness are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of digital image forensics and multimedia security technology, specifically to a method, device, and medium for deepfake detection using a visual language model. Background Technology

[0002] Deepfake technology utilizes artificial intelligence methods such as generative adversarial networks and diffusion models to alter faces in images or videos, including face replacement and facial reenactment. With the rapid development of generative models, the realism of deepfake content continues to improve, even reaching levels indistinguishable to the human eye, posing a serious threat to public safety, personal privacy, and information credibility. Therefore, efficient and accurate deepfake detection technology has become a research hotspot in the fields of digital image forensics and multimedia security.

[0003] Currently, deepfake detection methods based on Vision-Language Models (VLMs) have attracted widespread attention. Vision-Language Models (such as CLIP) learn cross-modal alignment capabilities through pre-training on large-scale image-text pairs, enabling them to simultaneously understand visual content and semantic descriptions. Existing deepfake detection techniques based on Vision-Language Models are mainly divided into two categories: one is adapter fine-tuning methods, which enhance the sensitivity to tampering traces by attaching specific task modules to the visual backbone network. For example, StA improves time modeling through a spatiotemporal adapter, Forda introduces a forensic perception adapter to highlight tampered regions, and EFFORT adjusts CLIP in orthogonal subspaces to retain pre-trained knowledge. The other category is cue learning methods, which optimize text cues to allow the model to adjust its decision boundaries. For example, RepDFD uses learnable visual perturbations and sample-adaptive text cues, CLIPping adjusts compact context tokens, and VLFFD synthesizes fine-grained sentence-level cues through a text-image generator.

[0004] However, the aforementioned existing technologies still have the following shortcomings: First, most methods treat the visual-language model merely as a pure visual feature extractor, utilizing only its visual encoding capabilities while ignoring the potential of language modalities in revealing forgery cues; Second, while adapter fine-tuning methods enhance visual feature modeling, their use of textual information is relatively indirect, failing to actively utilize textual semantics to guide the localization of forgery regions; Third, while cue learning methods optimize classification boundaries, they do not delve into the fine-grained alignment relationship between textual cues and visual labels, making it impossible to directly locate specific regions perceived as "real" or "fake" by the visual model; Fourth, with the development of generative models, traditional visible artifacts are gradually disappearing, and existing methods lack sensitivity to subtle forgery cues, making it difficult to cope with high-fidelity and novel tampering techniques. Summary of the Invention

[0005] This invention provides a method, device, and medium for deepfake detection using a visual-language model. Its purpose is to solve the technical problem that existing technologies only use visual-language models as pure visual feature extractors, which cannot actively use text semantics to reveal forged areas, resulting in insufficient sensitivity to subtle forgery clues.

[0006] To achieve the above objectives, the first aspect of the present invention provides a method for deepfake detection using a visual language model, comprising the following steps: Construct a text cue set containing multiple forged related text prompts; The input image is processed using a visual encoder to obtain the original visual label sequence corresponding to the input image; Uncertainty sampling is performed on the original visual marker sequence to obtain multiple visual marker sequences with uncertainty perception; Based on the text cue set, calculate the similarity between the visual markers in each uncertainty perception visual marker sequence and each text cue in the text cue set; Based on the similarity, at least one key visual marker is selected from the plurality of uncertain perception visual marker sequences; By replacing the global semantic markers of a specific layer in the visual encoder with at least one selected key visual marker, an enhanced visual representation is obtained; Based on the enhanced visual representation, the forgery detection result of the input image is output through a classifier.

[0007] Furthermore, the text prompt set includes fixed text prompts and learnable text prompts. The fixed text prompts are manually designed templates, and the learnable text prompts are a mixed sequence of learnable context vectors and fixed context vectors.

[0008] Furthermore, the method for performing uncertainty sampling on the original visual marker sequence to obtain multiple uncertainty-perceived visual marker sequences includes: The original visual label sequence is input into a Bayesian adapter, which is implemented by a multilayer perceptron block; Multiple Monte Carlo Dropout random forward propagations are performed through the Bayesian adapter to approximate Bayesian inference and model prediction uncertainty; Multiple visual label sequences with uncertainty perception are generated as sampling results.

[0009] Furthermore, when calculating the similarity between visual markers in each uncertainty-perceived visual marker sequence and each text prompt in the text prompt set, a first similarity between each visual marker in the visual marker sequence and a learnable prompt in the text prompt set, and a second similarity between each visual marker in the visual marker sequence and a fixed prompt in the text prompt set are calculated respectively; the first similarity and the second similarity are fused according to a preset weight, and the resulting aggregated similarity is used as the similarity between the visual marker and the text prompt set.

[0010] Furthermore, the method for selecting at least one key visual marker from the plurality of uncertain-perceived visual marker sequences based on the similarity includes: By integrating the aggregated similarity of each visual marker in the visual marker sequence of various uncertainty perceptions, the aggregated similarity score corresponding to each visual marker in the visual marker sequence is obtained; Based on the aggregated similarity score of each visual marker in the visual marker sequence, all visual markers in the visual marker sequence are sorted. The top K visual tags with the highest aggregation similarity scores are selected as key visual tags, where K is a preset positive integer.

[0011] Furthermore, the method of replacing the global semantic markers of a specific layer in the visual encoder with at least one selected key visual marker to obtain an enhanced visual representation includes: Based on the preset injection depth, determine the range of Transformer blocks that need to be replaced; For each Transformer block within the range, the global semantic markers in its input sequence are replaced with selected key visual markers; For each Transformer block outside the specified range, the global semantic tag output by the previous block is passed, and no replacement operation is performed.

[0012] Furthermore, the method also includes a training process in which the model parameters are optimized using a total loss function, which includes classification loss, alignment loss during the fake localization stage, and verification loss during the fake verification stage. The method for calculating the alignment loss in the forgery localization stage includes: calculating a first similarity between the global semantic markers output by the visual encoder and the text cue set; calculating a second similarity between the refined local features obtained by aggregating the key visual markers and the text cue set; and using the sum of the first similarity and the second similarity as the alignment loss. The method for calculating the verification loss in the forgery verification stage includes: inputting the enhanced visual representation into the second adapter to generate a text-aligned similarity score; and calculating the cross-entropy loss based on the similarity score as the verification loss.

[0013] Furthermore, the method for calculating the refined local features includes: performing a weighted average aggregation of all visual tags in the multiple uncertain perception visual tag sequences based on the aggregated similarity score of each visual tag to obtain the refined local features; the second adapter and the Bayesian adapter share the same multilayer perceptron architecture.

[0014] To achieve the above objectives, a second aspect of the present invention provides an electronic device including a memory and a processor, the memory being used to store a program that supports the processor in executing the method for deepfake detection using a visual language model, the processor being configured to execute the program stored in the memory.

[0015] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method for deepfake detection using a visual language model.

[0016] The beneficial effects of this invention are: Compared with existing technologies, this invention provides a method, device, and medium for deepfake detection using a visual language model. It constructs a text cue set containing diverse forgery-related text cues, extracts the original visual marker sequence from the input image using a visual encoder, and performs uncertainty sampling on this original visual marker sequence to obtain multiple uncertainty-perceived visual marker sequences. This simulates the model's prediction distribution under different random conditions to enhance robustness against subtle forgery clues. Furthermore, it calculates the similarity between each uncertainty-perceived visual marker and each text cue in the text cue set, and fuses the semantic information of learnable and fixed cues according to preset weights. It can accurately locate the regions perceived as "most fake" or "most real" in the visual model; further, it selects the top K most relevant key visual labels based on the aggregated similarity score and directly replaces the global semantic labels of a specific Transformer layer in the visual encoder with them, forcing the model to reinterpret the entire image under the semantic perception of forgery. This transforms the language modality from a passive aligner to an active semantic inferencer, effectively amplifying subtle forgery clues that pure visual models usually ignore. Ultimately, it solves the technical problem that existing technologies only use the visual-language model as a pure visual feature extractor and cannot actively use textual semantics to reveal forgery regions, resulting in insufficient sensitivity to subtle forgery clues. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0018] Figure 1 This is a flowchart of a method for deepfake detection using a visual language model, as disclosed in an embodiment of the present invention.

[0019] Figure 2 This is a comparison chart of Deepfake detection paradigms based on VLM disclosed in an embodiment of the present invention, wherein... Figure 2 (a) represents the traditional adapter fine-tuning method; Figure 2 (b) is a hint for fine-tuning methods; while Figure 2 (c) illustrates the TextProbe method proposed in this invention.

[0020] Figure 3 This is an illustration of the effect of key token injection in text guidance disclosed in an embodiment of the present invention, wherein... Figure 3 (a) is the native CLIP model; Figure 3 (b) is the training-free injection method of the present invention; Figure 3 (c) represents the frame-level AUC comparison.

[0021] Figure 4 This is a framework diagram of a deepfake detection system using a visual language model disclosed in an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] According to embodiments of the present invention, it should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the following methods, in some cases the steps shown or described may be executed in a different order than that shown here.

[0024] like Figure 1 As shown, this invention provides a method for deepfake detection using a visual language model, the method comprising: First, refer to Figure 4The diagram shows an overview of the framework proposed in this invention. This framework consists of two collaborative phases: a forgery localization phase (FLS) and a forgery verification phase (FVS). In FLS, the model locates visual tokens related to forgery through diverse textual cues and uncertainty-aware token mining. In FVS, the located forged tokens are injected into the visual encoder, and the forgery is finally verified through an adapter. The framework is described in detail below with specific steps.

[0025] Step S100: Construct a text prompt set containing multiple forged related text prompts; To fully leverage the cross-modal alignment capabilities learned during the pre-training phase of the visual-language model, this step designs the text cue set into two parallel subsets: fixed text cue and learnable text cue. This provides the model with multi-perspective text descriptions, promotes robust semantic alignment, and reduces the bias that may result from a single cue representation.

[0026] Specifically, fixed text prompt set Composed of fixed templates designed by humans, such as "a [ The photo, in which [ The symbol [] represents a category placeholder, corresponding to either "real" or "fake". This type of hint does not participate in parameter updates during training and is used to provide stable, prior semantic guidance.

[0027] Learnable text prompt set It is then constructed as an optimizable continuous vector sequence, with the following form:

[0028] in, Indicates the first Categories A sequence of learnable hint vectors; yes Learnable context vectors, which are optimized during training via gradient descent, to adaptively capture semantic features related to forgery; yes A fixed context vector is used to preserve general language knowledge in the pre-trained model; An embedding vector representing a specific category is used to indicate the category identity (real or fake) corresponding to the current prompt. Through this hybrid sequence design, the learnable prompt has both the flexibility for fake detection tasks and maintains consistency with pre-trained knowledge.

[0029] After constructing the two types of prompts mentioned above, the frozen CLIP text encoder is used. (in Encoding the cue set using fixed pre-trained weights yields the corresponding text embedding features:

[0030] in, Text embedding features; and These represent the text feature vectors obtained after encoding the fixed cue set and the learnable cue set, respectively.

[0031] like Figure 2 As shown, this invention differs significantly from existing VLM-based Deepfake detection paradigms. Figure 2 (a) is a traditional adapter fine-tuning method that attaches specific task modules to the visual backbone network; Figure 2 (b) is a cue fine-tuning method that optimizes learnable text cues for classification; while Figure 2 (c) illustrates the TextProbe method proposed in this invention, which performs training-free, text-guided true / false token detection by measuring the similarity between image patches and text prompts, thereby enabling direct location of forgery-related regions.

[0032] Step S200: Process the input image using a visual encoder to obtain the original visual tag sequence corresponding to the input image; Preprocess the input image. Given an RGB face image... ,in and These represent the height and width of the image, respectively (preferred in this embodiment). First, the image is resized to a uniform size (pixels). Then, the image is divided into non-overlapping regular image blocks, each preferably having a specific size. Pixels. The total number of image patches. The size is determined by both the image size and the block size, and the calculation formula is as follows:

[0033] In this embodiment, due to ,therefore That is, each image is divided into 256 image blocks.

[0034] Each image patch is then flattened into a one-dimensional vector and mapped to a fixed-dimensional embedding vector through a linear projection layer (i.e., the embedding layer in the visual encoder), thus forming the initial sequence of visual tags:

[0035] in, This is the initial visual marker sequence; Indicates the first The initial embedding vector corresponding to each flattened image patch.

[0036] The above initial visual marker sequence Input to the frozen visual encoder Deep feature extraction is performed within the visual encoder. The preferred approach is to use the pre-trained visual Transformer (ViT-L / 14) architecture from the CLIP model, whose parameters are kept fixed (i.e., frozen) during training to preserve the general visual representation capabilities learned in the pre-training phase. The visual encoder processes the input sequence layer by layer through multiple Transformer blocks, outputting a visually labeled sequence with deep semantic modeling.

[0037] in, This represents the final visual tag sequence output by the visual encoder, with the same length as the input sequence (i.e., containing 256 visual tags). Each visual tag corresponds to a deep semantic feature representation of a local region in the original image. This sequence contains rich visual information extracted from local image patches and will serve as the basis for uncertainty sampling and cross-modal alignment in subsequent steps.

[0038] It should be noted that the "visual token" mentioned in this step is the same as the "visual token" or "image patch embedding", which is the basic unit for the visual Transformer to process images; while "freezing" means that the pre-trained weights of the encoder will no longer be updated in subsequent training, and will only be used as a feature extractor, so as to avoid destroying the general knowledge of the pre-trained model while introducing forgery detection capabilities.

[0039] Step S300: Perform uncertainty sampling on the original visual marker sequence to obtain multiple uncertainty-perceived visual marker sequences; Since tampering traces in deepfake detection tasks are often subtle and irregularly distributed, a single-view visual representation is insufficient to comprehensively capture all potential forgery evidence. Therefore, this step employs an uncertainty sampling method based on Bayesian approximation. By applying random perturbations to the visual marker sequence and sampling multiple times, multiple copies of the visual representation reflecting the model's perception under different random states are generated. This simulates the model's prediction distribution under various uncertainty conditions, enhancing robustness to subtle and variable forgery clues.

[0040] In specific implementation, the visual marker sequence obtained in step S200 will be... Input to a specially designed Bayesian adapter The Bayesian adapter is implemented using Multilayer Perceptron (MLP) blocks, each containing two linear layers and an activation function between them, with a Dropout operation introduced after each linear layer. Dropout is a regularization technique that randomly discards a portion of the neuron output during training, and in this step it is used to approximate Monte Carlo sampling in Bayesian inference.

[0041] Bayesian adapter For the input visual label sequence Multiple random forward propagations are performed. During each forward propagation, the Dropout layer randomly discards a portion of neurons (i.e., randomly sets some outputs to zero), thus generating slightly different output results. By performing multiple forward propagations with random dropouts, the model's posterior distribution is approximated, thereby modeling the prediction uncertainty.

[0042] Specifically, the number of Monte Carlo sampling times is set to... The probability of Dropout is In this embodiment, the preferred option is... , For each forward propagation Bayesian adapter Processing input sequences under the condition of randomly discarding some neurons Generate a visual label sequence with uncertainty perception. .go through After the first forward propagation, a set of visual marker sequences for uncertainty perception is obtained:

[0043] Each of them Both are related to input Visual label sequences with the same dimension, but due to the effect of random discarding, they differ numerically, reflecting the diverse perceptual results of the model on the same input image under different random perturbations.

[0044] It's important to note that "uncertainty perception" refers to the fact that these sequences not only contain visual content information from the images themselves but also incorporate implicit estimates of the model's confidence in its own predictions. High-response regions that consistently appear across different samples often correspond to more reliable forgery cues, while regions with larger fluctuations may represent uncertain or easily confused background information. By generating multiple uncertainty-perceived visual tag sequences, this step provides a richer and more robust visual evidence foundation for subsequent cross-modal similarity calculations, enabling the model to comprehensively evaluate the association between each visual region and the forgery semantics from multiple random perspectives.

[0045] Step S400: Based on the text cue set, calculate the similarity between the visual markers in each uncertainty perception visual marker sequence and each text cue in the text cue set; This process makes full use of the image-text alignment capability learned during the pre-training stage of the visual-language model, aligning the visual feature space with the text semantic space, and achieving the goal of examining visual content from a linguistic perspective.

[0046] The input data involved in the calculation needs to be clearly defined. The visual input is a set of visually labeled sequences with uncertainty perception obtained in step S300. ,in For the number of Monte Carlo samplings (preferably 3 in this embodiment), each sequence Include One visual marker (in this embodiment) Each visual marker is denoted as The text input is obtained by freezing the text encoder in step S100. Two types of extracted text features: learnable cue features and fixed prompt features subscript Indicates semantic category.

[0047] For the An uncertainty perception sequence The first in A visual marker First, the first similarity between the feature and the learnable cue features is calculated. The similarity metric uses cosine similarity, calculated using the following formula:

[0048] in, The first similarity; Represents the vector dot product. and These represent the magnitudes of the visual markers and the text feature vectors, respectively. The range of values ​​is A higher value indicates a closer semantic similarity between the visual tag and the corresponding category of text. Similarly, a second similarity is calculated between the same visual tag and fixed cue features:

[0049] in, This represents the second similarity.

[0050] To comprehensively utilize the adaptive capabilities of learnable cues and the prior knowledge of fixed cues, this step weights and fuses the two similarity methods mentioned above to obtain a category-specific similarity for each visual tag. Aggregate similarity:

[0051] in, To aggregate similarity, It is a balancing hyperparameter used to adjust the contribution ratio of learnable cues and fixed cues in the similarity metric. Based on the experimental validation in the appendix, when... The model achieves state-of-the-art performance on multiple datasets, suggesting that learnable cues should dominate the aggregation process, while fixed cues serve as beneficial supplementary priors.

[0052] It should be noted that the above calculation process requires processing each uncertainty-aware sequence. Each visual marker in The process was performed separately, and aggregated similarity was calculated for both the real and fake categories. and Since subsequent steps require filtering the visual markers most relevant to forgery, this step ultimately outputs the aggregated similarity score for each visual marker based on its forgery category. (For the sake of simplicity, it will be referred to as in the following text) (This is used to measure how well the visual region matches the semantics of "forgery").

[0053] Through the above calculations, this step projects the visual markers from the pure visual feature space to the "real-forged" metric space defined by textual semantics, so that each local image region is assigned a quantifiable forgery confidence score.

[0054] Step S500: Based on the similarity, select at least one key visual marker from the plurality of uncertain perception visual marker sequences; This step aims to select the local visual regions that best represent forgery clues based on the aggregated similarity score between the visual tags calculated in step S400 and the semantics of "forgery," providing accurate semantic anchors for subsequent injection into the visual encoder. This selection process follows a "similarity-oriented" principle, retaining the visual tags with the highest alignment to the forgery semantics through sorting and truncation operations, while filtering out redundant or noisy information unrelated to forgery.

[0055] In practice, the first step is to integrate the similarity information from all uncertain sampling sequences. Step S300 generates... For each visual marker sequence with uncertainty perception, step S400 calculates an aggregated similarity score for the forgery category for each visual marker in each sequence. Therefore, the total is... There are several similarity scores, among which... The number of visual markers included in each sequence (in this embodiment) ).

[0056] Subsequently, for all The aggregated similarity scores of each visual marker are uniformly sorted. The sorting process proceeds in descending order of score; a higher score indicates a stronger alignment between the visual marker and the semantic meaning of "forgery," meaning that the local area is more likely to contain forgery clues. After sorting, a preset number of selections is used. Select the top from the sorted results One visual marker is used as a key visual marker. These visual markers constitute a refined set of fake perceptual markers, represented as:

[0057] in, For a refined set of fake perception markers, Indicates the selected number A key visual marker, This is a preset positive integer used to control the number of fake related visual markers selected.

[0058] It should be noted that the above selection process has two important characteristics: First, "unified ranking across samples," meaning that visual markers in all uncertain sampling sequences are placed in the same ranking space to compete, making it easier to select stable forgery cues that maintain high similarity under different random perturbations; second, "uncertainty perception," because the input sequence itself has already introduced diversity through Monte Carlo Dropout, the key visual markers finally selected essentially integrate the consensus of the model under multiple uncertainty conditions, thus being more robust and reliable than selection based on a single deterministic sequence.

[0059] Through this step, the original visual marker sequence is obtained. Condensed into a single containing only A compact set of key visual markers These markers focus on the most suspicious local areas in the image, providing visual evidence for semantic injection.

[0060] Step S600: Replace the global semantic marker of a specific layer in the visual encoder with at least one selected key visual marker to obtain an enhanced visual representation; The key visual markers highly relevant to forgery semantics selected in step S500 are integrated back into the forward propagation process of the visual encoder through a "forgery semantic injection" (FSI) operation, thereby guiding the model to reinterpret the entire image under the guidance of forgery-aware semantics. This replacement operation involves using local evidence focused on the forged region to cover the general semantic anchors originally used to aggregate global information, making subsequent visual processing more focused on tampering traces.

[0061] In practice, the first step is to refine the set of fake perception tags selected in step S500. Preprocessing is performed to integrate them into a single, compact forged perception representation. Although step S500 selects... Each layer contains an independent visual tag, but the global semantic tag (often called [CLS] tag) processed by the visual encoder at each layer is a fixed-dimensional vector. Therefore, it is necessary to transform this... The information from each tag is fused together. In this embodiment, a weighted average aggregation method is used, utilizing the aggregated similarity score of each key visual tag calculated in step S400. As weights, a weighted average is calculated to obtain refined fake perception local features:

[0062] in, For refined fake perception of local features, This represents the mean aggregation function weighted by similarity scores. It is a compact feature vector that integrates information from multiple key regions and represents the essence of image forgery.

[0063] Next, perform the fake semantic injection operation. Visual encoder It typically consists of multiple sequentially connected Transformer blocks, with a total depth of . (i.e., includes) (One Transformer block). This step introduces a hyperparameter (injection depth). ), used to specify from which layer to start using Replace the existing global semantic tags.

[0064] make Indicates the first Transformer blocks Indicates entering the first The sequence of image block markers preceding each block This represents the global semantic tags output by the previous block. For the previous... Transformer blocks (i.e.) Perform the following replacement injection operation:

[0065] in, Represents the first in the visual encoder (ViT) One Transformer block; Indicates the injection depth; For the updated global semantic tagging; Label the updated image patch sequence.

[0066] That is, in these layers, the global semantic tags that should have been passed down from the previous layer are completely replaced with fake perceptual features obtained from the FLS stage. and marked with the image blocks of the previous layer. After being concatenated, they are input together into the current Transformer block for processing. This operation ensures that the model consistently uses "fake" as the core semantic anchor to organize visual information throughout the processing of these layers.

[0067] For the remaining Transformer blocks (i.e.) If the substitution occurs, the normal propagation mechanism is restored, and instead of replacement, the global markers output by the previous layer, which already incorporate the forged semantics, continue to propagate forward.

[0068] here, It was after the previous The resulting global semantic tags, obtained after layer injection, carry the forged perceptual information. In this way, the forged semantics are naturally propagated to deeper layers of the network and participate in subsequent global context modeling.

[0069] It should be noted that the "global semantic tag" involved in the above operations refers to a special tag (i.e., [CLS] tag) in the visual Transformer architecture specifically used to aggregate global information. It interacts with other image patch tags at each layer through a self-attention mechanism, ultimately representing the global features of the entire image. This step uses fake perceptual features... Replacing it essentially forces the model to implant the task prior of "forgery detection" at the core of information aggregation, so that all subsequent self-attention calculations revolve around the core question of "where is the most fake".

[0070] After the above replacement injection operation, the visual encoder finally outputs the global semantic tag. It is no longer the general image representation in the original CLIP model, but an enhanced visual representation that has been semantically reinforced and is highly sensitive to tampering traces, denoted as... .

[0071] To verify the validity of the key token injection in the text-guided presentation, Figure 3 The results of the comparative experiment are presented. Figure 3 (a) is the native CLIP model, whose visual backbone network is frozen and only a single linear classification head is trained; Figure 3 (b) is the training-free injection method of the present invention, which injects the Top-K real / fake tokens into the CLIP global representation, and also trains only one linear classifier for alignment; Figure 3 (c) shows the frame-level AUC comparison, demonstrating a significant performance improvement across multiple datasets after key token injection. This indicates that language-guided token injection effectively enhances the model's sensitivity to forged cues.

[0072] Step S700: Based on the enhanced visual representation, output the forgery detection result of the input image through a classifier.

[0073] This step aims to utilize the enhanced visual representation obtained in step S600 for the final deep forgery detection. This representation has been infused with forgery-related local cues through forgery semantic injection (FSI), enabling the classifier to perceive subtle tampering traces at the global semantic level, thereby outputting an accurate forgery confidence score.

[0074] In specific implementation, the enhanced visual representation output in step S600 will be... The input is processed in the adapter module of the Forgery Validation (FVS) phase. This phase contains two parallel processing branches: one branch performs the final forgery classification, and the other branch further strengthens the consistency verification between the injected visual representation and the text semantics.

[0075] In the fake category branch, set up a lightweight adapter. Further feature transformations are performed on the enhanced visual representation. Lightweight adapter. Implemented using a multilayer perceptron (MLP), it consists of two linear layers and an activation function between them. Lightweight adapter. The input is the complete output sequence after processing through all Transformer layers in step S600, including the final global semantic tags. and all image block tags , represented as The adapter processing procedure is shown in the following formula:

[0076] in, It is the visual characteristic output by the adapter (including the global part) and local parts ), This is the predicted probability vector output by the adapter through its classification head, representing the confidence level that the input image belongs to either the "real" or "fake" category. (Classifier output) loss by standard classification To supervise, among which This represents the true label of the input image during the training phase (0 indicates real, 1 indicates fake). The cross-entropy loss function is used.

[0077] In the text consistency verification branch, set up another adapter. To enhance the alignment between the injected visual representation and the textual semantics. Adapter With the Bayesian adapter used in step S300 They share the same multilayer perceptron architecture, but their parameters are independent. Given an adapter... Output visual features ,adapter Perform a text-visual matching operation similar to step S400 to generate a similarity score aligned with the text cue set.

[0078] Specifically, computational visual features Text features obtained in step S100 The cosine similarity between them yields the alignment score matrix. Subsequently, the loss during the forgery verification phase was calculated based on this score:

[0079] in, To fabricate losses during the verification phase, Represents the cross-entropy function. For real labels, The similarity scores are converted into a category probability distribution. This loss ensures that the visual representation injected with forged semantics still aligns with the correct textual semantics, thereby further enhancing the discriminative power of the representation.

[0080] Finally, during the training phase, the two branches mentioned above, together with steps S100-S600, constitute the end-to-end optimization objective. The total loss function is defined as:

[0081] in, To fabricate alignment loss during the localization phase, and These are preset weight hyperparameters used to balance the contributions of classification loss, localization alignment loss, and validation consistency loss in joint optimization.

[0082] During the inference (testing) phase, only the output of the fake classification branch needs to be used. As the final result of the falsified test. Specifically, take The probability value corresponding to the "forgery" category is used as the forgery confidence score of the input image. .like If the value exceeds a preset threshold, the image is determined to be a fake image; otherwise, it is determined to be a real image.

[0083] It should be noted that the "adapter" involved in this step is a lightweight, learnable module. By adding a small number of parameters to the frozen pre-trained model for fine-tuning, it can adapt to downstream tasks while avoiding catastrophic forgetting and overfitting. The "fake consistency verification" branch, although only used to calculate the loss during the training phase... However, its existence is crucial for guiding enhanced visual representations to maintain alignment with textual semantics.

[0084] Through the processing of step S700 above, the present invention finally realizes the deep forgery detection of the input image and outputs an interpretable forgery confidence score, providing a reliable decision basis for digital image forensics.

[0085] This invention belongs to the field of digital image processing and computer vision technology, specifically relating to a deepfake detection method based on a vision-language model. It can be used in scenarios such as digital image forensics, multimedia content authenticity verification, and video surveillance security. Addressing the technical problems of existing deepfake detection methods, such as insufficient sensitivity to subtle tampering traces and weak generalization ability across datasets, this invention introduces a text-semantic guided forgery clue localization and injection mechanism. This transforms the vision-language model from a passive feature extractor into an active semantic inferencer, thereby enhancing the model's ability to perceive forged regions while retaining pre-trained knowledge, significantly improving detection accuracy and robustness. This invention can be widely applied to scenarios requiring automatic verification of the authenticity of images or videos.

[0086] For example, in a content moderation system, a user uploads a video of a person giving a speech. The platform uses this method to process each frame of the video: first, the video is decoded into image frames, the facial region of each frame is cropped, and the data is input into the visual-language model constructed in this invention; the model automatically identifies visual regions highly aligned with the semantics of "forgery" (such as facial boundaries, areas of abnormal lighting) during the forgery localization stage, and injects the features of these regions into the global representation; finally, the classifier outputs the forgery confidence score of the frame. If the confidence scores of multiple consecutive frames exceed a preset threshold, the system determines that the video is a deepfake, automatically marks it, and initiates a manual review process, thereby effectively preventing the spread of fake videos. This example demonstrates that this invention can be directly embedded into existing content moderation pipelines to achieve efficient and accurate automated forgery detection.

[0087] To verify the technical effectiveness of the method of this invention, systematic experimental verification was conducted. The experimental setup, dataset, comparison methods, implementation details, main results, and ablation analysis are described in detail below.

[0088] Following common benchmarks in deepfake detection, this method is trained only on the FaceForensics++ (FF++) dataset and tested on multiple cross-datasets to evaluate its generalization ability. The test datasets include: Classic datasets: Celeb-DF-v1 (CDF-v1), Celeb-DF-v2 (CDF-v2), DeepFakeDetectionChallengePreview (DFDCP), DeepFakeDetection (DFD), and DeepFake Detection Challenge (DFDC).

[0089] High-fidelity modern datasets: DF40 and Celeb-DF++. DF40 covers advanced generation techniques such as BleFace, MobSwap, FSGAN, and SimSwap; Celeb-DF++ includes diverse tampering types such as BlendFace, Celeb-DF, GHOST, MobileFaceSwap, FSRT, LivePortrait, TPSMM, and LIA.

[0090] To establish a comprehensive comparative benchmark, the proposed method was evaluated against twelve state-of-the-art Deepfake detectors: SRM, Recce, SBI, UCF, ED, IID, ProDet, LSDA, CFM, CDFA, ForAda, and Effort. All baseline methods were trained on FF++ (c23) and evaluated on the remaining datasets following the DeepfakeBench protocol. Results on classic datasets are summarized in Table 1, while experimental results on state-of-the-art Celeb-DF++ and DF40 benchmarks are reported in Tables 3 and 4, respectively.

[0091] The model was initialized using CLIP ViT-L / 14 pre-trained weights. The input image resolution was 256×256, and the detected face cropped images were resized to 224×224 with a patch size of 14×14. Training was performed using the Adam optimizer with an initial learning rate of 2×10⁻⁶. -4 During training and testing, T=32 frames are sampled uniformly from each video.

[0092] In the UFTM module, the MC-Dropout probability used is: The number of random samples is .choose A forged related token was injected using the "end" pattern to define the dimension. Learnable vectors.

[0093] For the total loss function The loss weights for FLS and FVS are respectively and All experiments were conducted on a single NVIDIA L20 GPU.

[0094] Tables 1 and 2 present the comparison results of frame-level AUC (Area Under the Curve) and video-level AUC on classic datasets, respectively. Our method achieves best or near-best performance on all test sets, with an average frame-level AUC of 0.910, significantly outperforming existing methods. For example, compared to the ForAda method, which is also based on CLIP, our method improves the average frame-level AUC by 1.4%, validating the effectiveness of the forged semantic injection strategy.

[0095] Table 1. Cross-dataset evaluation results (frame-level AUC) (All methods were trained on FF++ and tested on other datasets)

[0096] Table 2. Evaluation results across datasets (video-level AUC) (the upper part shows video-based methods, and the lower part shows frame-based methods)

[0097] Tables 3 and 4 present the video-level AUC results on Celeb-DF++ and DF40, respectively. Celeb-DF++ includes eight tampering types across two main categories: Face-Swap (FS) and Face-Reenactment (FR). Our method achieves the best results on most subclasses, with an average AUC of 0.822, a 1.3% improvement over the second-best method, Effort (0.809). On DF40, our method achieves an average AUC of 0.949 across seven different generation methods, also outperforming all comparison methods and demonstrating strong generalization ability.

[0098] Table 3. Evaluation results across datasets (Celeb-DF++, video-level AUC) (All detectors were trained on FF++-c23)

[0099] Table 4. Evaluation results across datasets (DF40, video-level AUC) (All detectors were trained on FF++-c23)

[0100] To gain a deeper understanding of the contributions of each module, a series of ablation experiments were conducted, covering the key designs of the forgery localization phase (FLS) and the forgery verification phase (FVS).

[0101] Table 5 examines the fusion weights of learnable hints and fixed hints. ( The corresponding weights of learnable cues are fixed, with the cue weight being... The impact on performance. The results show that... The best AUC was achieved on all three test sets, indicating that learnable hints should dominate, but the prior knowledge provided by fixed hints is still helpful.

[0102] Table 5. Ablation experiments on semantic fusion weights for diverse prompts

[0103] Table 6 shows the different Dropout probabilities Impact on generalization ability: Moderate. The optimal level of uncertainty modeling is achieved when the level is too high or too low, as both result in performance degradation. This suggests that moderate uncertainty modeling helps to filter out reliable forgery clues.

[0104] Table 6. Ablation experiments with MC-Dropout rate

[0105] Table 7 compares the different The impact of values ​​on performance. The goal is to achieve the optimal balance; too little information will cause important clues to be missed, while too much information will introduce noise.

[0106] Table 7. Ablation experiments on Top-K selection in forged token injection

[0107] Table 8 compares the two injection strategies, "replacement" and "concatenation". Replacing the original [CLS] token is significantly better than concatenation, indicating that covering global semantic anchors can more effectively guide the model to focus on forged regions.

[0108] Table 8. Ablation experiments of the global token replacement strategy

[0109] Table 9 explores the effects of replacement injection at different Transformer layers. Layer 10 (out of 24) represents the optimal injection depth, balancing local features with global abstraction.

[0110] Table 9. Ablation experiments showing the effect of injection depth

[0111] This invention re-examines the role of language in visual-language models from the perspective of forgery detection. By constructing a two-stage process, subtle forgery cues are first located, and then injected into the model to guide focused inference—effectively guiding the Visual-Language Model (VLM) to focus on tampered regions. In extensive experiments covering classic datasets such as FF++, DFD, DFDC, DFDCP, and CDF-v1 / v2, as well as challenging modern datasets such as DF40 and Celeb-DF++, this method achieves consistent and state-of-the-art performance. Furthermore, the model exhibits good transferability across different tampering styles, generation processes, and backbone architectures, demonstrating that language-guided cue injection is a promising and scalable paradigm for deep forgery detection, contributing to more robust and comprehensive forgery understanding.

[0112] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0113] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0115] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0117] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for deepfake detection using a visual language model, characterized in that, Includes the following steps: Construct a text hint set containing multiple forged related text hints; The input image is processed using a visual encoder to obtain the original visual label sequence corresponding to the input image; Uncertainty sampling is performed on the original visual marker sequence to obtain multiple visual marker sequences with uncertainty perception; Based on the text cue set, calculate the similarity between the visual markers in each uncertainty perception visual marker sequence and each text cue in the text cue set; Based on the similarity, at least one key visual marker is selected from the plurality of uncertain perception visual marker sequences; By replacing the global semantic markers of a specific layer in the visual encoder with at least one selected key visual marker, an enhanced visual representation is obtained; Based on the enhanced visual representation, the forgery detection result of the input image is output through a classifier.

2. The method for deepfake detection using a visual language model as described in claim 1, characterized in that, The text prompt set includes fixed text prompts and learnable text prompts. The fixed text prompts are manually designed templates, and the learnable text prompts are a mixed sequence of learnable context vectors and fixed context vectors.

3. The method for deepfake detection using a visual language model as described in claim 1, characterized in that, A method for performing uncertainty sampling on the original visual marker sequence to obtain multiple uncertainty-perceived visual marker sequences includes: The original visual label sequence is input into a Bayesian adapter, which is implemented by a multilayer perceptron block; Multiple Monte Carlo Dropout random forward propagations are performed through the Bayesian adapter to approximate Bayesian inference and model prediction uncertainty; Multiple visual label sequences with uncertainty perception are generated as sampling results.

4. The method for deepfake detection using a visual language model as described in claim 2, characterized in that, When calculating the similarity between visual markers in each uncertainty-perceived visual marker sequence and each text prompt in the text prompt set, a first similarity between each visual marker in the visual marker sequence and a learnable prompt in the text prompt set, and a second similarity between each visual marker in the visual marker sequence and a fixed prompt in the text prompt set are calculated respectively. The first similarity and the second similarity are fused according to a preset weight, and the resulting aggregated similarity is used as the similarity between the visual tag and the text prompt set.

5. The method for deepfake detection using a visual language model as described in claim 4, characterized in that, The method for selecting at least one key visual marker from the plurality of uncertainty-perceived visual marker sequences based on the similarity includes: By integrating the aggregated similarity of each visual marker in the visual marker sequence of various uncertainty perceptions, an aggregated similarity score corresponding to each visual marker in the visual marker sequence is obtained; Based on the aggregated similarity score of each visual marker in the visual marker sequence, all visual markers in the visual marker sequence are sorted. The top K visual tags with the highest aggregation similarity scores are selected as key visual tags, where K is a preset positive integer.

6. The method for deepfake detection using a visual language model as described in claim 1, characterized in that, Methods for obtaining enhanced visual representations by replacing the global semantic markers of a specific layer in the visual encoder with at least one selected key visual marker include: Based on the preset injection depth, determine the range of Transformer blocks that need to be replaced; For each Transformer block within the range, the global semantic markers in its input sequence are replaced with selected key visual markers; For each Transformer block outside the specified range, the global semantic tag output by the previous block is passed, and no replacement operation is performed.

7. The method for deepfake detection using a visual language model as described in claim 3, characterized in that, The method also includes a training process in which the model parameters are optimized using a total loss function, which includes classification loss, alignment loss during the fake localization stage, and verification loss during the fake verification stage. The method for calculating the alignment loss in the forgery localization stage includes: calculating a first similarity between the global semantic markers output by the visual encoder and the text cue set; calculating a second similarity between the refined local features obtained by aggregating the key visual markers and the text cue set; and using the sum of the first similarity and the second similarity as the alignment loss. The method for calculating the verification loss in the forgery verification stage includes: inputting the enhanced visual representation into the second adapter to generate a text-aligned similarity score; and calculating the cross-entropy loss based on the similarity score as the verification loss.

8. The method for deepfake detection using a visual language model as described in claim 7, characterized in that, The method for calculating the refined local features includes: performing a weighted average aggregation of all visual tags in the multiple uncertain perception visual tag sequences based on the aggregated similarity score of each visual tag to obtain the refined local features; the second adapter and the Bayesian adapter share the same multilayer perceptron architecture.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the method for deepfake detection using a visual language model as described in any one of claims 1-8, the processor being configured to execute the program stored in the memory.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the method for deepfake detection using a visual language model as described in any one of claims 1-8.

Images