Face spoofing detection method based on query-driven forensic adapter

Abstract

This invention presents a face forgery detection method based on a query-driven forensic adapter, belonging to the fields of artificial intelligence and machine learning. It aims to address issues such as insufficient modeling of local forgery regions, limited interaction between semantic and visual features, and poor cross-distribution generalization performance. First, this invention proposes a dynamic semantic query module. This semantic query is injected as a dynamic attention signal into the multi-head attention mechanism of the CLIP encoder, effectively guiding CLIP to focus on potential forgery regions, improving its response to local anomalies such as boundary discontinuities and texture inconsistencies, and compensating for its insufficient local modeling capabilities without altering the CLIP's core structure. Second, this invention proposes a multi-scale feature aggregation mechanism. It proposes a semantically guided attention fusion module and a feature enhancement strategy based on linear interpolation. This method can effectively improve sample diversity and enhance the model's robustness and generalization ability across forgery methods and datasets.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and machine learning, and particularly relates to a face forgery detection method based on deep neural networks, specifically involving the use of a query-driven forensic adapter to detect face forgery images or videos. Background Technology

[0002] With the rapid development of deep generative models (such as Generative Adversarial Networks (GANs) and Diffusion Models), face-based image and video forgery techniques are becoming increasingly realistic in visual effects and are widely disseminated in social media, the entertainment industry, and the political sphere. The proliferation of such forged content not only threatens public cognitive security but also poses serious challenges to judicial evidence collection, news media, and social trust systems. Therefore, researching efficient, accurate, and generalizable face forgery detection methods has become an important topic in artificial intelligence security and multimedia forensics.

[0003] Most existing face forgery detection methods are based on deep neural network architectures, such as convolutional neural networks (CNNs) or Transformer models, modeling the detection problem as a binary classification task. These methods mainly rely on spatial domain artifacts in images (such as inconsistencies in facial structure, texture abnormalities, and unnatural boundary blending) or frequency domain analysis (such as differences in high-frequency noise distribution and artifacts caused by upsampling operations) for detection. However, these methods generally suffer from the following problems: on the one hand, they require large-scale, high-quality labeled data for training, which is costly and difficult to adapt to new forgery techniques; on the other hand, their generalization performance is limited, and they are prone to failure in detection tasks across datasets and forgery methods.

[0004] In recent years, the development of multimodal pre-trained models has brought new directions to forgery detection. Large-scale pre-trained models, represented by CLIP, have shown excellent performance in image-text alignment and general visual feature extraction, and have been introduced into face forgery detection to improve cross-task transfer capabilities. However, CLIP itself was not designed specifically for forgery detection, and its visual encoder has significant shortcomings in modeling local forgery regions: on the one hand, it lacks sensitivity to local details (such as boundary artifacts and local texture anomalies); on the other hand, its attention mechanism focuses more on global consistency rather than the fine-grained anomalous region perception required for forgery detection.

[0005] To overcome this deficiency, some studies have attempted to introduce lightweight adapter structures onto the frozen CLIP visual encoder to enhance its forgery detection capabilities. Existing adapters are mainly divided into two categories: post-processing and parallel processing. Post-processing adapters often directly act on the CLIP output features, only supplementing the final representation level and lacking guidance for the CLIP's internal feature learning process. Parallel adapters, on the other hand, introduce additional semantic tokens to jointly model with the CLIP visual tokens; however, these tokens are usually static and lack adaptive interaction with specific input images, making it difficult to fully capture the diversity of forgery patterns. Both types of methods suffer from insufficient interaction with the CLIP backbone, making it difficult to truly improve its ability to perceive local forgery features, thus limiting overall detection performance and generalization ability.

[0006] In summary, existing face forgery detection methods still suffer from insufficient robustness and generalization when handling cross-distribution data and novel forgery methods. Furthermore, they lack accurate modeling of local anomaly regions and effective activation of the latent semantic capabilities within the pre-trained model. Therefore, a new detection method is urgently needed that combines a query-driven mechanism with an evidence adapter design to achieve deep semantic interaction between the adapter and the CLIP encoder. This would guide the model to focus on forged regions without altering the core structure, thereby effectively improving the accuracy and cross-domain robustness of face forgery detection. Summary of the Invention

[0007] This invention proposes a face forgery detection method based on a query-driven forensic adapter, aiming to address the shortcomings of existing detection methods, such as insufficient modeling of local forgery regions, limited interaction between semantic and visual features, and poor cross-distribution generalization performance. Based on a frozen multimodal pre-trained visual model (such as CLIP), this invention designs a novel deep forgery detection framework, CSEA-Net. By introducing a query-driven mechanism and multi-module collaborative modeling, it achieves accurate perception and robust detection of forgery regions. First, this invention proposes a dynamic semantic query module. This module can adaptively generate semantic queries related to the current task from a specified layer of the adapter and dynamically adjust them in conjunction with the features of the input image, thereby achieving deeper and more flexible semantic interaction between the adapter and CLIP. This semantic query is injected as a dynamic attention signal into the multi-head attention mechanism of the CLIP encoder, effectively guiding CLIP to focus on potential forgery regions, improving its response to local anomalies such as boundary discontinuities and texture inconsistencies, and compensating for its insufficient local modeling capabilities without changing the CLIP backbone structure. Second, this invention proposes a multi-scale feature aggregation mechanism. This mechanism can extract semantic information from multiple intermediate layers of the visual encoder and aggregate them at the same spatial scale. By jointly modeling multi-scale features and semantic queries, the model can simultaneously capture global consistency and fine-grained forgery traces, significantly improving the detection capability of local abnormal regions (such as boundary artifacts and unnatural region fusion). Furthermore, to achieve deep information interaction between the adapter and CLIP, this invention proposes a semantically guided attention fusion module. This module uses global semantic features extracted by CLIP to guide the adapter, making it focus more on key forgery regions, thereby achieving complementary enhancement of visual and semantic features, further improving detection performance and interpretability. Finally, this invention proposes a feature enhancement strategy based on linear interpolation. By generating linear interpolation of intra-class samples in the feature space, this method can effectively improve sample diversity and enhance the model's robustness and generalization ability across forgery methods and datasets. Extensive experimental results show that the proposed method achieves excellent performance on multiple public face forgery datasets. In frame-level AUC detection, it improves DFDC by 4.2%; in video-level AUC detection, it improves CDF-v2 by 1.1%, DFDC by 6.0%, and DFDCP by 4.3%. It effectively improves CLIP's adaptability and generalization ability in forgery detection tasks. Attached Figure Description

[0008] Figure 1 Schematic diagram of a face forgery detection method based on CLIP and semantic enhancement adapter

[0009] Figure 2 Dynamic semantic query module

[0010] Figure 3 Multi-scale feature aggregation module

[0011] Figure 4 Semantic-guided attention fusion module Detailed Implementation

[0012] This invention provides a face forgery detection method based on CLIP and a semantic enhancement adapter, aiming to improve the model's ability to perceive local forged regions, enhance interpretability, and improve generalization performance across forgery methods. This method treats the forgery detection task as a cross-distribution image classification problem, with the overall framework as follows: Figure 1 As shown, the top channel represents the CLIP model, and the bottom channel represents the semantic enhancement adapter. The structural design of the dynamic semantic query module enables the adapter channel to generate semantic representations with semantic diversity and spatial selectivity. The adapter channel uses a multi-scale feature aggregation module to extract multi-scale features and utilizes a semantically guided attention fusion module to achieve semantically guided feature fusion.

[0013] The invention mainly includes the following steps: Step 1, global semantic feature extraction based on Clip pre-trained visual encoder; Step 2, dynamic modeling of fake regions and generation of artifacts based on semantic enhancement adapter; Step 3, artifact decoding and interpretability enhancement; Step 4, dynamic attention signal injection; Step 5, multi-task supervision and classification.

[0014] Step 1: Global semantic feature extraction based on Clip pre-trained visual encoder

[0015] To ensure the robustness and transferability of the global semantic representation, a pre-trained CLIP ViT-L / 14 visual encoder is used as a frozen global semantic stream (i.e., this part does not participate in the update of the backbone weights). Based on this, the input image is uniformly preprocessed to match the CLIP input resolution and the adapter's patch resolution, thereby providing multi-level intermediate features for the subsequent semantic enhancement adapter to facilitate semantic interaction and guidance.

[0016] During the input phase, the original image tensor is denoted as... Where N and C represent the batch size and the number of channels, respectively, and N=16, C=3. The original image resolution is 256×256. The image is scaled to the CLIP-required size of 224×224 using bilinear interpolation. Another input is... This specifies the mapping relationship between the Adapter layer and the CLIP vision backbone Transformer encoder layer, where the key-value pairs 6:10 and 7:15 represent the Adapter layer number (6 and 7) and the layer number of the Transformer encoder in the CLIP vision backbone (layer 10 and layer 15), respectively.

[0017] Next, the CLIP visual backbone is loaded and frozen, and the CLIP feature extraction interface is called to obtain multi-layer visual features organized in dictionary form. This dictionary contains the layers appearing in the fusion_map, as well as the cls token and patch token for layer 0. Layer 0 is the input embedding layer, responsible for image segmentation, linear mapping, and positional encoding, but it does not perform self-attention computation; it is equivalent to the "word embedding layer" in ViT. The other layers are the actual Transformer blocks, containing structures such as self-attention mechanisms and multilayer perceptrons, responsible for extracting semantic features at different levels. The cls token is used to represent the global semantics of the entire image, and its shape is... The shape of the patch token is The number of patches, L, is 256, and the dimension of each patch, D, is 1024. Pass in the Adapter (see step 2) for cross-branch fusion.

[0018] Step 2: Dynamic modeling of fake regions and generation of artifact maps based on semantic enhancement adapter

[0019] Since image forgery is often localized, a single global feature is insufficient to accurately capture potential forgery regions. Therefore, this method introduces a semantic enhancement adapter while freezing the CLIP backbone. This adapter extracts feature representations with dynamic semantic differences, thereby enhancing the capture and interpretation capabilities of local forgery regions while maintaining CLIP's global perception capabilities. This module receives multi-level features from the frozen CLIP encoder output in step one and uses a dynamic semantic query mechanism to explicitly model the forgery regions.

[0020] Structurally, the Adapter is built using the first 8 layers of the ViT-Tiny Transformer, each layer containing Multi-Head Self-Attention (MHSA), LayerNorm normalization, and a feedforward network (MLP). The input image size is 256×256, which is first divided into 16×16 patches, generating 256 patch tokens (L=256). Each patch is linearly projected to obtain an embedding representation, resulting in the initial patch token representation. Where d is 192. Then, position encoding is performed. Let the two-dimensional position encoding of ViT be... Adding the position code to the patch tokens and normalizing them yields:

[0021]

[0022] Update layer by layer on each block of ViT. The dynamic semantic query module is triggered at layers 4 and 8 (e.g., Figure 2 (As shown): First, dynamic semantic query initialization is performed, introducing a learnable parameter matrix. The number of semantic queries selected The value is 128, and the feature dimension d is 192. The query expands during forward propagation to... Where N represents the batch size. Subsequently, the semantic query vector is constructed. Patch features of the current layer Interaction is achieved through a multi-head attention mechanism, where L=256 represents the number of patch tokens in ViT. The calculation formula is:

[0023]

[0024] This process can generate representations with semantic diversity and spatial selectivity. Attention map obtained simultaneously .

[0025] Finally, the dynamic semantics are obtained by enhancing the non-linear expressive power of the query representation through an MLP. :

[0026]

[0027] To address the semantic shift issue that may result from the disconnect between the Adapter module and the backbone features, this invention designs a semantically guided fusion module based on an attention mechanism, using CLIP intermediate layer features. As a guide, features are dynamically fused into the Adapter representation stream to achieve semantic collaboration between the backbone and auxiliary paths. In the ViT backbone network, features are updated layer by layer in each Transformer block. When the current ViT layer is a preset fusion layer, a semantically guided adaptive fusion module (such as...) is triggered. Figure 4 (As shown). In this invention, features from CLIP layers 10 and 15 are fused with Adapter layers 6 and 7, respectively. Features from the Adapter are... Reconstructed into two-dimensional form Where H=W=16. Guiding features The data is reconstructed into a two-dimensional form and projected onto the channel dimension using a 1×1 convolutional layer, mapping it to a feature space consistent with the main features.

[0028]

[0029] Utilizing the guiding features after projection Generate query vector Q and key vector K, and use the main features Generate a value vector V, and compute a spatial semantic weighted representation using a self-attention mechanism:

[0030] Among them, the scaling factor used to balance the inner product scale The attention output is reconstructed into spatial features and fused with the original master features using a residual approach.

[0031]

[0032] in The characteristics after fusion This is a learnable scaling factor, initialized to 0, which automatically learns the optimal value through backpropagation and is used to adjust the fusion intensity.

[0033] Furthermore, to fully explore semantic information at different levels and improve the model's ability to model multi-scale targets, a multi-scale feature aggregation module (such as...) was designed. Figure 3 (As shown). In this invention, the features of each layer are processed sequentially from deep to shallow (8→6→4). Let the features output by layers 4, 6, and 8 of the Adapter be respectively... For each layer of features Channel alignment is achieved through 1×1 convolution. : Then, feature fusion is performed using a top-down approach. The deepest layer ( =8) Directly used as the initial aggregation feature:

[0034]

[0035] For other layers (i=6,4), the features from the previous layer and the features from the current layer are added element-wise, then smoothed by a 3×3 convolution. Finally, the aggregated features from the shallowest layer are taken as the multi-scale fusion output. .

[0036]

[0037] Step 3: Mask Feature Decoding and Attention Bias Generation

[0038] Fusion features The specific artifact features are generated through two independent MLP projection tasks. ) and attention features ( ), and dynamic semantics from SQM After enhancing the nonlinear expressive power through MLP, the final semantic meaning is obtained. Generate interpretable forgery probability maps by leveraging the interaction of semantic and artifact features. Attention bias is generated by leveraging the interaction of semantic and attention features. .

[0039]

[0040]

[0041] in This represents the dot product contraction along the feature dimension.

[0042] Step 4: Dynamic attention signal injection

[0043] To explicitly guide the CLIP backbone to focus on potentially forged regions, this invention employs a feature injection method based on dynamic attention bias. Specifically, it utilizes the attention bias generated by the adapter. As an external signal, a multi-head self-attention mechanism is injected into the CLIP encoder to dynamically adjust its attention distribution. The number of attention heads is... It is 16. =16 This refers to spatial resolution.

[0044] Before performing attention injection, the input sequence for the CLIP Transformer is constructed. Specifically, multiple semantic tokens, the original CLIP classification token, and the visual patch token are concatenated to form the updated input sequence:

[0045]

[0046] CLIP original classification token The number of replications, q, is 128, consistent with the number of queries performed by the dynamic semantic query module in step 2, to ensure that the q attention biases generated by the Adapter can form a one-to-one injection relationship with the q semantic tokens in CLIP. The semantic tokens are obtained by copying the original CLIP classification tokens during the initialization phase and are dynamically updated through a self-attention mechanism during subsequent Transformer encoding. This represents the visual token corresponding to the image patch.

[0047] Then Flatten it in the spatial dimension and reconstruct it into a form that matches the multi-head attention mechanism: Based on the length of the concatenated input sequence, a corresponding attention mask matrix is ​​constructed to adjust the attention relationships between different types of tokens. This attention mask is expanded into a mask sequence with the same number of CLIP Transformer layers: The Transformer has 24 layers (m), and the attention mask for each layer is... This bias term is injected into the attention calculation of the corresponding Transformer block. In the multi-head self-attention mechanism, the updated attention calculation takes the following form:

[0048]

[0049] in, This indicates that the feature dimension corresponding to each attention head in multi-head self-attention is 64. Used to dynamically adjust the distribution of attention.

[0050] Step 5: Multi-task supervision and classification

[0051] In this step, multiple supervision and optimization strategies were employed to improve model performance. First, for the classification task, the global output token of CLIP was frozen and used for binary classification to distinguish between real and fake samples. Next, artifact supervision was introduced, generating artifact maps from the intermediate output of the Adapter and optimizing them using the supervised loss with the real boundary mask, thereby improving the model's ability to capture details. Furthermore, query diversity loss and attention focus loss were added within the Adapter to enhance the model's semantic interpretability. Regarding loss calculation, intra-class contrastive loss was designed to constrain the feature distribution aggregation of samples of the same class, and inter-class contrastive loss was designed to improve the separation of features from different classes, thereby further optimizing the overall model performance. Finally, through end-to-end training, the model achieved co-modeling of global and local forgery information, simultaneously ensuring detection accuracy and interpretability.

[0052] In the last layer (layer 24) of the CLIP encoder, q semantic tokens enhanced by dynamic attention are extracted. First, linear interpolation (LERP) enhancement is performed on the semantic tokens of the same type of samples, as shown in Equation (14):

[0053]

[0054] in, These are semantic token feature pairs randomly sampled from samples of the same category. For the new enhanced features, These are the interpolation coefficients. Indicates that it conforms to the interval [ The uniform distribution of ], in which .

[0055] After feature enhancement, the semantic token is mapped to two-dimensional classification logits using a classification head composed of 1D convolution and MLP, and then converted into spoofing probability predictions via Softmax. Cross-entropy loss is used for classification.

[0056]

[0057] in For real labels, This represents the forgery probability predicted by the model.

[0058] The boundary modeling loss compares the consistency between the predicted mask's response intensity within the boundary region and the fake mask, while the constrained model focuses on the details of the fake edge.

[0059]

[0060] in, This represents the forgery probability graph generated by the model. A predefined boundary mask for filtering. Its generation method is to merge real boundaries. Divide into 16×16 patches. If a patch contains boundary pixels, set all elements in B corresponding to that patch to 1; otherwise, set them to 0. To truly integrate boundaries, This indicates element-wise multiplication.

[0061] Query diversity loss ( This encourages queries with different semantics to capture different information by minimizing their cosine similarity.

[0062]

[0063] in, It is the cosine similarity matrix between each pair of queries within each batch. Is it an off-diagonal mask (where (identity matrix), weights It is 0.1. This indicates that the average of all elements in the matrix is ​​calculated.

[0064] Loss of attention ( This encourages each query to focus its attention on a few patches (i.e., sparsify the attention distribution), which is achieved by minimizing the entropy of the attention distribution.

[0065]

[0066] in, This represents the attention distribution of the p-th semantic query of the n-th sample across L patches, with weights... =0.01, ϵ=10 -9 It is a very small number introduced for numerical stability. This represents the average of the entropy values ​​for all samples (n=1,2,...,N) and all semantic queries (p=1,2,...,q).

[0067] The self-supervised semantic loss obtained through the above calculations for:

[0068] To enhance the model's ability to model the differences between fake and real regions and improve its discrimination performance in complex fake scenarios, this invention introduces a contrastive learning loss function design that combines block-level and sample-level approaches. This design employs differentiated contrastive learning strategies based on the functional positioning of different network modules, thereby achieving a balance between local detail modeling and global semantic distinction. Let the feature representation be X, and the contrastive learning loss function be defined as:

[0069]

[0070] in, and This represents positive sample features from the same category, where D represents the entire sample feature set, and the superscript indicates that the features are from the same category. "Specific labels for negative sample sets", Indicates and A set of negative samples belonging to different categories. Cosine similarity function, temperature coefficient The loss is 0.5. The above contrastive learning loss... For a general form, this invention instantiates block-level contrastive learning loss according to different contrast granularities. Comparative learning loss with sample level .

[0071] In the Adapter, only block-level contrastive learning loss is introduced. Specifically, the Adapter extracts feature representations corresponding to multiple image patches from the intermediate layer features and performs contrastive learning only on forged samples. Based on the spatial annotation information of the forged region, each forged sample is divided into multiple image patches, and each image patch is labeled with a category: each element in the feature corresponds to a patch after the division; when the forged region covers more than 10% of the patch, it is marked as forged, otherwise it is marked as real. In the CLIP module, a block-level contrastive learning mechanism is also introduced, and its form is consistent with the block-level contrastive learning in the Adapter. Further, sample-level contrastive learning is introduced, using only real samples as anchors. By narrowing the feature distance between real samples and widening the distance between real samples and forged samples, global semantic-level discrimination enhancement is achieved. For the adapter, the features of intermediate layers 4, 5, and 6 are used for contrastive learning. Since the backbone parameters of CLIP are frozen and cannot be updated directly through backpropagation, a trainable 1×1 convolutional layer is introduced in the sixth layer of CLIP through a 1D convolutional sequence to perform feature transformation on the visual token, and contrastive learning is performed on the transformed features. Subsequently, the features enhanced by contrastive learning are weighted by preset coefficients. The original visual token is added back to 0.05, thereby achieving adaptive enhancement of features without destroying CLIP's original semantic representation capabilities.

[0072]

[0073]

[0074] in This represents the contrastive learning loss of the adapter module. The block-level contrastive learning loss uses only real image blocks as the optimization objective. It optimizes by enhancing the feature consistency between real image blocks and using the difference between real blocks and fake blocks as a negative constraint. Using sample-level contrastive learning loss, real samples are used as anchors. By narrowing the feature distance between real samples and widening the feature distance between real samples and fake samples, global semantic-level discrimination enhancement is achieved.

[0075] Therefore, the total loss for:

[0076]

[0077] in This represents the classification loss, used to supervise the model in binary classification of real samples and fake samples. This represents the region or mask-related loss, used to constrain the model's ability to perceive the spatial distribution of fake regions. This represents semantic consistency loss, used to enhance the stability and discriminative power of high-level semantic features. Loss weight coefficients. It is 10. It is 200. It is 20. It is 10. The loss function is set to 10. End-to-end co-training is performed by optimizing this loss function. The entire model is trained on the FF++c23 training set with a batch size of 16, using the Adam optimizer, a learning rate of 0.0002, and a weight decay of 0.0005. 16 frames are extracted from each video for training or testing.

Claims

1. A face forgery detection method based on CLIP and semantic enhancement adapter, characterized in that, The process includes the following steps: Step 1, global semantic feature extraction based on Clip pre-trained visual encoder; Step 2, dynamic modeling of fake regions and generation of artifact maps based on semantic enhancement adapter; Step 3, artifact map decoding and interpretability enhancement; Step 4, dynamic attention signal injection; Step 5, multi-task supervision and classification.

2. The method according to claim 1, characterized in that: Step 1, Global Semantic Feature Extraction Based on Clip Pre-trained Visual Encoder, specifically includes: During the input phase, the original image tensor is denoted as... Where N and C represent the batch size and the number of channels, respectively, and N=16, C=3; The original image resolution is 256×256. The image is scaled to the CLIP-required size of 224×224 using bilinear interpolation. Another input is... This specifies the mapping relationship between the Adapter layer and the CLIP vision backbone Transformer encoder layer, where the key-value pairs 6:10 and 7:15 represent the Adapter layer numbers 6 and 7 and the 10th and 15th layers of the Transformer encoder in the CLIP vision backbone, respectively. Next, the CLIP visual backbone is loaded and frozen, and the CLIP feature extraction interface is called to obtain multi-layer visual features organized in dictionary form. This dictionary contains the layers that appear in the fusion_map, as well as the cls token and patchtoken for layer 0; Layer 0 is the input embedding layer, responsible for image segmentation, linear mapping, and positional encoding, but it does not perform self-attention computation; it is equivalent to the "word embedding layer" in ViT. The other layers are the actual Transformer blocks, containing self-attention mechanisms and multilayer perceptrons. The cls token is used to represent the global semantics of the entire image, and its shape is... The shape of the patch token is The number of patches L is 256, and the dimension of the patches D is 1024; The Adapter is passed in for cross-branch fusion.

3. The method according to claim 2, characterized in that, Step 2, Dynamic Modeling of Forged Regions and Generation of Artifact Maps Based on Semantic Enhancement Adapter, specifically includes: Structurally, the Adapter is built using the first 8 layers of the ViT-Tiny Transformer, each layer containing multi-head self-attention, normalization, and a feedforward network (MLP). The input image size is 256×256, which is first divided into 16×16 patches, generating 256 patch tokens (L=256). Each patch is linearly projected to obtain an embedding representation, resulting in the initial patch token representation. Where d is 192; then position encoding processing is performed; let The two-dimensional position encoding of ViT is Adding the position code to the patch tokens and normalizing them yields: Update layer by layer on each block of ViT. The dynamic semantic query module is triggered at layers 4 and 8: first, dynamic semantic query initialization is performed, introducing a learnable parameter matrix. The number of semantic queries selected The value is 128, and the feature dimension d is 192; the query expands to the following during forward propagation: Where N represents the batch size; subsequently, the semantic query vector is constructed. Patch features of the current layer Interaction is achieved through a multi-head attention mechanism, where L=256 represents the number of patch tokens in ViT; the calculation formula is: This process generates representations with semantic diversity and spatial selectivity. Attention map obtained simultaneously ; Finally, the dynamic semantics are obtained by enhancing the non-linear expressive power of the query representation through an MLP. : To address the semantic shift issue that may result from the disconnect between the Adapter module and the backbone features, a semantically guided fusion module based on an attention mechanism is designed, using CLIP intermediate layer features. As a guide, features are dynamically fused into the Adapter representation stream to achieve semantic collaboration between the backbone and auxiliary paths. In the ViT backbone network, features are updated layer by layer in each Transformer block. When the current ViT layer is a preset fusion layer, the semantically guided adaptive fusion module is triggered to fuse the features from CLIP layers 10 and 15 with the Adapter layers 6 and 7, respectively. The features from the Adapter are then used as... Reconstructed into two-dimensional form Where H=W=16; guiding features The data is reconstructed into a two-dimensional form and projected onto the channel dimension using a 1×1 convolutional layer, mapping it to a feature space consistent with the main features. Utilizing the guiding features after projection Generate query vector Q and key vector K, and use the main features Generate a value vector V, and compute a spatial semantic weighted representation using a self-attention mechanism: Among them, the scaling factor used to balance the inner product scale The attention output is reconstructed into spatial features and fused with the original master features using a residual approach. in The characteristics after fusion This is a learnable scaling factor, initialized to 0, which automatically learns the optimal value through backpropagation to adjust the fusion intensity. Furthermore, to fully explore semantic information at different levels and improve the model's ability to model multi-scale targets, a multi-scale feature aggregation module was designed (as shown in Figure 3). In this invention, features at each layer are processed sequentially from deep to shallow (8→6→4). Let the features output by layers 4, 6, and 8 of the Adapter be respectively... For each layer of features Channel alignment is achieved through 1×1 convolution. : Then, a top-down approach is used for feature fusion; the deepest layer ( =8) Directly used as the initial aggregation feature: For other layers (i=6,4), the features from the previous layer and the features from the current layer are added element-wise, then smoothed using a 3×3 convolution. Finally, the aggregated features from the shallowest layer are taken as the multi-scale fusion output. ; 。 4. The method according to claim 3, characterized in that, Step 3, mask feature decoding and attention bias generation, specifically involves: Fusion features The specific artifact features are generated through two independent MLP projection tasks. ) and attention features ( ), and dynamic semantics from SQM After enhancing the nonlinear expressive power through MLP, the final semantic meaning is obtained. Generate interpretable forgery probability maps by leveraging the interaction of semantic and artifact features. Attention bias is generated by leveraging the interaction of semantic and attention features. ; in This represents the dot product contraction along the feature dimension.

5. The method according to claim 4, characterized in that, Step 4, Dynamic Attention Signal Injection, specifically involves: To explicitly guide the CLIP backbone to focus on potential forgery regions, a feature injection method based on dynamic attention bias is adopted; specifically, attention bias generated by the adapter is utilized. As an external signal, a multi-head self-attention mechanism is injected into the CLIP encoder to dynamically adjust its attention distribution; wherein, the number of attention heads is... It is 16. =16 Spatial resolution; Before performing attention injection, an input sequence for the CLIP Transformer is constructed; specifically, multiple semantic tokens, the original CLIP classification token, and the visual patch token are concatenated to form the updated input sequence: CLIP original classification token The number of replications q is 128, which is consistent with the number of queries in the dynamic semantic query module in step 2, to ensure that the q attention biases generated by the Adapter can form a one-to-one injection relationship with the q semantic tokens in CLIP; the semantic tokens are obtained by copying the original classification tokens in CLIP during the initialization phase, and are dynamically updated through the self-attention mechanism in the subsequent Transformer encoding process; This represents the visual token corresponding to the image patch; Then Flatten it in the spatial dimension and reconstruct it into a form that matches the multi-head attention mechanism: Based on the length of the concatenated input sequence, a corresponding attention mask matrix is ​​constructed to adjust the attention relationship between different types of tokens. This attention mask is expanded into a mask sequence with the same number of CLIP Transformer layers. The Transformer has 24 layers (m), and the attention mask for each layer is... This bias term is injected into the attention calculation of the corresponding Transformer block; in the multi-head self-attention mechanism, the updated attention calculation form is as follows: in, This indicates that the feature dimension corresponding to each attention head in multi-head self-attention is 64. Used to dynamically adjust the distribution of attention.

6. The method according to claim 5, characterized in that, Step 5, Multi-task Supervision and Classification, specifically includes: In this step, multiple supervision and optimization strategies were employed to improve model performance. First, for the classification task, the global output token of CLIP was frozen and used for binary classification to distinguish between real and fake samples. Next, artifact supervision was introduced, generating artifact maps from the intermediate output of the Adapter and optimizing them using the supervision loss with the real boundary mask, thereby improving the model's ability to capture details. In addition, query diversity loss and attention focus loss were added inside the Adapter to improve the model's semantic interpretability. In terms of loss calculation, intra-class contrastive loss was designed to constrain the feature distribution aggregation of samples of the same class, and inter-class contrastive loss was designed to improve the separation of features of different classes, thereby further optimizing the overall performance of the model. Finally, through end-to-end training, the model achieved co-modeling of global and local forgery information, which can simultaneously ensure detection accuracy and interpretability. In the last layer (layer 24) of the CLIP encoder, q semantic tokens enhanced by dynamic attention are extracted. First, linear interpolation (LERP) enhancement is performed on the semantic tokens of the same type of samples, as shown in Equation (14): in, These are semantic token feature pairs randomly sampled from samples of the same category. For the new enhanced features, These are the interpolation coefficients. Indicates that it conforms to the interval [ The uniform distribution of ], in which ; After feature enhancement, the semantic token is mapped to two-dimensional classification logits through a classification head composed of 1D convolution and MLP, and then converted into spoofing probability predictions via Softmax; the classification loss uses cross-entropy loss. in For real labels, The probability of forgery predicted by the model; Boundary modeling loss constrains the model to focus on the details of the fake edge by comparing the consistency between the predicted mask's response intensity within the boundary region and the fake mask. in, This represents the forgery probability graph generated by the model. A predefined boundary mask for filtering. Its generation method is to merge real boundaries. Divide into 16×16 patches. If a patch contains boundary pixels, set all elements in B corresponding to that patch to 1; otherwise, set them to 0. To truly integrate boundaries, This represents element-wise multiplication; Query diversity loss ( This encourages queries with different semantics to capture different information by minimizing their cosine similarity. in, It is the cosine similarity matrix between each pair of queries within each batch. It is an off-diagonal mask, in which The identity matrix has weights. It is 0.

1. This represents the average of all elements in the matrix. Loss of attention ( This encourages each query to focus its attention on a few patches (i.e., sparsify the attention distribution), which is achieved by minimizing the entropy of the attention distribution; in, This represents the attention distribution of the p-th semantic query of the n-th sample across L patches, with weights... =0.01, ϵ=10 -9 It is a very small number introduced for numerical stability. This represents the average of the entropy values ​​for all samples (n=1,2,...,N) and all semantic queries (p=1,2,...,q); The self-supervised semantic loss obtained through the above calculations for: To enhance the model's ability to model the differences between fake and real regions and improve its discrimination performance in complex fake scenarios, this invention introduces a contrastive learning loss function design that combines block-level and sample-level approaches. This design employs differentiated contrastive learning strategies based on the functional positioning of different network modules, thereby achieving a balance between local detail modeling and global semantic distinction. Let the feature representation be X, and the contrastive learning loss function be defined as: in, and This represents positive sample features from the same category, where D represents the entire sample feature set, and the superscript indicates that the features are from the same category. "Specific label for negative sample set", Indicates and A set of negative samples belonging to different categories. Cosine similarity function, temperature coefficient The loss is 0.5; the contrastive learning loss mentioned above For a general form, this invention instantiates block-level contrastive learning loss according to different contrast granularities. Comparative learning loss with sample level ; In the Adapter, only block-level contrastive learning loss is introduced. Specifically, the Adapter extracts feature representations corresponding to multiple image patches from the intermediate layer features and performs contrastive learning only on fake samples. Based on the spatial annotation information of the fake region, each fake sample is divided into multiple image patches, and each image patch is labeled with a category: each element in the feature corresponds to a patch after division; when the fake region covers more than 10% of the patch, it is marked as fake, otherwise it is marked as real. In the CLIP module, a block-level contrastive learning mechanism is also introduced, which is consistent with the block-level contrastive learning in the Adapter. Sample-level contrastive learning is further introduced, using only real samples as anchors. By narrowing the feature distance between real samples and widening the distance between real samples and fake samples, global semantic-level discrimination enhancement is achieved. For the Adapter, the features of intermediate layers 4, 5, and 6 are used for contrastive learning. In the sixth layer of CLIP, a trainable 1×1 convolutional layer is introduced through a 1D convolutional sequence to transform the visual token features, and contrastive learning is performed on the transformed features. Subsequently, the features enhanced by contrastive learning are weighted by preset coefficients. Add back the original visual token to 0.05; in This represents the contrastive learning loss of the adapter module; For block-level contrastive learning loss, only real image blocks are used as optimization targets. The optimization is achieved by enhancing the feature consistency between real image blocks and using the difference between real blocks and fake blocks as negative constraints. Using sample-level contrastive learning loss, real samples are used as anchors only. By narrowing the feature distance between real samples and widening the feature distance between real samples and fake samples, global semantic level discrimination enhancement is achieved. Therefore, the total loss for: in This represents the classification loss, used to supervise the model in performing binary classification between real and fake samples; This represents the region or mask-related loss, used to constrain the model's ability to perceive the spatial distribution of fake regions; This represents the semantic consistency loss, used to enhance the stability and discriminative power of high-level semantic features; the loss weight coefficients... It is 10. It is 200. It is 20. It is 10. The loss function is set to 10; end-to-end co-training is performed by optimizing this loss function; the entire model is trained on the FF++c23 training set with a batch size of 16, using the Adam optimizer with a learning rate of 0.0002 and a weight decay of 0.0005; 16 frames are extracted from each video for training or testing.

Images