Deep forgery detection method based on visual language model

Abstract

The invention discloses a deep forgery detection method based on a visual language model, and relates to the field of image forensics. The deep forgery detection method based on the visual language model aims to combine multi-source information to improve the discrimination capability of the model on a real image and a generated image. The method comprises the following steps: firstly, extracting image features through an image encoder of a pre-trained CLIP model; meanwhile, a frequency domain enhanced counterfeit perception adapter is embedded in the image encoder to mine potential anomalies of counterfeit images in the image domain and the frequency domain. Secondly, a manual feature extraction module is provided, discriminative low-dimensional features are extracted from the four aspects of the edge, the texture, the frequency and the symmetry of the image, and the discriminative low-dimensional features are used as auxiliary information input in the forgery detection process, so that the robustness and the interpretability of the model are improved; meanwhile, the text cue words are converted into feature vectors through a text encoder of a pre-training CLIP model; and finally, the model predicts a forgery score by calculating the cosine similarity between the image features and the text features so as to realize the discrimination of the authenticity of the image. According to the method, the problem that the detection capability of the model on the cross-dataset is insufficient is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image forensics, specifically to a deepfake detection method based on a visual language model. Background Technology

[0002] With the rapid development of technologies such as Generative Adversarial Networks (GANs) and Diffusion Models (DMs), generated images are widely used in various applications. While these images bring creative convenience and artistic innovation, they also pose serious challenges such as the spread of misinformation, privacy violations, and public misleading. Furthermore, it is undeniable that different generation methods will emerge in the future, resulting in even more realistic generated images. Therefore, developing a universal method for detecting forgery has become an important issue in ensuring the credibility of visual media.

[0003] Current generative image detection methods typically employ a supervised learning paradigm, training a binary classification model to distinguish between real and forged images. These methods generally achieve good detection performance within the training data distribution, but they often exhibit significant performance degradation when faced with unknown generative models or novel forged images, revealing their insufficient generalization ability. This is because they focus more on salient and easily detectable features in forged images. Therefore, the model may ignore discriminative features in real images and incorrectly assume they are inconsistent with learned forged features, leading to misclassification.

[0004] Given the powerful zero-shot capability exhibited by large-scale pre-trained models (such as Contrastive Language-Image Pre-training, CLIP), an increasing number of methods have incorporated them into generative image detection tasks in recent years, achieving significant results. The CLIP model is a multimodal (visual language) model based on contrastive learning, trained with 400 million image-text pairs, used for matching images and text. For example, given an input image of a dog and a text description, the model determines the image category by calculating the degree of matching between the image and text. Among existing methods, Ojha et al. achieved effective identification of forged images by freezing the pre-trained CLIP model and training a linear classifier on it. Tan et al. proposed a gradient-based general artifact representation method, effectively achieving cross-model, cross-category GAN-generated image detection by utilizing image gradients extracted from a pre-trained CNN model as forgery traces. Khan et al. investigated the performance of four different transfer learning strategies on the CLIP model: Linear Probing, Fine-tuning, Prompt Tuning, and CLIP-Adapter. They verified that the transfer learning strategy combining the visual and language branches of CLIP significantly outperformed linear detection methods using only visual features.

[0005] Based on the search of the above information, the following shortcomings can be identified in the existing technology:

[0006] (1) Most of these methods are based on general transfer learning mechanisms and are not designed specifically for forgery detection tasks. Therefore, they still have shortcomings in fine-grained forgery trace modeling and frequency domain anomaly perception. In particular, the robustness and generalization ability of existing methods still need to be improved when facing high-quality generated images (such as diffusion models).

[0007] (2) The above methods all have certain limitations: they usually only have good discrimination ability for one type of generated image, and they show insufficient detection ability when facing images generated by unknown generation models. Summary of the Invention

[0008] This invention aims to provide a deep forgery detection method based on a visual language model, comprising CLIP pre-trained images, a text encoder, a hand-crafted feature extraction module, and a Frequency-Enhanced Forgery-Aware Adapter (FEFA). It aims to combine multi-source information to improve the discriminative power of forgery detection.

[0009] The technical solution of the present invention includes the following steps:

[0010] Step 1. Image Feature Extraction:

[0011] This step aims to deeply mine potential forgery features in images. Image features are extracted using the Vision Transformer (ViT) image encoder of the pre-trained CLIP model. Frequency-domain enhanced forgery-aware adapters are then inserted at layers 8, 16, and 24 of the image encoder to mine forgery features from both the image and frequency domains. The frequency-domain enhanced forgery-aware adapter is an improvement on the Forgery-aware adapter (FAA) module proposed by the FatFormer model, incorporating frequency band attention and frequency domain feature enhancement modules in its frequency domain branch, effectively improving the model's ability to model frequency-domain forgery features. The extracted features are non-linearly adjusted through an adapter network to enhance their adaptability in deep forgery detection tasks, and then weighted and fused with the image features output by CLIP using residual fusion, resulting in image features with stronger discriminative power.

[0012] The specific formula for the frequency domain detail enhancement module is as follows:

[0013] f enh =Conv 1×1 (ReLU(G-Conv 3×3 (f dwt )))

[0014] Among them, f dwt ∈R B×C×H×W This represents the frequency domain features obtained by concatenating the four frequency bands (LL, LH, HL, HH) obtained from DWT. G-Conv represents a grouped convolution with 8 groups, used to model local forgery patterns and improve the expressive power of edge and high-frequency regions. enh This indicates the enhanced features.

[0015] The specific formula for the band attention module is as follows:

[0016] w c =Softmax(MLP(LN(MeanPool(f c c∈{LL,LH,HL,HH}

[0017] Among them, f c w represents the characteristics of the c-th frequency band. c This corresponds to its attention weight. Weight w cThis method uses weighted feature aggregation across four frequency bands for fusion, effectively improving the model's focus on frequency regions with more forged information. MeanPool represents average pooling, LN represents layer normalization, and MLP represents multilayer perceptron, used for feature mapping and learning complex nonlinear relationships. Softmax represents the normalized exponential function.

[0018] Step 2. Manual feature extraction:

[0019] This invention designs a handcrafted feature extraction module based on image statistical characteristics and human visual inspiration. It extracts discriminative low-dimensional features from four dimensions: edge, texture, frequency, and symmetry. These features are used as auxiliary information input in the forgery detection process to improve the robustness and interpretability of the model. Specific sub-steps are as follows:

[0020] (1) Edge strength analysis: The Laplacian operator is used to enhance the edges of the input image, and the average edge response of the entire image is used as the edge strength index. The specific formula is as follows:

[0021] f edge =E[|Laplacian(I)|]

[0022] Among them, f edge Let I represent the edge intensity feature, I represent the input grayscale image, Laplacian(·) represents the convolution operation of applying the two-dimensional Laplacian operator to the image, and E[·] represents the average value of all elements.

[0023] (2) Texture consistency analysis evaluates the texture consistency of an image from two perspectives: texture contrast and local homogeneity. Texture contrast is based on the horizontal gray-level difference, reflecting the intensity changes of the image in local directions. The specific formula is:

[0024] f contrast =E[|I(x,y+1)-I(x,y)|]

[0025] Among them, f contrast This represents the texture contrast feature, where x represents the column coordinate of the image and y represents the row coordinate of the image.

[0026] Local homogeneity is calculated by using a sliding window to determine the pixel standard deviation of a local region and then modeling its smoothness using an exponential function. The specific formula is:

[0027] f homo =E[exp(-σ local )]

[0028] Among them, f homo σ represents the characteristic of local homogeneity. local This represents the standard deviation of pixel grayscale within the sliding window.

[0029] (3) Symmetry analysis: The mean square error between the original image and its horizontally flipped image was calculated as an indicator of symmetry violation. The specific formula is as follows:

[0030] f sym =E[(I(x,y)-I flip (x,y)) 2 ]

[0031] Among them, f sym Indicating symmetry, I flip This represents the image after it has been horizontally flipped.

[0032] (4) Spectral energy analysis: A Fast Fourier Transform (FFT) is performed on the input grayscale image, and the logarithmic mean of the spectral amplitude is used as the frequency domain energy feature. The specific formula is as follows:

[0033] f freq =E[log(|FFT)] shift (I)|+ε)]

[0034] Among them, f freq FFT represents the spectral energy characteristics. shift (I) indicates that a centering operation is applied to the Fourier transform result, shifting the low-frequency components in the spectrum to the center position to enhance the symmetry and interpretability of the spectral distribution. ε = 1e-5 is a small constant to prevent numerical instability.

[0035] (5) The high-frequency information analysis module uses the absolute difference between the original image and its Gaussian blurred image as a measure of the degree of high-frequency detail preservation. The specific formula is:

[0036] f highfreq =E(|IG σ (I)|)

[0037] Among them, f highfreq G represents high-frequency information features. σ (I) represents the blurred image obtained by convolving the input grayscale image with a two-dimensional Gaussian kernel of size 5×5 with a standard deviation σ = 1.5.

[0038] All the aforementioned handcrafted features are extracted and concatenated into a set of low-dimensional forgery feature vectors, which serve as auxiliary inputs to the model to improve its ability to detect forged images and the interpretability of the decision-making process.

[0039] Step 3. Feature Fusion:

[0040] The image features obtained in step 1 and the handcrafted features obtained in step 2 are combined by the feature fusion module, and then subjected to layer normalization and linear transformation to achieve effective fusion of features of different modalities.

[0041] Step 4. Text Features:

[0042] The CLIP model's text encoder transforms the input text into feature vectors. The text features consist of fixed prompts "real" and "fake," which are used to represent real and fake images, respectively, guiding the visual language model to perform binary classification semantic alignment.

[0043] Step 5. Align text and images:

[0044] The cosine similarity between the fused features obtained in step 3 and the text features obtained in step 4 is calculated using cosine similarity to predict whether the image is a fake image.

[0045] Compared to existing methods, this invention proposes a deep forgery detection method based on a visual language model. Addressing the shortcomings of insufficient model feature extraction and the fact that most existing methods rely on general transfer learning mechanisms without being tailored to forgery image detection tasks, this invention proposes a manual feature extraction module that runs in parallel with the original image encoder. This module extracts discriminative low-dimensional features from four dimensions: edge, texture, frequency, and symmetry. These features are used as auxiliary information input in the forgery detection process to improve the model's robustness and interpretability. Simultaneously, a frequency-domain enhanced forgery perception adapter, improved from the FAA module, is embedded into the CLIP image encoder to enhance the model's ability to extract both image and frequency domain features. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the deepfake detection method framework of the present invention;

[0047] Figure 2 This is the distribution of logits values ​​of the forgery features extracted by this invention;

[0048] Figure 3 This relates to the impact of the number of embedded frequency-domain enhanced fake perception adapters on the performance of the method.

[0049] Figure 4 This invention relates to the detection performance of images after post-processing (JPEG compression and Gaussian blur).

[0050] Figure 5 This describes the performance of the present invention on a smaller training set;

[0051] Figure 6 This is a comparison of the average accuracy of the present invention with other methods on multiple publicly available test sets;

[0052] Figure 7 This is a comparison of the accuracy of this invention with other methods on multiple publicly available test sets;

[0053] Figure 8 This is the flow of the CLIP model in this invention. Detailed Implementation

[0054] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0055] Please see Figure 1-8 The present invention provides the following technical solution: a deepfake detection method based on a visual language model, comprising the following steps:

[0056] S1. Pre-trained Model Preparation: The pre-trained CLIP-L / 14 model is used, where L stands for Large and 14 indicates that the image patch size in the image encoder is 14×14. The CLIP model was trained by OpenAI using 400 million image-text pairs. It aims to enable the model to perform image classification tasks in zero-shot learning scenarios through contrastive learning, thereby alleviating the problem of insufficient generalization ability in forgery detection caused by limited labeled data or diverse forgery forms. Its workflow is shown in the attached figure. Figure 8 As shown.

[0057] S2. Preparation of Pre-training and Test Sets: The ProGAN dataset constructed by Wang et al. was used. This dataset contains 360,000 real images from LUSN and 360,000 images synthesized by ProGAN, covering 20 categories (airplane, bicycle, bird, boat, bottle, bus, car, cat, hair, cow, dining table, dog, horse, motorbike, potted plant, sheep, sofa, train, TV monitor). Each class contains 18,000 real images and 18,000 fake images. Following the setup of other methods in this field, this invention was trained on a 200k dataset (keeping the categories the same, reducing the number of images per class to 10,000) and a 4-class dataset (horse, chair, cat, car). These two datasets can be downloaded from https: / / tinyurl.com / 2ek7t4sb and https: / / github.com / Michel-liu / FatFormer, respectively. The test set includes images generated using 18 different methods, along with their corresponding real-world images. Details of the test set are shown in Table 1. The test set can be downloaded from https: / / tinyurl.com / 5b3fh7fh.

[0058] Table 1. Statistical data for the test set

[0059]

[0060]

[0061] S3. Model Setup: Experimental Hardware Environment for this Invention: CPU is... The system used a Silver 4210R processor with an NVIDIA GeForce RTX 3090 GPU and 24GB of GPU memory. The software environment consisted of Linux as the operating system, Python 3.8, CUDA 11.6, and PyTorch 1.13.0. The model was initialized by freezing the image and text encoders of the pre-trained CLIP model. The frequency-domain enhanced spoofing perception adapter, manual feature extraction module, adapter network, and feature fusion module were trained, and a random seed was set to ensure reproducibility of the training process. A stochastic gradient descent (SGD) optimizer was used, with an initial learning rate of 0.002, 3 training epochs, and 1 warm-up epoch. The learning rate during the warm-up phase was 1e-5, and during training, the learning rate was gradually decreased using a cosine annealing strategy.

[0062] S4. Training the Model: First, import the 200k dataset or 4-class dataset from S2 into the initial model in S1 and S3 for training. This includes:

[0063] Step 1 involves preprocessing the input image, including random cropping, random horizontal flipping, normalization, JPEG compression, and Gaussian blur enhancement. The preprocessed image then passes through the CLIP model's image encoder, ViT. After the attention modules at layers 8, 16, and 24 of ViT, the image passes through a frequency domain enhancement forgery perception adapter. This module performs feature analysis in both the image and frequency domains. The image domain uses a lightweight structure consisting of two one-dimensional convolutions and a non-linear activation function to capture low-level forgery features in the image. The frequency domain consists of a frequency domain detail enhancement module, a frequency band attention module, and a feedforward network (FNN). After decomposition using DWT (wavelet transform), four frequency bands (LL, LH, HL, HH) are obtained. These features are concatenated and input into the frequency domain detail enhancement module to enhance the detail and texture information in each frequency band. The enhanced features are then input into the frequency band attention module to improve the model's focus on frequency regions with more forgery information. A feedforward network is then used to further extract forgery semantics. Finally, the model is restored to the image domain using IDWT (inverse wavelet transform) and weighted and fused with image domain features to serve as the adapter's output. The frequency domain enhanced forgery-aware adapter runs in parallel with the MLP branch of ViT, jointly outputting the results. The extracted features undergo nonlinear adjustment through the adapter network and are then weighted and residually fused with the image features output by CLIP to obtain image features with stronger discriminative capabilities.

[0064] Step 2: The preprocessed image is input into a handcrafted feature extraction module that runs parallel to the CLIP model's image encoder to extract discriminative handcrafted features. This invention designs a handcrafted feature extraction module based on image statistical properties and human visual inspiration, comprising five sub-modules: edge intensity analysis module, texture consistency analysis module, symmetry analysis module, spectral energy analysis module, and high-frequency information analysis module. It extracts discriminative low-dimensional features from four dimensions: edge, texture, frequency, and symmetry, which are used as auxiliary information input in the forgery detection process to improve the model's robustness and interpretability.

[0065] Step 3: The image features obtained in Step 1 and the handcrafted features obtained in Step 2 are concatenated and input into the feature fusion module. This module adopts a layer normalization and linear transformation structure to map the concatenated features to a unified dimension, so as to achieve effective fusion and unified expression of cross-modal features.

[0066] Step 4: Construct fixed text prompts “real” and “fake” to represent the categories of real and fake images, respectively, and input them into the text encoder of the CLIP model. The text encoder first performs word segmentation on the input text, maps it to the corresponding word embedding vector sequence, and adds learnable positional encoding. Then, it extracts semantic features through a multi-layer Transformer structure and outputs a global representation vector of the text to represent the semantic center of each category, which is used for subsequent semantic alignment with image features and for forgery recognition.

[0067] Step 5: Align the image features fused in Step 3 with the text features extracted in Step 4. Calculate the cosine similarity between the image features and the text features to obtain the matching scores of the image in the two semantic categories. Determine which text semantics the image is closer to based on the similarity score, thereby completing the prediction and classification of forged images and achieving deep forgery detection under image-text alignment.

[0068] After completing the image-text alignment and obtaining the fake prediction results, the cross-entropy loss function is used to measure the error between the prediction results and the true labels, and the model parameters are backpropagated and updated through the stochastic gradient descent (SGD) optimizer.

[0069] The training process employs iterative optimization, with each epoch based on the updated model weights from the previous epoch, ensuring continuous parameter optimization. Finally, the model weights that perform best on the validation set are selected as the final model for the inference phase.

[0070] Results Analysis: The deep forgery detection method based on a visual language model proposed in this invention was tested on datasets generated by 18 different generation methods (such as generative adversarial networks and diffusion models). The test results are attached. Figure 6 As shown in Figure 7, mean accuracy (mAP) and mean precision (mAcc) are used as the main evaluation metrics. It can be observed that the method proposed in this invention has an mAP of 99.09% and a mAcc of 95.37%, both superior to other methods.

[0071] Appendix Figure 2 The logits value distribution of the forged features extracted by this invention shows that, regardless of whether the image is generated by ProGAN within the training distribution or by other unknown generation models, this method can still maintain good class distinction, with clear distribution and distinct boundaries between real and forged images in the feature space.

[0072] Appendix Figure 4 The invention demonstrates its performance on JPEG compression and Gaussian blur post-processed images, exhibiting good performance without special training.

[0073] Appendix Figure 5 The results demonstrate the performance of the proposed method with a reduced training set size, showing that the model often performs better on a larger training set, although the difference is not significant. It also indicates that the proposed method exhibits good performance even with limited training resources.

[0074] In summary, the deep forgery detection method based on a visual language model proposed in this invention significantly improves the model's detection performance on both real and generated images through a manual feature extraction module and a frequency domain-enhanced forgery perception adapter. Even with limited training samples or images that have undergone post-processing, this method still exhibits good detection performance.

Claims

1. A deepfake detection method based on a visual language model, characterized in that, include: Step 1. Image Feature Extraction: Image features are extracted using the Vision Transformer (ViT) image encoder of the pre-trained CLIP (Contrastive Language-Image Pre-training) model. Frequency-Enhanced Forgery-Aware Adapter (FEFA) is inserted at layers 8, 16, and 24 of the image encoder to mine forgery features from both the image and frequency domains. The extracted features are then fused with the image features output by CLIP through a weighted residual fusion via an adapter network, resulting in image features with stronger discriminative capabilities. Step 2. Manual feature extraction: A manual feature extraction module (human-designed feature extraction) based on image statistical characteristics and human vision inspiration is proposed. Discriminative low-dimensional features are extracted from four dimensions of image: edge, texture, frequency and symmetry. These features are used as auxiliary information input in the forgery detection process to improve the robustness and interpretability of the model. Step 3. Feature fusion: The image features obtained in Step 1 and the handcrafted features obtained in Step 2 are combined by the feature fusion module, and then subjected to layer normalization and linear transformation to achieve effective fusion of features from different modalities. Step 4. Text features: The input text is transformed into feature vectors using the text encoder of the CLIP model; Step 5. Image-text alignment: Predict whether an image is a forgery by calculating the cosine similarity between the fused features obtained in Step 3 and the text features obtained in Step 4.

2. The deepfake detection method based on a visual language model according to claim 1, characterized in that: The frequency domain enhanced forgery perception adapter is a dual-branch structure in the image domain and frequency domain, which captures forgery traces from different modalities, significantly improving the method's ability to identify forged images. The image domain employs a lightweight structure consisting of two one-dimensional convolutions and a nonlinear activation function, used to capture low-level forgery features in the image. The frequency domain module includes a frequency domain detail enhancement module, a frequency band attention module, and a feedforward neural network module. It acquires multi-scale frequency domain features through DWT (wavelet transform) and enhances hidden frequency domain anomalies in the forged image using the frequency band attention mechanism. The feedforward neural network module further refines and enhances the semantic expressive power of the multi-frequency fusion features at the end of the frequency domain enhancement module through nonlinear transformation and residual normalization, improving the separability and representation stability of the forged region. Finally, it restores the image domain through IDWT (inverse wavelet transform) and weightedly fuses it with image domain features to obtain the output of the adapter module. The specific formula for the frequency domain detail enhancement module is as follows: f enh =Conv 1×1 (ReLU(G-Conv 3×3 (f dwt ))) Among them, f dwt ∈R B×C×H×W This represents the frequency domain features obtained by concatenating the four frequency bands (LL, LH, HL, HH) obtained from DWT. G-Conv represents a grouped convolution with 8 groups, used to model local forgery patterns and improve the expressive power of edge and high-frequency regions. enh This indicates the enhanced features. The specific formula for the frequency band attention module is as follows: w c =Softmax(MLP(LN(MeanPool(f c )))),c∈{LL,LH,HL,HH} Among them, f c w represents the characteristics of the c-th frequency band. c This corresponds to its attention weight. Weight w c This method uses weighted feature aggregation across four frequency bands for fusion, effectively improving the model's focus on frequency regions with more forged information. MeanPool represents average pooling, LN represents layer normalization, and MLP represents multilayer perceptron, used for feature mapping and learning complex nonlinear relationships. Softmax represents the normalized exponential function.

3. The deepfake detection method based on a visual language model according to claim 1, characterized in that: The adapter network is a fully connected layer structure. By nonlinearly adjusting the image features output by CLIP, its adaptability in deepfake detection tasks is improved.

4. The deepfake detection method based on a visual language model according to claim 1, characterized in that: The manual feature extraction module includes an edge intensity analysis module, a texture consistency analysis module, a symmetry analysis module, a spectral energy analysis module, and a high-frequency information analysis module, which respectively extract low-dimensional statistical features of the image in spatial structure and frequency domain distribution to assist the model in perceiving local and global anomalies in the image. The edge strength analysis module uses the Laplacian operator to enhance the edges of the input image and uses the average edge response of the entire image as the edge strength index. The specific formula is: f edge =E[|Laplacian(I)|] Among them, f edge Let I represent the edge intensity feature, I represent the input grayscale image, Laplacian(·) represents the convolution operation of applying the two-dimensional Laplacian operator to the image, and E[·] represents the average value of all elements. The texture consistency analysis module evaluates image texture consistency from two perspectives: texture contrast and local homogeneity. Texture contrast is based on the horizontal gray-level difference, reflecting the intensity changes of the image in local directions. The specific formula is: f contrast =E[|I(x,y+1)-I(x,y)|] Among them, f contrast This represents the texture contrast feature, where x represents the column coordinate of the image and y represents the row coordinate of the image. Local homogeneity is calculated by using a sliding window to determine the pixel standard deviation of a local region and then modeling its smoothness using an exponential function. The specific formula is: f homo =E[exp(-σ local )] Among them, f homo σ represents the characteristic of local homogeneity. local This represents the standard deviation of pixel grayscale within the sliding window. The symmetry analysis module calculates the mean square error between the original image and its horizontally flipped image as an indicator of symmetry violation. The specific formula is: f sym =E[(I(x,y)-I flip (x,y)) 2 ] Among them, f sym Indicating symmetry, I flip This represents the image after it has been horizontally flipped. The spectral energy analysis module performs a Fast Fourier Transform (FFT) on the input grayscale image and uses the logarithmic mean of the spectral amplitude as the frequency domain energy feature. The specific formula is as follows: f freq =E[log(|FFT shift (I)|+ε)] Among them, f freq FFT represents the spectral energy characteristics. shift (I) indicates that a centering operation is applied to the Fourier transform result, shifting the low-frequency components in the spectrum to the center position to enhance the symmetry and interpretability of the spectral distribution. ε = 1e-5 is a small constant to prevent numerical instability. The high-frequency information analysis module uses the absolute difference between the original image and its Gaussian blurred image as a measure of the degree of high-frequency detail preservation. The specific formula is: f highfreq =E(|I-G σ (I)|) Among them, f highfreq G represents high-frequency information features. σ (I) represents the blurred image obtained by convolving the input grayscale image with a two-dimensional Gaussian kernel of size 5×5 with a standard deviation σ = 1.

5.

5. The deepfake detection method based on a visual language model according to claim 1, characterized in that: The feature fusion module adopts a layer normalization and linear transformation structure to map the stitched image features and hand-crafted features to a unified dimension, so as to achieve effective fusion and unified expression of cross-modal features.

6. The deepfake detection method based on a visual language model according to claim 1, characterized in that: The text features consist of fixed prompts "real" and "fake," which are used to represent real and fake images, respectively, guiding the visual language model to perform binary classification semantic alignment.

Images