Image forgery detection method and device based on CLIP model
By constructing a multimodal tampering dataset and fine-tuning the CLIP model, and combining frequency domain information and multi-scale feature extraction, the generalization inadequacy of existing image forgery detection models and the difficulty in identifying forgery methods are solved. This achieves efficient localization of multi-category forgery regions and detection of splicing traces, and improves the interpretability and accuracy of detection results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2025-07-22
- Publication Date
- 2026-05-15
AI Technical Summary
Existing image forgery detection models lack generalization ability in real-world scenarios, cannot identify specific forgery methods, and have difficulty locating splicing traces, resulting in a lack of interpretability in the detection results.
A multimodal tampering dataset is constructed. The CLIP model is fine-tuned using LoRA, and feature fusion is performed by combining the frequency domain information of the image to achieve multi-class forgery region localization and splicing trace detection. The low and high frequency maps of the input image are decomposed using discrete cosine transform, and multi-scale features are extracted using SegFormer and ConvNext networks. The detection results are optimized by combining cross attention and loss function.
It improves the model's detection performance in real-world scenarios, enabling it to identify various forgery methods and locate pixel-level splicing traces, providing a complete chain of evidence of tampering and enhancing the security of digital media.
Smart Images
Figure CN120997613B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image forgery detection technology, specifically relating to a method and device for image forgery feature learning and splicing trace detection based on CLIP model. Background Technology
[0002] CLIP, as a widely used multimodal representation model, can support downstream tasks such as image classification and segmentation with greater generalization capabilities. However, CLIP mainly focuses on the high-level semantics of images and lacks sensitivity to pixel-level forgery features, making it unsuitable for direct use in forgery detection and representing a significant research gap.
[0003] Currently, image forgery detection models are mainly based on CNN and ViT models. However, when a user inputs a new type of forged image, the model performs poorly. Furthermore, existing methods generally only achieve authentication and forgery region localization, but cannot support the detection of forgery traces and methods, making it difficult to support effective tamper evidence collection. In other words, existing methods mainly have the following limitations:
[0004] 1) Insufficient generalization in real-world scenarios
[0005] In the field of image forgery detection, insufficient generalization manifests as models performing well only on the training set but poorly on the test set or in real-world scenarios, significantly reducing their effectiveness in practical applications. This is because most current image forgery detection methods rely on training small neural networks, learning low- or mid-level features (such as edge discontinuities or local noise variations) from data to determine whether an image has been tampered with. However, this approach means the model only learns the data distribution characteristics of the training set, making it difficult to generalize when dealing with unseen forgery samples, especially given the rapid evolution of forgery techniques.
[0006] 2) Lack of identification of counterfeiting methods
[0007] Existing methods mostly employ a binary classification framework, meaning the model only performs real / fake identification and cannot distinguish specific forgery methods (such as GAN generation, image stitching, content erasure, etc.). For example, in Deepfake detection, the model can only determine whether a face image is real, but cannot differentiate between tampering implemented by different generators such as StyleGAN and StarGAN. This coarse-grained discrimination result cannot construct a complete chain of evidence of tampering.
[0008] 3) Unable to locate splicing marks
[0009] Mainstream forgery detection methods use binary masks to mark suspicious areas, but they cannot analyze specific splicing and tampering traces. Specifically, existing models can locate forged areas, but cannot accurately pinpoint the physical traces of edge transitions. This results in a lack of interpretability in the detection results, making it difficult to provide pixel-level evidence of tampering for image forgery detection. Summary of the Invention
[0010] To address the aforementioned problems, we propose a method and device for image forgery feature learning and splicing trace detection based on the CLIP model. The key feature is the fine-tuning of CLIP to achieve pixel-level forgery feature learning by the CLIP model, and the execution of multi-class forgery region localization and splicing trace detection after user input image. The method includes the following steps:
[0011] Step 1: Construct a multimodal image manipulation dataset. This involves collecting real and forged image samples to build a training set. The forged images include various types such as spliced images and AI-generated images. Add a text description to each image, including the object being manipulated, the parameters being manipulated, and the method being manipulated.
[0012] Step 2: Using a multimodal tampering dataset, CLIP is fine-tuned using Low-Rank Adaptation (LoRA) to enhance its pixel-level forgery feature extraction capabilities. Through image-text pair fine-tuning, the CLIP model learns the differential representations of tampered and real images in the feature space.
[0013] Step 3: The low-frequency map of the input image is decomposed using discrete cosine transform. Based on the CLIP model parameters obtained in step 2, the feature map of the fine-tuned CLIP visual encoder is fused with the low-frequency features, and the multi-class forgery localization result is output through the decoder.
[0014] Step 4: The high-frequency map of the input image is decomposed by discrete cosine transform. The feature map of the CLIP visual encoder in Step 2 is fused with the high-frequency features, and the pixel-level stitching trace prediction map is output by the decoder.
[0015] Preferably, the eight types of forged images include traditionally stitched and tampered images, face-tampered images generated by StarGAN, complete fake images generated by StyleGAN, cross-domain style transfer images generated by CycleGAN, images modified by the Inpainting algorithm, and synthetic images generated by Stable Diffusion.
[0016] Preferably, in step 2, a LoRA layer is inserted into each Transformer block of the CLIP image encoder and text encoder for fine-tuning.
[0017] Preferably, the cross-attention fusion calculation in steps 3 and 4 is as follows:
[0018]
[0019] Q is the query vector, which is the minimum scale low-frequency feature output by SegFormer in step 3 and the minimum scale high-frequency feature output by ConvNext in step 4; K and V are key vectors, both of which are fake features output by TamperCLIP.
[0020] Preferably, in step 4, a random sampling strategy is used to construct a set of positive and negative samples, and the splicing trace prediction mask is optimized by InfoNCE loss.
[0021] Preferably, in step 3, the weighted cross-entropy loss function is used to optimize the localization results of multi-class forged regions; in step 4, the Dice loss is used to optimize the splicing trace prediction mask.
[0022] Preferably, in steps 3 and 4, the weight map is generated by a weighting module, which includes a convolutional input layer, a feature inversion layer, and a convolutional layer.
[0023] Preferably, the calculation process of the weighted graph generation module is as follows:
[0024]
[0025] in, It is the original input image. It is a low-frequency image extracted by DCT. This is a high-frequency image extracted using DCT. Conv represents a convolutional layer, and LayerNorm represents layer normalization.
[0026] Preferably, in step 3, the multi-class forged region prediction map is ultimately generated by a multi-class classification head, which includes a convolutional input layer, group normalization, and a GELU activation function.
[0027] Based on the same inventive concept, the present invention also discloses an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform an image forgery detection method based on the CLIP model.
[0028] Based on the same inventive concept, the present invention also discloses a computer-readable medium having a computer program stored thereon, characterized in that: when the program is executed by a processor, it implements an image forgery detection method based on the CLIP model.
[0029] The advantages of this invention are as follows: By constructing a multimodal tampering dataset, using LoRA fine-tuning to endow the CLIP model with the ability to encode pixel-level forgery features, and combining image frequency domain information, feature fusion is performed to achieve multi-category forgery region localization and splicing trace detection. This invention effectively improves the problems of existing forgery detection methods' insufficient utilization of the CLIP model, insufficient generalization of existing technologies, inability to identify specific forgery methods, and difficulty in locating splicing traces. By developing a high-performance forgery detection technology, this invention provides strong protection for maintaining the security of digital media. Attached Figure Description
[0030] Figure 1 : Flowchart of an embodiment of the present invention.
[0031] Figure 2 The data distribution of the multimodal tampering dataset in this embodiment of the invention.
[0032] Figure 3 : An embodiment diagram of the LoRA fine-tuning CLIP of this invention.
[0033] Figure 4 : A network architecture diagram of a multi-category fake region positioning system according to an embodiment of the present invention.
[0034] Figure 5 : Network architecture diagram for splicing trace detection in an embodiment of the present invention. Detailed Implementation
[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0036] Example 1
[0037] This embodiment discloses an image forgery detection method based on the CLIP model, including the following steps:
[0038] Step 1: Construct a multimodal image manipulation dataset. This involves collecting real and forged image samples to build a training set. The forged images include various types such as spliced images and AI-generated images. A text description containing information about the object being manipulated, the parameters being manipulated, and the method being manipulated is added to each image. The specific process is as follows:
[0039] A multimodal image manipulation dataset was constructed, comprising real images and eight types of forged images. The forged images were generated using specific algorithms, including traditional image stitching manipulation, face manipulation images generated using StarGAN, complete fake images generated using StyleGAN, images with cross-domain style transfer based on CycleGAN, forged images modified using the inpainting algorithm, and synthetic images using Stable Diffusion technology. Real images are associated with a textual description of "real image - semantic content in the image," while forged images are associated with a textual description of "manipulation method - manipulation parameters - location of the manipulated region." This means the textual description precisely annotates the manipulation method, the corresponding manipulation parameters, and the manipulated region with a location description. The final multimodal image manipulation dataset contains 18,000 image-text pairs. The dataset distribution is as follows: Figure 2 As shown.
[0040] Step 2: Fine-tune CLIP using a multimodal tampering dataset to enhance its pixel-level forgery feature extraction capabilities. Through image-text pairing fine-tuning, the CLIP model learns the differential representations of tampered and genuine images in the feature space. The specific process is as follows:
[0041] Based on the multimodal tampering dataset constructed in step 1, the CLIP model is fine-tuned using LoRA to enhance its pixel-level forgery feature extraction capability. Specifically, LoRA layers are inserted into each Transformer block of the CLIP image encoder and text encoder for fine-tuning. A contrastive learning strategy is employed, aligning the tampered image with its corresponding text description in the feature space while rejecting non-matching combinations. A contrastive loss function is constructed for the image-text pairs in each training batch. The final result is a TamperCLIP model capable of encoding forgery features. The specific process is as follows: Figure 3 As shown.
[0042] Step 3: Implement multi-class forgery region localization. This process includes the following modules: a Discrete Cosine Transform (DCT) module, which takes the image to be detected as input and outputs the corresponding high-frequency and low-frequency maps. In this step, the low-frequency map is concatenated with the original image along the channel dimension to obtain an enhanced low-frequency input; a SegFormer backbone network, which takes the enhanced low-frequency input as input and outputs low-frequency feature maps at four scales; a TamperCLIP module, which takes the image to be detected as input and outputs the forgery features of the image; a feature fusion module, which takes the smallest-scale low-frequency features and forgery features as input; and a decoding module, which includes a weighting module and a multi-class classification head. The network structure is as follows: Figure 4 As shown. The specific process is as follows:
[0043] The low-frequency map of the input image is decomposed using Discrete Cosine Transform (DCT) and concatenated with the original image along the channel dimension to obtain an enhanced low-frequency input. This enhanced low-frequency input is then fed into the SegFormer network to obtain low-frequency feature maps at four scales: 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size. In forgery detection tasks, since forgery traces may appear at different scales, features at different scales can complement each other. Small-scale features contain rich global contextual semantic information, which helps the model understand whether an image is forged, while large-scale features contain rich spatial details and edge information, which helps the model locate forged regions.
[0044] The SegFormer network is based on the Transformer architecture and has powerful multi-scale feature extraction capabilities and the ability to capture long-range dependencies and global contextual information. Therefore, the SegFormer network is preferred in this step.
[0045] For feature fusion, the lowest-frequency features at the smallest scale are cross-attentioned with the fake features generated by TamperCLIP. The query is the lowest-frequency feature at the smallest scale, and the key and value use the fake features generated by TamperCLIP. The cross-attention operation is represented as follows:
[0046]
[0047] Where Q is the query vector: in step 3, it is the minimum-scale low-frequency feature output by SegFormer; in step 4, it is the minimum-scale high-frequency feature output by ConvNext. K and V are key-value vectors, both being forged features output by TamperCLIP. It is the transpose of the key vector K; It is the dimension of the key vector, used to scale the dot product result. The purpose is to prevent the dot product result from becoming too large, which would cause the softmax gradient to vanish, that is, to prevent the gradient saturation phenomenon from occurring.
[0048] Finally, low-frequency features at four scales are decoded, and an initial prediction mask is obtained by summing the weighted map multiplied by the feature map generated by SegFormer. This initial prediction mask is then input into a multi-class classification head to generate multi-class forgery region localization results. The final output is a pixel-level multi-class forgery region prediction map, with different colors marking different forgery categories.
[0049] The weight map is generated by a weighting module, which includes a 7×7 convolutional input layer, a feature inversion layer, and a 7×7 convolutional layer. The operation process of the weight map generation module is as follows:
[0050]
[0051] in, It is the original input image. It is a low-frequency image extracted by DCT. This is the high-frequency image extracted via DCT. Conv is a convolutional layer, and LayerNorm is a layer normalization layer. Specifically, the input to the weight map generation module is the original image, the high-frequency image, and the low-frequency image. Figure 3 The overall input, concatenated along the channel dimension, is fed into a 7×7 convolutional layer for preliminary feature extraction and fusion, producing an initial feature map W. temp It captures both global and local information from the overall input. Then, to further refine and enhance the features, W is first processed through the residual path. temp The application layer normalizes the data to stabilize the feature distribution. Then, a 3×3 convolutional layer is used to extract finer local features. Two consecutive 1×1 convolutional layers are then used to fuse information between channels and transform the dimensions. Finally, a feature map W is output after passing through the GELU activation function. res Subsequently, the module will use the initial feature W temp and feature W res Residual connections are performed, and the fused features are then passed through another 7×7 convolutional layer for final global feature integration and mapping. Finally, the Softmax function is applied for normalization to ensure that the sum of the weights of all channels at each spatial location is 1, thus generating the final weight map W. This weight map has eight channels; the first four channels are used for stitching artifact detection, and the last four channels are used for multi-class forgery region localization.
[0052] The multi-class classification head consists of two repetitive layers (convolutional layer, normalization layer, and activation function) and a convolutional output. The operation process of the multi-class classification head is as follows:
[0053]
[0054]
[0055]
[0056] Where H and W represent the height and width of the input image, respectively, W1 and W2 are intermediate variable names, and the final output W... outThis is a multi-class forgery region localization map, a fine-grained prediction map where each pixel carries the multi-class classification result, used to locate and identify different types of forgery regions in an image. `GroupNorm` is a group normalization operation. Specifically, the classification head receives an initial prediction mask with 64 channels as input, generated by operations on a weight map and four SegFormer-scale low-frequency features. The entire process is as follows: First, a 3×3 convolution is used to increase the number of channels from 64 to 128, combined with group normalization and the GELU activation function to enhance feature representation; then, a second 3×3 convolution further deepens feature extraction, maintaining 128 channels while enhancing spatial feature representation; finally, a 1×1 convolution is used to compress the number of channels to the number of forgery categories. The entire process maintains the same spatial resolution of the input and output, ultimately generating a multi-class forgery region prediction map of the same size as the initial prediction mask, with each channel representing the original prediction score for one forgery category. The application performs Softmax normalization on the output and generates a pixel-level multi-class prediction map through argmax operation. Different colors are used to intuitively mark regions of different forgery types, realizing visualized multi-class forgery region localization.
[0057] During training, a weighted cross-entropy loss function is used. To mitigate the class imbalance caused by an excessively high proportion of real background pixels, a weighted strategy is employed to correct the cross-entropy loss, thereby improving the model's attention to fake regions. The weighted cross-entropy loss is calculated as follows:
[0058]
[0059] Where N is the number of pixels; This is the true category label for that pixel; The model predicts that the pixel belongs to its true category. The probability. If it is a real pixel, its weight. Setting it to 0.1 reduces the loss contribution to 10%, making the model less sensitive to its classification errors. If the class is a fake, its weight... Set the value to 1.0 to maintain the original loss contribution.
[0060] Step 4: Implement stitching trace detection. This process includes the following modules: a Discrete Cosine Transform (DCT) module, which takes the image to be detected as input and outputs the corresponding high-frequency and low-frequency maps. In this step, the high-frequency map is stitched with the original image along the channel dimension to obtain an enhanced high-frequency input; a ConvNext backbone network, which takes the enhanced high-frequency input as input and outputs high-frequency feature maps at four scales; a TamperCLIP module, which takes the image to be detected as input and outputs the forgery features of the image; a feature fusion module, which takes the minimum-scale high-frequency features and forgery features as input; and a decoding module using a weighted module. The network structure is as follows. Figure 5As shown. The specific process is as follows:
[0061] The high-frequency map of the input image is decomposed by discrete cosine transform and concatenated with the original image in the channel dimension to obtain an enhanced high-frequency input. The enhanced high-frequency input is then fed into the ConvNext network to obtain feature maps at four scales: 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size.
[0062] ConvNext, based on a convolutional structure, has a good ability to capture high-frequency details in images, such as edges and textures, which is crucial for detecting stitching artifacts. Therefore, the ConvNext network is preferred in this step.
[0063] For feature fusion, the minimum-scale high-frequency features are cross-attentioned with the fake features generated by TamperCLIP. The query is the minimum-scale high-frequency feature, and the key and value use the fake features generated by TamperCLIP.
[0064] Simultaneously, the highest-frequency features at the largest scale are extracted, and the boundary mask is sampled to the size of the feature map and projected to obtain the positions of positive and negative samples. A positive and negative sample set is constructed using a random sampling strategy, and further optimization is performed using InfoNCE loss. Finally, the features at the four scales are decoded, and the splicing trace prediction mask is obtained by multiplying the weight map by the feature map and then summing the results. The weight map is generated by inputting the tampered image and high- and low-frequency images into a weighted module and performing convolution calculations.
[0065] During training, Dice loss was also used for optimization, which enhances the model's detection accuracy for subtle tampering edges by maximizing the overlap between the splicing trace prediction segmentation map and the real label.
[0066]
[0067] Where X represents the predicted stitching region, Y represents the actual stitching region, and X∩Y represents the number of pixels at the intersection of the predicted and actual regions. Dice loss is used to improve the model's ability to detect stitching artifacts.
[0068] It should be understood that the SegFormer network used in step 3 can be replaced by other suitable backbone networks. The replacement network needs to possess the following characteristics: strong feature extraction capabilities, able to understand the semantic content of the image and capture information at different scales; multi-scale output features, the network must be able to output multiple feature maps at different scales to cover the different scale requirements in the forgery detection task. The replacement network also needs to be able to handle irregular input channel numbers. In this invention, the input is an enhanced low-frequency input, i.e., the result of concatenating the original image and the DCT low-frequency image in the channel dimension, with 6 input channels.
[0069] It should be understood that the ConvNext network used in step 4 can be replaced by other suitable backbone networks. The replacement network needs to have the following characteristics: multi-scale feature extraction capability, the replacement network should be able to extract and output feature maps of different scales; strong feature representation capability, the network itself needs to be a powerful feature extractor, able to effectively learn rich features from the input for subsequent tamper detection tasks; the replacement network also needs to be able to handle irregular input channel numbers. The input of this invention is an enhanced high-frequency input, that is, the result of concatenating the original image and the DCT high-frequency image in the channel dimension, with 6 input channels.
[0070] Example 2
[0071] Based on the same inventive concept, the present invention also discloses an electronic device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform an image forgery detection method based on the CLIP model.
[0072] Since the device described in Embodiment 2 of this invention is the same electronic device used to implement the image forgery detection method based on the CLIP model in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.
[0073] Example 3
[0074] Based on the same inventive concept, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.
[0075] Since the device described in Embodiment 3 of this invention is a computer-readable medium used to implement the image forgery detection method based on the CLIP model in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.
[0076] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. An image forgery detection method based on the CLIP model, characterized in that, Includes the following steps: Step 1: Construct a multimodal tampering dataset, which contains real images and various forged images. Add a text description to each forged image, including the object to be tampered with, the tampering parameters, and the tampering method. Step 2: Fine-tune the CLIP model using the multimodal tampering dataset. Insert low-rank adaptive LoRA layers into each Transformer block of the CLIP model's image encoder and text encoder for fine-tuning. The fine-tuning process uses a contrastive learning strategy to jointly optimize the visual encoder and text encoder. Align the tampered image and text description through a symmetric contrastive loss function to obtain a TamperCLIP model with the ability to encode forgery features. Step 3: Perform discrete cosine transform on the input image to extract low-frequency components, and fuse these low-frequency components with the real image to generate enhanced low-frequency input; extract the multi-scale feature map of the enhanced low-frequency input, and perform cross-attention calculation using minimum-scale features and TamperCLIP spoofing features; The cross-attention calculation result is fused with the multi-scale feature map, and an initial prediction mask is generated by weighted summation of the weight map. Then, a multi-class classification head is used to generate multi-class fake region localization results. The multi-class classification head includes a convolutional input layer, group normalization, and GELU activation function. Step 4: Perform discrete cosine transform on the input image to extract high-frequency components and fuse them with the original image to generate enhanced high-frequency input; use a convolutional neural network to extract multi-scale feature maps of the enhanced high-frequency input, and perform cross-attention calculation using minimum-scale features and TamperCLIP spoofing features; Based on the maximum scale features, a contrast loss is calculated using a random sampling strategy to generate a pixel-level splicing trace prediction mask, which is used to locate the edge traces of splicing and tampering in the image.
2. The image forgery detection method based on the CLIP model according to claim 1, characterized in that: The forged images include traditionally stitched and tampered images, face-tampered images generated by StarGAN, complete fake images generated by StyleGAN, cross-domain style transfer images generated by CycleGAN, images modified by the Inpainting algorithm, and synthetic images generated by Stable Diffusion.
3. The image forgery detection method based on the CLIP model according to claim 1, characterized in that: The cross-attention fusion calculation in steps 3 and 4 is as follows: Q is the query vector. In step 3, the minimum scale low-frequency features of the network output are extracted using SegFormer multi-scale features. In step 4, the minimum scale high-frequency features of the network output are extracted using ConvNext multi-scale features. K and V are key vectors, both of which are fake features output by TamperCLIP.
4. The image forgery detection method based on the CLIP model according to claim 1, characterized in that: In step 4, a random sampling strategy is used to construct a set of positive and negative samples, and the splicing trace prediction mask is optimized by InfoNCE loss.
5. The image forgery detection method based on the CLIP model according to claim 1, characterized in that: In steps 3 and 4, the weighted graph is generated by a weighting module. The calculation process for generating the weighted graph by the weighting module is as follows: in, It is the original input image. It is a low-frequency image extracted by DCT. It is a high-frequency image extracted by DCT, Conv is a convolutional layer, and LayerNorm is a layer normalization.
6. An electronic device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the image forgery detection method based on the CLIP model as described in any one of claims 1-5.
7. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the image forgery detection method based on the CLIP model as described in any one of claims 1-5.