Image forgery detection method and device based on CLIP model

Abstract

The invention discloses an image forgery detection method and device based on a CLIP model. The method comprises the steps that firstly, a multi-mode tampering data set containing multiple forgery types and corresponding text descriptions is constructed; the CLIP model is finely adjusted through LoRA by using the data set, so that the CLIP model learns pixel-level counterfeit features, and the characterization capability of the difference between a tampered image and a real image is enhanced; the method comprises the following steps of: decomposing low-frequency and high-frequency components of an input image by discrete cosine transform; respectively fusing the fine-tuned characteristic patterns of the CLIP visual encoder with low-frequency characteristics, and outputting a multi-category forged region positioning result through a decoder; meanwhile, the pixel-level splicing trace prediction image is fused with the high-frequency features, and a pixel-level splicing trace prediction image is output through a decoder. According to the method, the generalization ability of the model in a real scene is effectively improved, identification of specific counterfeiting means is realized, edge traces of splicing tampering are accurately revealed, and more comprehensive technical support is provided for image tampering evidence obtaining.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image forgery detection, and particularly relates to an image forgery feature learning and splicing trace detection method and device based on a CLIP model. BACKGROUND

[0002] As a widely used multi-modal representation model, CLIP can support downstream tasks such as image classification and segmentation with more generalization ability. However, CLIP mainly focuses on the high-level semantics of images, and is not sensitive enough to pixel-level forgery features, so it cannot be directly used for forgery detection, and there is a significant research gap.

[0003] At present, image forgery detection models are mainly implemented based on CNN and ViT models. When a user inputs a new type of forged image, the model performs poorly. At the same time, existing methods generally only implement true-false discrimination and forgery region positioning, but cannot support forgery trace and forgery method detection, making it difficult to support effective tampering forensics. That is, the existing methods mainly have the following limitations: 1) Lack of generalization in real scenes The lack of generalization in the field of image forgery detection is that the model only performs well on the training set, and performs poorly on the test set or in real scenes, resulting in a significant reduction in actual application effectiveness. This is because most current image forgery detection methods rely on small neural networks for training, which learn low-level or intermediate features (such as edge discontinuity or local noise changes) from data to determine whether an image has been tampered with. However, such an approach will cause the model to only learn the data distribution characteristics of the training set, which makes it difficult to generalize when dealing with unseen forgery samples, especially in the face of rapidly changing forgery technologies.

[0004] 2) Lack of forgery method identification Existing methods mostly use a binary classification framework, i.e., the model only performs true-false discrimination and cannot identify specific forgery methods (such as GAN generation, image splicing, content erasure, etc.). For example, in Deepfake detection, the model can only determine whether a face image is real, but cannot distinguish between different generators such as StyleGAN and StarGAN. Such coarse-grained discrimination results cannot build a complete tampering evidence chain.

[0005] 3) Unable to locate splicing traces Mainstream forgery detection methods use a binary mask to mark suspicious areas, but cannot analyze specific splicing tampering trace features. Specifically, existing models can locate forgery regions, but cannot accurately locate the physical traces of edge transition zones. This results in a lack of explainability in the detection results, making it difficult to provide pixel-level tampering evidence for image forgery detection. SUMMARY

[0006] To solve the above problems, we propose a method and device for image forgery feature learning and splicing trace detection based on the CLIP model. The method includes the following steps: Step 1, construct a multi-modal tampering dataset, collect real images and forged image samples to build a training set, where the forged images include splicing tampered images, AI generated images and other types. Add text descriptions containing tampered objects, tampered parameters, tampering means, etc. to each image.

[0007] Step 2, use the multi-modal tampering dataset to fine-tune CLIP using Low-Rank Adaptation (LoRA), and strengthen the pixel-level forgery feature extraction capability. Fine-tune the CLIP model by text and image to learn the difference between tampered images and real images in the feature space.

[0008] Step 3, decompose the low-frequency image of the input image using discrete cosine transform, and based on the CLIP model parameters obtained in step 2, fuse the feature map of the fine-tuned CLIP visual encoder with the low-frequency feature, and output the multi-class forgery positioning result through the decoder.

[0009] Step 4, decompose the high-frequency image of the input image using discrete cosine transform, and fuse the feature map of the CLIP visual encoder in step 2 with the high-frequency feature, and output the pixel-level splicing trace prediction map through the decoder.

[0010] Preferably, the eight types of forged images include traditional splicing tampered images, StarGAN generated face tampered images, StyleGAN generated complete false images, CycleGAN cross-domain style transfer images, Inpainting algorithm modified images, and Stable Diffusion synthesized images.

[0011] Preferably, in step 2, LoRA layers are inserted in each Transformer block of the image encoder and text encoder of CLIP for fine-tuning.

[0012] Preferably, the cross-attention fusion calculation in steps 3 and 4 is:

[0013] Q is the query vector, which is the smallest scale low-frequency feature output by SegFormer in step 3, and the smallest scale high-frequency feature output by ConvNext in step 4; K and V are key and value vectors, which are both the forgery features output by TamperCLIP.

[0014] Preferably, in step 4, a random sampling strategy is used to construct the positive and negative sample set, and the splicing trace prediction mask is optimized by InfoNCE loss.

[0015] Preferably, in step 3, a weighted cross-entropy loss function is used to optimize the multi-class fake region positioning result; in step 4, a Dice loss is used to optimize the splicing trace prediction mask.

[0016] Preferably, in steps 3 and 4, the weight map is generated by a weighting module, which includes a convolution input layer, a feature inversion layer, and a convolution layer.

[0017] Preferably, the operation process of the weight map generation module is as follows:

[0018] wherein, is the original input image, is the low-frequency graph extracted by DCT, is the high-frequency graph extracted by DCT. Conv is a convolution layer, and LayerNorm is a layer normalization.

[0019] Preferably, in step 3, the multi-class fake region prediction map is finally generated by a multi-class classification head, and the class classification head includes a convolution input layer, a group normalization, and a GELU activation function.

[0020] Based on the same inventive concept, the present application also discloses an electronic device, comprising a memory and a processor, which are mutually connected in communication, the memory stores computer instructions, and the processor executes the computer instructions to perform the image forgery detection method based on the CLIP model.

[0021] Based on the same inventive concept, the present application also discloses a computer readable medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the image forgery detection method based on the CLIP model.

[0022] The present application has the advantages that by constructing a multi-modal tampering dataset, using LoRA fine-tuning to endow the CLIP model with the ability to encode pixel-level forgery features, and combining the frequency domain information of the image, feature fusion is performed to realize multi-class fake region positioning and splicing trace detection. The present application can effectively improve the problem that the existing forgery detection method does not fully utilize the CLIP model, the existing technology has insufficient generalization, cannot identify specific forgery means, and is difficult to locate the splicing trace. By developing an excellent forgery detection technology, the present application provides a strong guarantee for maintaining the security of digital media. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 Flowchart of embodiments of the present application.

[0024] Figure 2 Data distribution of the multi-modal tampered dataset of embodiments of the present application.

[0025] Figure 3 Implementation diagram of LoRA fine-tuning CLIP of embodiments of the present application.

[0026] Figure 4 Multi-class fake region positioning network architecture diagram of embodiments of the present application.

[0027] Figure 5 Network architecture diagram of splicing trace detection of embodiments of the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0029] Embodiment one The present embodiment discloses a CLIP model-based image forgery detection method, comprising the following steps: Step 1, construct a multi-modal tampered dataset, collect real images and fake image samples to build a training set, wherein the fake images include splicing tampered images, AI generated images and other types. Add text description containing tampered objects, tampered parameters, tampered means, etc. to each image, and the specific process is as follows: A multi-modal tampered dataset containing real images and eight types of fake images is constructed, wherein the fake images are generated by a specific algorithm, including traditional splicing tampered images, face tampered images generated using StarGAN, complete false images generated using StyleGAN, cross-domain style transfer images based on CycleGAN, fake images modified by inpainting algorithm, and synthetic images applying Stable Diffusion technology. The real image association structure is a text description such as "real image-image semantic content", while the fake image association structure is a text description such as "tampering method-tampering parameter-tampering region orientation", i.e. the tampering means, corresponding tampering operating parameters, and tampering region with orientation description in the text description. Finally, a multi-modal tampered dataset containing 18,000 image-text pairs is constructed. The dataset distribution is shown in Figure 2 .

[0030] Step 2, fine-tune CLIP using the multi-modal tampering dataset to enhance the ability to extract pixel-level fake features. Fine-tune by image-text pairs, so that the CLIP model learns the difference between tampered images and real images in the feature space. The specific process is as follows: According to step 1, the multi-modal tampering dataset is used to fine-tune the CLIP model using LoRA, which enhances its ability to extract pixel-level fake features. The specific implementation is to insert LoRA layers in each Transformer block of the image encoder and text encoder of CLIP for fine-tuning. The process uses a contrastive learning strategy to align the tampered images with their corresponding text descriptions in the feature space while excluding non-matching combinations. A contrastive loss function is constructed for each training batch of image-text pairs. Finally, TamperCLIP is obtained, which can encode fake features. The specific process is shown in Figure 3 .

[0031] Step 3, realize multi-class fake region positioning. This process includes the following modules: Discrete Cosine Transform (DCT) module, input is the image to be detected, output is the high-frequency image and low-frequency image corresponding to the image. In this step, the low-frequency image is used to enhance the low-frequency input by concatenating it with the original image in the channel dimension; SegFormer backbone network, input is the enhanced low-frequency input, output is four scales of low-frequency feature maps; TamperCLIP, input is the image to be detected, output is the fake feature of the image; feature fusion module, input is the smallest scale low-frequency feature and fake feature; decoding module, including weighting module and multi-class classification head. The network structure is shown in Figure 4 . The specific process is as follows: Discrete Cosine Transform is used to decompose the low-frequency image of the input image, and the enhanced low-frequency input is obtained by concatenating it with the original image in the channel dimension. The enhanced low-frequency input is input into the SegFormer network to obtain four scales of low-frequency feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image. In the fake detection task, since the fake traces may appear at different scales, different scale features can complement each other. Small-scale features contain rich global context semantic information, which helps the model understand whether the image is fake or not. Large-scale features contain rich spatial details and edge information, which help the model locate the fake region.

[0032] SegFormer network is based on Transformer architecture, which has strong multi-scale feature extraction capability and the ability to capture long-distance dependencies and global context information, so SegFormer network is used as the preferred network in this step.

[0033] For feature fusion, cross-attention is performed between the smallest scale low-frequency feature and the fake feature generated by TamperCLIP, where the query is the smallest scale low-frequency feature, and the key and value use the fake feature generated by TamperCLIP. The cross-attention operation process is represented as:

[0034] where Q is the query vector: in step 3, it is the smallest scale low-frequency feature output by SegFormer, and in step 4, it is the smallest scale high-frequency feature output by ConvNext. K and V are key and value vectors, which are both fake features output by TamperCLIP, is the transpose of the key vector K; is the dimension of the key vector, used to scale the dot product result, the purpose is to prevent the dot product result from being too large, causing the Softmax gradient to disappear, that is, to prevent the occurrence of gradient saturation.

[0035] Finally, the four scale low-frequency features are decoded, and the initial prediction mask is obtained by using the weight map to multiply the feature map generated by SegFormer and then summing. The initial prediction mask is input into the multi-class classification head to generate multi-class fake region positioning results. Finally, the pixel-level multi-class fake region prediction map is output, and different colors mark different fake categories.

[0036] The weight map is generated by a weighting module, which includes a 7x7 convolution input layer, a feature inversion layer, and a 7x7 convolution layer. The operation process of the weight map generation module is represented as:

[0037] wherein, is the original input image, is the low-frequency image extracted by DCT, is the high-frequency image extracted by DCT. Conv is a convolution layer, and LayerNorm is a layer normalization. Specifically, the input of the weight map generation module is the original image, the high-frequency image, and the low-frequency Figure Three image spliced in the channel dimension, and the overall input is sent to a 7x7 convolution layer for preliminary feature extraction and fusion to generate an initial feature map W temp , which captures global and local information from the overall input. Then, in order to further refine and enhance the features, the residual path is used to first apply layer normalization to W temp to stabilize the feature distribution, then extract more detailed local features through a 3x3 convolution layer, and then use two 1x1 convolution layers in succession for information fusion and dimension transformation between channels, and finally output a feature map W res through the GELU activation function. Subsequently, the module will use the initial feature Wtemp and feature W res The fused feature is connected with the residual, and then is input into another 7x7 convolution layer to integrate and map the global feature. Finally, the Softmax function is applied to normalize the weight values of all channels at each spatial position, so as to generate the final weight map W. The weight map has eight channels, and the first four channels of the final weight map are used for trace detection, and the last four channels are used for multi-class fake region positioning.

[0038] The multi-class classification head includes two layers of repeated structures composed of convolution layers, normalization layers, activation functions, and a convolution output. The operation process of the multi-class classification head is represented as:

[0039]

[0040]

[0041] wherein H and W represent the height and width of the input image respectively, W1 and W2 are intermediate variable names, and W is the final output. out The multi-class fake region positioning map is a fine prediction map with multi-class classification results for each pixel, which is used to locate and identify different types of fake regions in the image. GroupNorm is a group normalization operation. Specifically, the classification head receives an initial prediction mask with 64 channels as input, which is generated by the operation of the weight map and the four SegFormer scale low-frequency features. The entire processing process is represented as: first, use 3x3 convolution to increase the channel number from 64 to 128, and use group normalization and GELU activation function to strengthen feature representation; then, further deepen the feature extraction through the second 3x3 convolution, maintain 128 channels, and enhance the spatial feature expression ability; finally, use 1x1 convolution to compress the channel number to the number of fake classes. The entire process always maintains the spatial resolution of the input and output unchanged, and finally generates a multi-class fake region prediction map with the same size as the initial prediction mask, each channel representing the original prediction score of a fake class. The output is normalized by applying Softmax, and a pixel-level multi-class prediction map is generated by argmax operation, which directly marks different fake type regions with different colors, achieving visual multi-class fake region positioning.

[0042] During training, a weighted cross-entropy loss function is used to alleviate the class imbalance problem caused by the high proportion of real background pixels. A weighting strategy is used to correct the cross-entropy loss to improve the model's attention to fake regions. The weighted cross-entropy loss is calculated as follows:

[0043] where N is the number of pixels; is the true class label of the pixel; is the probability that the model predicts the pixel belongs to its true class If it is a real pixel, its weight takes 0.1, and the loss contribution is reduced to 10%, and the model is less sensitive to its classification error. If it is a fake class, its weight takes 1.0, keeping the original loss contribution.

[0044] Step 4, implement splice trace detection. This process includes the following modules: Discrete Cosine Transform (DCT) module, the input of which is the image to be detected, and the output is the high-frequency graph and low-frequency graph corresponding to the image. In this step, the high-frequency graph is used to splice the original image in the channel dimension to obtain an enhanced high-frequency input; ConvNext backbone network, input is the enhanced high-frequency input, output four scales of high-frequency feature map; TamperCLIP, input is the image to be detected, output is the fake feature of the image; feature fusion module, input is the smallest scale high-frequency feature and fake feature; decoding module uses a weighted module. The network structure is shown in Figure 5 . The specific process is as follows: The high-frequency graph of the input image is decomposed using Discrete Cosine Transform, and the original image is spliced in the channel dimension to obtain an enhanced high-frequency input. The enhanced high-frequency input is input into the ConvNext network to obtain four scale feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size.

[0045] ConvNext is based on convolutional structure, which has good capture ability for high-frequency details of images such as edges and textures, which is crucial for detecting splice traces. Therefore, ConvNext network is used as the preferred network in this step.

[0046] For feature fusion, the smallest scale high-frequency feature and the fake feature generated by TamperCLIP are used for cross-attention, where the query is the smallest scale high-frequency feature, and the key and value use the fake feature generated by TamperCLIP.

[0047] At the same time, the maximum scale high-frequency feature is taken, the boundary mask is sampled to the size of the feature map and projected to obtain the positions of the positive and negative samples, and the positive and negative sample sets are constructed through a random sampling strategy, and further optimized using InfoNCE loss. Finally, the four scale features are decoded, and the splice trace prediction mask is obtained by multiplying the weight map with the feature map and summing them up. The weight map is generated by inputting the tampered image and the high and low frequency images into the weighting module and performing convolution calculation.

[0048] Dice loss is also used for optimization during training, which strengthens the detection accuracy of the model for subtle tampering edges by maximizing the overlapping area between the splicing trace prediction segmentation graph and the real label.

[0049]

[0050] wherein X is the predicted splicing area, Y is the real splicing area, and X∩Y is the number of pixels of the intersection of the predicted area and the actual area. The dice loss is used to improve the detection ability of the model for splicing traces.

[0051] It should be understood that the SegFormer network used in step 3 can be replaced by other suitable backbone networks, and the replacement network needs to have the following characteristics: strong feature extraction capability, which can understand the semantic content of the image and capture information of different scales; output multi-scale features, the network must be able to output multiple feature maps of different scales to cover the different scale requirements in the forgery detection task. The replacement network also needs to be able to process irregular input channels, and the input of the present application is an enhanced low-frequency input, that is, the result of splicing the original image and the DCT low-frequency image in the channel dimension, and the input channel number is 6 channels.

[0052] It should be understood that the ConvNext network used in step 4 can be replaced by other suitable backbone networks, and the replacement network needs to have the following characteristics: multi-scale feature extraction capability, the replacement network should be able to extract and output feature maps of different scales; strong feature representation capability, the network itself needs to be a powerful feature extractor, which can effectively learn rich features from the input for subsequent tampering detection tasks; the replacement network also needs to be able to process irregular input channels, and the input of the present application is an enhanced high-frequency input, that is, the result of splicing the original image and the DCT high-frequency image in the channel dimension, and the input channel number is 6 channels.

[0053] Embodiment two Based on the same inventive concept, the present application also discloses an electronic device, comprising: a memory and a processor, which are mutually connected in communication, the memory stores computer instructions, and the processor executes the computer instructions to perform the image forgery detection method based on the CLIP model.

[0054] Since the device introduced in embodiment two of the present application is an electronic device used to implement the image forgery detection method based on the CLIP model of embodiment one of the present application, the specific structure and modifications of the electronic device can be understood by those skilled in the art based on the method introduced in embodiment one of the present application, and therefore will not be described here. Any electronic device used in the method of embodiment one of the present application belongs to the scope of protection of the present application.

[0055] Embodiment three Based on the same inventive concept, the application also provides a computer readable medium, which stores a computer program, and the program is executed by a processor to implement the method described in embodiment one.

[0056] Since the device introduced in the embodiment three of the application is the computer readable medium used to implement the image forgery detection method based on the CLIP model in the embodiment one of the application, the specific structure and deformation of the electronic device can be understood by those skilled in the art based on the method introduced in the embodiment one of the application, and thus will not be described here. Any electronic device used in the method of the embodiment one of the application belongs to the scope of the application.

[0057] The specific embodiments described herein are merely illustrative of the spirit of the application. Those skilled in the art can make various modifications or supplements to the described specific embodiments or replace them with similar ways, without departing from the spirit of the application or exceeding the scope defined by the appended claims.

Claims

1. A CLIP model-based image forgery detection method, characterized in that, The method comprises the following steps: Step 1: Construct a multi-modal tampering dataset containing real images and various fake images, and add a text description including a tampered object, a tampering parameter and a tampering means to each fake image; Step 2: Fine-tune the CLIP model using the multi-modal tampering dataset to obtain a TamperCLIP model having a coding fake feature capability; Step 3: Perform discrete cosine transformation on the input image, extract a low-frequency component, and fuse the low-frequency component with a real image to generate an enhanced low-frequency input; extract a multi-scale feature map of the enhanced low-frequency input, and perform cross-attention calculation on the smallest scale feature and the TamperCLIP fake feature; and generate a multi-class fake region positioning result through a weight map weighting and a multi-class classification head; Step 4: Perform discrete cosine transformation on the input image, extract a high-frequency component and fuse the high-frequency component with the original image to generate an enhanced high-frequency input; extract a multi-scale feature map of the enhanced high-frequency input, and perform cross-attention calculation on the smallest scale feature and the TamperCLIP fake feature; Calculate the contrast loss based on the largest scale feature through a random sampling strategy to generate a splicing trace prediction mask.

2. The CLIP model-based image forgery detection method of claim 1, wherein: The fake image includes a traditional splicing tampering image, a StarGAN generated face tampering image, a StyleGAN generated complete false image, a CycleGAN cross-domain style transfer image, an Inpainting algorithm modified image and a Stable Diffusion synthesized image.

3. The CLIP model-based image forgery detection method of claim 1, wherein: In step 2, a low-rank adaptive LoRA layer is inserted in each Transformer block of the CLIP model image encoder and text encoder for fine-tuning, and a contrast learning strategy is used to jointly optimize the visual encoder and the text encoder, and a symmetric contrast loss function is used to align the tampering image and the text description to obtain a TamperCLIP model having a coding fake feature capability.

4. The CLIP model-based image forgery detection method of claim 1, wherein: The cross-attention fusion calculation in steps 3 and 4 is as follows: Q is a query vector, which is the smallest scale low-frequency feature output by the SegFormer multi-scale feature extraction network in step 3, and is the smallest scale high-frequency feature output by the ConvNext multi-scale feature extraction network in step 4; K and V are key and value vectors, which are both fake features output by the TamperCLIP.

5. The CLIP model-based image forgery detection method of claim 1, wherein: In step 4, a random sampling strategy is used to construct a positive and negative sample set, and an InfoNCE loss is used to optimize the splicing trace prediction mask.

6. The CLIP model-based image forgery detection method of claim 1, wherein: In steps 3 and 4, the weight map is generated by a weighting module, which includes a convolution input layer, a feature inversion layer and a convolution layer.

7. The CLIP model-based image forgery detection method of claim 1, wherein: The operation process of the weight map generation module is as follows: wherein, is the original input image, is the low frequency map extracted by DCT, is the high frequency map extracted by DCT, Conv is a convolution layer, and LayerNorm is a layer normalization.

8. The CLIP model-based image forgery detection method of claim 1, wherein: In step 3, the multi-class fake region prediction map is finally generated through a multi-class classification head, and the class classification head includes a convolution input layer, a group normalization and a GELU activation function.

9. An electronic device, comprising: The method comprises the following steps: A memory and a processor in communication connection with each other, the memory storing computer instructions, the processor executing the computer instructions to perform the CLIP model-based image forgery detection method according to any one of claims 1-8.

10. A computer readable medium having stored thereon a computer program, characterized in that: The program is executed by the processor to implement the CLIP model-based image forgery detection method according to any one of claims 1-9.

Images