Image forgery multi-type detection method and system based on visual big language model

Abstract

The invention discloses an image counterfeiting multi-type detection method and system based on a visual large language model, and belongs to the technical field of image processing and computer vision. The method comprises the following steps of: preprocessing a data set containing normal, copy and paste forged and spliced forged images, extracting visual and language features by utilizing a pre-training model, and realizing feature fusion by adopting a multi-modal alignment technology. Whether an image is forged or not is judged through a classification task, a forged area is positioned through a segmentation task, and meanwhile a forged text description is output through a text generation module. The loss function is optimized in combination with classification, segmentation, comparison and text cross entropy loss. And adopting an end-to-end training and course learning strategy training model, and outputting a forgery probability, a regional mask and text description during reasoning. According to the method, multiple counterfeit types can be processed at the same time, high-precision positioning is achieved, multi-modal information is effectively fused, and detailed counterfeit text description can be provided.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and particularly to an image forgery detection method combining vision large language models (VLMs), which can simultaneously achieve copy-paste forgery detection and splicing forgery detection for images. Background Art

[0002] With the popularization of image processing technology, image forgery means (such as copy-paste, splicing, etc.) are becoming increasingly sophisticated. Traditional detection methods have the following deficiencies:

[0003] 1. Single-modal limitation: relying only on visual features, it is difficult to capture logical contradictions at the semantic level (such as marine organisms appearing in the desert).

[0004] 2. Insufficient multi-type detection ability: Existing methods are usually designed for a single forgery type, and it is difficult to efficiently detect copy-paste and splicing forgeries simultaneously.

[0005] 3. Low positioning accuracy of forgery areas: The ability to locate small or complex forgery areas is limited.

[0006] Therefore, there is an urgent need for a method that can fuse multi-modal information, take into account the detection and positioning of multiple forgery types. Summary of the Invention

[0007] Object of the Invention

[0008] The present invention aims to provide an image forgery detection method based on vision large language models. Through multi-modal feature fusion and context reasoning, it can simultaneously achieve high-precision detection and positioning of copy-paste forgery and splicing forgery, and output corresponding forgery text descriptions.

[0009] Technical Solution

[0010] An image forgery multi-type detection method based on vision large language models, as Figure 1 shown, includes the following steps:

[0011] S1. Data preprocessing:

[0012] · S11 Image annotation: Construct a data set containing normal images, copy-paste forgery images, and splicing forgery images, annotate the forgery areas and generate text descriptions. For copy-paste forgery images, the text description can be "there are duplicate object structures in the image", "some areas have been copy-pasted", etc.; for splicing forgery images, the text description can be "different illumination areas are spliced together", "the image has unnatural splicing boundaries", etc.

[0013] · S12 Image enhancement: Perform data enhancement (rotation, scaling, adding noise, etc.) on the images to improve the generalization ability of the model.

[0014] S2. Multi-modal Feature Extraction and Fusion:

[0015] · S21 Visual Feature Extraction: Use pre-trained convolutional neural networks (CNNs) such as ResNet or vision transformers (ViTs) to extract low-level features such as local textures and edges of images, as well as global scene semantic features. For copy-paste forgeries, visual features can capture the similarities in textures, colors, etc. of repeated regions; for splicing forgeries, the discontinuities in colors, illuminations, etc. at the splicing boundaries can be detected.

[0016] · S22 Language Feature Extraction: Convert text descriptions into semantic vectors through pre-trained language models (such as BERT). These semantic vectors contain key information about copy-paste and splicing forgeries, such as concepts like "repetition" and "splicing boundary".

[0017] · S23 Multi-modal Fusion: Adopt multi-modal alignment techniques to achieve deep fusion of visual and language features.

[0018] Specifically, it includes the following steps:

[0019] ◆ Intra-modal Feature Clustering: Perform clustering operations on visual and language features respectively. For visual features, use the K-Means clustering algorithm to divide them into multiple clusters, and each cluster represents a visual semantic pattern; for language features, also use the K-Means algorithm for clustering, and each cluster represents a language semantic pattern.

[0020] ◆ Cross-modal Feature Matching: Calculate the similarity between visual feature clusters and language feature clusters, using cosine similarity measurement. For each visual feature cluster, find the language feature cluster with the highest similarity to it and establish cross-modal matching relationships.

[0021] ◆ Feature Fusion: According to the cross-modal matching relationships, concatenate the matched visual and language features, and then perform non-linear transformation through a multi-layer perceptron (MLP) to obtain the fused multi-modal features.

[0022] S3. Forgery Detection and Localization:

[0023] · S31 Classification Task: On the fused multi-modal features, use a fully connected layer to determine whether the image is forged. The features after the fusion of the vision-language model can make comprehensive judgments from semantic and visual levels. For copy-paste forgeries, it can make judgments not only based on visual similarities but also in combination with the semantic concept of "repetition"; for splicing forgeries, it can be discriminated based on the visual anomalies at the splicing boundaries and the semantic information of "splicing boundary".

[0024] · S32 Segmentation Task: Add a semantic segmentation head to predict the mask of the forged area and achieve precise spatial positioning.

[0025] In copy-paste forgery localization, the model can more accurately circle the copied and pasted areas by leveraging multimodal features; in splicing forgery localization, it can accurately locate the splicing area based on the information about the splicing boundary in the fused features.

[0026] · S33 Forged Text Description Generation: Add a text generation module at the output end of the model. Based on the fused multimodal features, use a sequence-to-sequence (Seq 2Seq) model to generate a forged text description. This text description will detail the detected forgery types and features.

[0027] S4. Loss Function Design:

[0028] · S41 Classification Loss: Adopt the cross-entropy loss function to measure the difference between the predicted forgery probability and the true label. The formula is: where yi is the true label, p i is the predicted forgery probability, and N is the number of samples.

[0029] · S42 Segmentation Loss: Adopt the Dice loss function to measure the similarity between the predicted mask of the forged area and the true mask. The formula is: where X is the predicted mask of the forged area and Y is the true mask.

[0030] · S43 Contrastive Loss: Adopt the triplet loss function. Randomly select a positive sample (normal area), a negative sample (forged area), and an anchor sample, such that the distance between the anchor sample and the positive sample is less than the distance between the anchor sample and the negative sample. The formula is: L contrast = max(d(a, p) - d(a, n) + φ, 0), where a is the anchor sample, p is the positive sample, n is the negative sample, d is the distance metric (such as Euclidean distance), and φ is a positive constant.

[0031] · S44 Text Loss: For the text generation module, adopt the cross-entropy loss function to measure the difference between the generated forged text description and the true text description. Let T true be the true text description, T pred be the generated text description, and the text loss where T is the length of the text, w t is the t-th token in the true text, and p t is the token probability at the corresponding position in the generated text.

[0032] The total loss L = αL cls + βL seg + γL contrast+δL text , where α, β, γ, and δ are hyperparameters used to balance the weights of the four losses.

[0033] S5. Model Training and Inference:

[0034] · S51 End-to-end Training: Simultaneously optimize the classification, segmentation, contrast, and text generation tasks, and gradually increase the model complexity using the curriculum learning strategy. During the training process, the vision-language model continuously learns the association between visual features and language features, improving the detection ability for copy-paste and splicing forgeries.

[0035] · S52 Inference Phase: Input the image to be detected, and the model outputs the forgery probability, forgery region mask, and forgery text description. Combine language prompts (such as "detect copy-paste region", "detect splicing forgery boundary") to enhance the detection effect of specific types of forgeries. The language prompts further guide the model to focus on the key features of specific types of forgeries, improving the detection accuracy and efficiency.

[0036] Beneficial Effects

[0037] 1. Multi-type Detection Ability: Through the multi-modal feature fusion of the vision-language model, it can simultaneously handle copy-paste and splicing forgeries, enhancing the generalization ability in complex scenarios. The model can use language features to guide the analysis of visual features and identify different types of forgeries from different angles.

[0038] 2. Semantic-level Context Reasoning: Utilize the logical reasoning ability of the language model to capture semantic contradictions in the image content. In copy-paste forgery detection, it can understand the semantics of "repetition" and combine visual features for judgment; in splicing forgery detection, it can analyze the rationality of the image based on semantic information such as "splicing boundary", improving the detection accuracy.

[0039] 3. High-precision Localization: Combining the classification and segmentation tasks, the multi-modal features of the vision-language model enable the model to more accurately locate the forgery region, whether it is a small region of copy-paste or a complex boundary of splicing.

[0040] 4. Better Multi-modal Fusion: Adopt multi-modal alignment technology, through intra-modal clustering and cross-modal matching, to better fuse image features and text features, enhancing the model's ability to understand and utilize different modal information.

[0041] 5. Output of Forged Text Description: Provide a detailed forged text description to help users more intuitively understand the detected forgery situation. Description of the Drawings

[0042] Figure 1 Method Functional Module Diagram

[0043] Figure 2Method implementation flowchart Detailed implementation manners

[0044] Specific architecture description of the model

[0045] The visual large language model of the present invention mainly consists of a visual encoder, a language encoder, a multi-modal fusion module, a classification head, a segmentation head, and a text generation module. Its process is as Figure 2 shown. The following are the detailed descriptions of each part:

[0046] Visual encoder

[0047] Adopt ViT (Vision Transformer) as the visual encoder. ViT divides the input image into multiple image patches of a fixed size, linearly projects each image patch into a one-dimensional vector, and then adds positional encoding information and inputs it into the encoder of the Transformer. The specific steps are as follows:

[0048] 1. Image patching: Divide the input image into non-overlapping image patches, and the size of each image patch is P×P×C, where H and W are the height and width of the image respectively, and C is the number of channels of the image.

[0049] 2. Linear projection: Flatten each image patch into a one-dimensional vector, and then project it into a D-dimensional space through a linear layer to obtain the projected vector

[0050] 3. Positional encoding: Add positional encoding to each projected vector to retain the spatial information of the image patch, and obtain the input x = x patch +E pos .

[0051] 4. Transformer encoder: Input x into a multi-layer Transformer encoder. Each encoder layer consists of a multi-head self-attention mechanism (Multi-Head Self-Attention) and a feed-forward neural network (Feed-Forward Network). After multi-layer encoding, visual features are obtained

[0052] Language encoder

[0053] Use BERT (Bidirectional Encoder Representations from Transformers) as the language encoder. BERT is a pre-trained language model that can convert the input text description into a semantic vector. The specific steps are as follows:

[0054] 1. Text tokenization: Tokenize the input text description to obtain a series of tokens.

[0055] 2. Token embedding: Convert each token into a D-dimensional token embedding vector through the embedding layer.

[0056] 3. Position encoding and segment encoding: Add position encoding and segment encoding to each token embedding vector to represent the position of the token in the text and the sentence it belongs to.

[0057] 4. BERT encoder: Input the encoded token embedding vectors into the multi-layer Transformer encoder of BERT, and obtain the language features after encoding where M is the number of tokens in the text.

[0058] Multi-modal fusion module

[0059] Adopt multi-modal alignment technology to achieve the fusion of visual features and language features, as follows:

[0060] 1. Intra-modal feature clustering:

[0061] · Visual feature clustering: Perform K-Means clustering on the visual feature V, and set the number of clusters to K V 。By iteratively updating the cluster centers, divide the visual features into K V clusters, and each cluster has a cluster center

[0062] · Language feature clustering: Perform K-Means clustering on the language feature L, and set the number of clusters to K L 。Similarly, by iteratively updating the cluster centers, divide the language features into K L clusters, and each cluster has a cluster center

[0063] 2. Cross-modal feature matching:

[0064] Calculate the cosine similarity between each visual feature cluster center and each language feature cluster center ​For each visual feature cluster, select the language feature cluster with the highest similarity for matching and establish cross-modal matching relationships.

[0065] 3. Feature fusion:

[0066] According to the cross-modal matching relationships, splice the matched visual features and language features. For each pair of matches, splice the corresponding visual feature vector and language feature vector into a new vector, and then perform a non-linear transformation through a multi-layer perceptron (MLP) to obtain the fused multi-modal feature F. The MLP consists of multiple fully connected layers and activation functions (such as ReLU).

[0067] Classification head

[0068] Perform global average pooling on the fused multi-modal feature F to obtain a vector of fixed length, and then perform binary classification through a fully connected layer to output the probability that the image is forged.

[0069] Segmentation head

[0070] Adopt a decoder with a U-Net structure to gradually upsample the fused multi-modal feature F to restore it to the same size as the input image and predict the mask of the forged area.

[0071] The text generation module adopts a Transformer-based Seq2Seq model, uses the fused multi-modal feature F as the input, and generates forged text descriptions. The model includes an encoder and a decoder. The encoder encodes the multi-modal features, and the decoder generates a text sequence based on the encoded information.

[0072] Description of the training method

[0073] Training data preparation: Divide the constructed dataset into a training set, a validation set, and a test set, and the ratio can be set to 7:1:2. Perform data augmentation on the images in the training set, including operations such as random rotation (angle range from -180° to 180°), random scaling (scaling ratio range from 0.8 to 1.2), and random noise addition (Gaussian noise, standard deviation range from 0 to 0.1) to increase the diversity of the data.

[0074] Training process

[0075] Adopt an end-to-end training method and optimize the classification, segmentation, contrast, and text generation tasks simultaneously. The specific steps are as follows:

[0076] 1. Initialize the model parameters: Load the weights of the pre-trained ViT and BERT models, and randomly initialize the parameters of the multi-modal fusion module, classification head, segmentation head, and text generation module.

[0077] 2. Forward propagation: The input image and its corresponding text description are used to extract visual features and language features through a visual encoder and a language encoder respectively. Then, the two types of features are fused through a multi-modal fusion module. Subsequently, the forgery probability, forgery region mask, and forged text description are output through a classification head, a segmentation head, and a text generation module respectively.

[0078] 3. Loss calculation: According to the formulas in the loss function design part above, calculate the classification loss L cls , segmentation loss L seg , contrastive loss L contrast and text loss L text , and calculate the total loss L.

[0079] 4. Backward propagation: Calculate the gradients based on the total loss, and use the Adam optimizer to update the parameters of the model.

[0080] 5. Curriculum learning strategy: At the beginning of training, mainly focus on the classification task, that is, α is larger, and β, γ, and δ are smaller; as training progresses, gradually increase the weights of the segmentation, contrastive, and text generation tasks to improve the comprehensive performance of the model.

[0081] Training parameter settings

[0082] · Learning rate: The initial learning rate is set to 1e-4, and a cosine annealing learning rate scheduler is adopted to gradually reduce the learning rate as training progresses.

[0083] · Batch size: Set to 16.

[0084] · Number of training epochs: Train for 100 epochs.

[0085] Inference process

[0086] 1. Preprocess the image to be detected and resize it to the size required for model input.

[0087] 2. Generate a corresponding text description for the image to be detected, such as "Detect copy-paste area" or "Detect stitching forgery boundary".

[0088] 3. Input the image and text description into the trained model, and obtain the forgery probability, forgery region mask, and forged text description through forward propagation.

[0089] 4. Determine whether the image is forged based on the forgery probability. If the probability is greater than the set threshold (such as 0.5), the image is considered forged.

[0090] 5. For forged images, locate the forged region according to the forgery region mask and combine the forged text description to understand the forgery situation in detail.

Claims

1. A multi-type detection method for image forgery based on a vision-language model, characterized in that Including: ·S1 data preprocessing step, constructing a dataset containing normal images, copy-paste forged images, and spliced forged images, annotating the forged regions and generating corresponding text descriptions, and performing data augmentation on the images; ·S2 multi-modal feature extraction and fusion step, using a pre-trained visual encoder to extract visual features of the image, using a pre-trained language encoder to extract language features of the text description, and using multi-modal alignment technology to fuse the visual features and language features, which includes intra-modal feature clustering, cross-modal feature matching, and feature fusion; ·S3 forged detection and localization step, performing a classification task on the fused multi-modal features to determine whether the image is forged, and at the same time predicting the mask of the forged region through a semantic segmentation head to achieve the localization of the forged region, and generating a forged text description through a text generation module; ·S4 loss function design step, combining categorical cross-entropy loss, segmentation Dice loss, contrastive loss, and text cross-entropy loss to optimize the detection, localization, and text generation capabilities of the model; ●S5 model training and inference step, adopting an end-to-end training method, training the model in combination with a curriculum learning strategy, and inputting the image to be detected and the corresponding text description in the inference stage to output the forged probability, the forged region mask, and the forged text description.

2. The method according to claim 1, wherein The visual encoder is ViT, which divides the input image into multiple image patches, and after linear projection and position encoding, inputs them into a multi-layer Transformer encoder to output visual features.

3. The method according to claim 1, characterized in that, The language encoder is BERT, which tokenizes, token embeds, position encodes, and segment encodes the input text description and then inputs it into the multi-layer Transformer encoder of BERT to output language features.

4. The method according to claim 1, characterized in that The intra-modal feature clustering uses the K-Means clustering algorithm to cluster the visual features and language features respectively to obtain visual feature clusters and language feature clusters.

5. The method according to claim 1, wherein The cross-modal feature matching calculates the cosine similarity between the visual feature clusters and the language feature clusters, finds the language feature cluster with the highest similarity to each visual feature cluster, and establishes a cross-modal matching relationship.

6. The method according to claim 1, characterized in that The feature fusion concatenates the matched visual features and language features, and then performs a non-linear transformation through a multi-layer perceptron to obtain the fused multi-modal features.

7. The method according to claim 1, characterized in that The classification head uses global average pooling and a fully connected layer for binary classification to output the probability of whether the image is forged.

8. The method according to claim 1, wherein The segmentation head uses a decoder with a U-Net structure to gradually upsample the fused multi-modal features to restore them to the same size as the input image and predict the mask of the forged region.

9. The method according to claim 1, wherein The text generation module uses a Transformer-based Seq2Seq model, taking the fused multi-modal features as input to generate a forged text description.

10. The method according to claim 1, wherein The contrastive loss uses a triplet loss function, randomly selecting a positive sample, a negative sample, and an anchor sample, such that the distance between the anchor sample and the positive sample is less than the distance between the anchor sample and the negative sample.

11. The method according to claim 1, characterized in that, The curriculum learning strategy mainly focuses on the classification task at the beginning of training, and gradually increases the weights of the segmentation, contrastive, and text generation tasks as training progresses.

12. An image forgery multi-type detection system based on a vision large language model, characterized in that, Including: ● The data preprocessing module is used to construct a data set, annotate forged regions, generate text descriptions, and perform data augmentation; · The feature extraction and fusion module includes a visual encoder, a language encoder, and a multimodal fusion module, and is used to extract visual features and language features and fuse them; · The detection and localization module includes a classification head and a segmentation head, and is used to determine whether an image is forged and locate the forged region; ● The text generation module is used to generate forged text descriptions; ● The loss calculation module is used to calculate classification loss, segmentation loss, contrast loss, and text loss; · The training and inference module is used to train the model in an end-to-end training manner and output the forgery probability, the forgery region mask, and the forged text description in the inference stage.

13. The system according to claim 12, wherein The specific structures and working methods of the visual encoder, the language encoder, the multimodal fusion module, the classification head, the segmentation head, and the text generation module are as described in claims 2-9.

Images