Fine-grained tampering information detection and positioning method based on large model
By using large models for tampering warning learning and cross-modal comparative learning, combined with knowledge injection and bridging adapters, the problem of difficulty in identifying subtle tampering in existing technologies is solved, achieving accurate detection and localization of multimodal tampering information and providing understandable interpretation results.
Patent Information
- Application Number
- CN202511531280.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2026-02-13
AI Technical Summary
Existing multimodal tampering detection methods struggle to accurately identify subtle tampering, such as minor adjustments to facial expressions or slight text modifications. Furthermore, CLIP prompts are not sufficiently effective, and the methods suffer from weak multimodal understanding, leading to positioning errors.
We employ a large model for tampering cue learning, cross-modal contrastive learning, knowledge injection, and large language model inference. By generating general and hierarchical tampering cuees, we utilize modal-aware contrastive loss and bridging adapter to extract interactive features between images and text for accurate detection and localization.
It achieves accurate detection and location of multimodal tampering information, can identify pixel-level image modifications and character-level text adjustments, and provides traceable detection results.
Smart Images
Figure CN121527486A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the cross field of artificial intelligence and natural language processing, and in particular to a fine-grained tampering information detection and positioning method based on a large model.
BACKGROUND
[0002] With the development of image editing technology (such as diffusion model) and text generation technology (such as AIGC), it becomes increasingly easy to manufacture multi-modal (image-text) tampered information, such as fake face images and tampered news texts. Such tampered information may be used to create fake news and forge evidence, posing a serious threat to social security and personal privacy, and therefore precise detection and positioning technology is urgently needed.
[0003] Existing multi-modal tampering detection methods (DGM) mainly rely on identifying subtle artifacts in images or tampering traces in texts, extracting forgery features through neural networks or frequency analysis, and using CLIP and other models to learn image-text semantic similarity to assist in locating tampered areas. However, the limitations of these methods are:
[0004] 1. Fine tampering is difficult to detect. Pixel-level image modification such as fine-tuning facial expressions, or slight text adjustment such as replacing individual words, is difficult to accurately identify due to the small changes;
[0005] 2. CLIP prompt effectiveness is insufficient. Traditional CLIP-based methods lack targeted text prompts, limiting the adaptability of their multi-modal learning capabilities in detection tasks;
[0006] 3. Weak multi-modal understanding. If you cannot understand specific entities in the text, such as "a person's hairstyle", it is difficult to align the semantics of images and texts, leading to positioning bias.
[0007] Therefore, the present application is based on the above deficiencies.
SUMMARY
[0008] The present application aims to overcome the shortcomings of the prior art and provide a fine-grained tampering information detection and positioning method based on a large model, which realizes accurate detection, positioning and explanation of multi-modal tampered information through tampering prompt learning, cross-modal contrast learning, knowledge injection and large model reasoning.
[0009] The present application is achieved by the following technical solutions:
[0010] A fine-grained tampering information detection and positioning method based on a large model, characterized by the following steps:
[0011] S1, tampering prompt learning is performed, general tampering prompts and hierarchical tampering tokens are generated, and the general tampering prompts and the hierarchical tampering tokens are input into a text encoder together with description text of an image to obtain text prompt features, and meanwhile, image pixel features are processed through a pixel decoder, a similarity score of the image features and the text features is calculated, and a tampering segmentation map is generated;
[0012] S2, modal perception contrast losses of image-to-text and text-to-image are constructed based on InfoNCE losses, and semantic differences between the image and the text are learned;
[0013] S3, the image and the text features are encoded by using a multi-modal encoder, knowledge related to the text is retrieved through a search engine, the retrieved knowledge and the text features are fused by using a bridge adapter, interactive features of the image and the text are extracted through a bidirectional cross-attention mechanism, positioning results of tampered image regions and tampered text regions are obtained by inputting the interactive features into a decoder, and tampering type classification results are obtained by inputting the interactive features into a classifier;
[0014] S4, the interactive features, the positioning results and the classification results are input into a large language model for reasoning, and explanation results are generated according to a prompt learning template.
[0015] The fine-grained tampering information detection and positioning method based on a large model as described above, characterized in that the general tampering prompts in the step S1 include general tampering characters for capturing common fake modes of all manipulated images and specific tampering characters for supplementing fake modes other than the general features, the hierarchical tampering tokens are trainable tokens and are used to encode fine-grained and image-dependent artifacts, and the text encoder is a CLIP text encoder, and when the CLIP text encoder is frozen, the hierarchical tampering tokens are input into a Transformer layer of the CLIP text encoder.
[0016] The fine-grained tampering information detection and positioning method based on a large model as described above, characterized in that the step of generating the tampering segmentation map in the step S1 includes:
[0017] S11, normal text prompts and abnormal text prompts are combined with the general tampering prompts and the hierarchical tampering tokens, and then input into the CLIP text encoder to obtain text prompt features
[0018] S12, a pixel decoder of a continuous deconvolution layer converts low-resolution image pixel features into high-resolution features
[0019] S13, a similarity score of the high-resolution image features and the text prompt features is calculated, and the calculation formula is:
[0020]
[0021] The tampering segmentation map is generated by a softmax function, and the calculation formula is:
[0022]
[0023] The large model-based fine-grained tampering information detection and positioning method as described above, characterized in that: the image-to-text contrast learning loss in step S2, the calculation formula is:
[0024]
[0025] Where τ represents the temperature parameter, T - represents negative text, T + represents positive text, -E p represents the contrast learning loss, K represents the number of samples;
[0026] The text-to-image contrast learning loss, the calculation formula is:
[0027]
[0028] Where I - represents a negative sample image, I + represents a positive sample image.
[0029] The large model-based fine-grained tampering information detection and positioning method as described above, characterized in that, step S3 includes:
[0030] S31, multi-modal feature encoding, using ImageBind encoder to encode image or text features, the encoding features X v of the image and the encoding features X t of the text are represented as:
[0031] X v ={v cls ,v pat}
[0032] X t ={t cls ,t tok}
[0033] Where v cls represents image category encoding, v pat represents image pixels, t cls represents text category encoding, and t tok represents text characters.
[0034] S32, knowledge injection, using LLM to generate a search phrase, and then using a search engine to search the generated phrase to obtain knowledge E t , E t and Xt Input into the bridge adapter, after fusion, get the feature f0;
[0035] S33, image and text tampering positioning, through the normalized query Q, key K, value V feature to calculate the attention function, the calculation formula is:
[0036]
[0037] Calculate the image-text interaction feature, the calculation formula is:
[0038] X v→t =Attention(f0,X v ,X v )
[0039] Put X v→t into the BBox decoder D v , output the prediction result of the tampered area of the image, the calculation formula is:
[0040] b=D v ([X v→t ])
[0041] Calculate the text-image interaction feature, the calculation formula is:
[0042] X t→v =Attention(X v ,X t ,X t )
[0043] Put X t→v into the text character decoder D t , get the text tampering positioning result, the calculation formula is:
[0044] t=D t ([X t→v ])
[0045] Where d represents the feature dimension.
[0046] S34, tampering classification, input X t→v into the classifier, get the classification result of the multi-modal tampering information, the calculation formula is:
[0047] Y=MLP(X t→v ).
[0048] The fine-grained tampering information detection and positioning method based on large model as described above, characterized in that: in step S31, when the input image contains a face, the face feature code is extracted using the Resnet50 network and fused into the image code feature generated by the ImageBind encoder.
[0049] The fine-grained tampering information detection and positioning method based on a large model as described above, characterized in that the inference process of the large language model in step S4 comprises:
[0050] S41, analyze whether there is face manipulation in the image, if yes, record the type and classify the news as false;
[0051] S42, analyze whether there is manipulation in the text, if yes, record the type and manipulation words and classify the news as false;
[0052] S43, if neither the image nor the text is manipulated, classify the news as true.
[0053] Compared with the prior art, the present application has the following advantages:
[0054] 1. The present application introduces tampering prompt learning, generates different text embeddings for different images, and locates the tampered image area under the guidance of the prompt. The general tampering prompt can be generalized to unknown fake types, solving the problem of poor adaptability of traditional methods to new tampering modes. Moreover, through hierarchical tampering tokens and pixel-level feature recovery, pixel-level image modification and character-level text adjustment can be recognized. Secondly, using modal perception contrast loss, the semantic difference between image-text is learned, the text tampering area is located, and the classification result of tampering information is given. At the same time, a bridge adapter is designed to inject the retrieved knowledge into the text modality. The modal perception contrast loss and knowledge injection mechanism improve the semantic difference recognition ability of image and text. Finally, the recognized image tampering clues, text tampering clues, image features and text features are input into the LLM to provide multi-modal tampering information detection and positioning explanation, making the detection conclusion traceable and easy to understand. BRIEF DESCRIPTION OF DRAWINGS
[0055] Figure 1 The schematic diagram of the present application. DETAILED DESCRIPTION
[0056] The present application will be further described below in conjunction with the drawings:
[0057] As shown in the drawings, a fine-grained tampering information detection and positioning method based on a large model comprises the following steps: Figure 1 S1, tampering prompt learning is performed to generate general tampering prompts and hierarchical tampering tokens, which are input into a text encoder together with the description text of the image to obtain text prompt features. At the same time, the image pixel features are processed through a pixel decoder to calculate the similarity score of the image features and the text features and generate a tampering segmentation map;
[0058]
[0059] S2, construct the modal perception contrast loss of image to text and text to image based on InfoNCE loss, learn the semantic difference between image and text;
[0060] S3, encode the image and text features using a multi-modal encoder, retrieve knowledge related to the text through a search engine, fuse the retrieved knowledge and text features using a bridge adapter, extract the interactive features of the image and text through a bidirectional cross-attention mechanism, input the decoder to obtain the positioning results of the tampered image region and the tampered text region, and input the classifier to obtain the tampering type classification results;
[0061] S4, input the interactive features, positioning results and classification results into a large language model for reasoning, and generate an explanation result according to a prompt learning template.
[0062] In step S1, the main goal of tampering prompt learning is to fully utilize language knowledge to guide image tampering and forgery positioning. It includes two key components: general tampering prompts and hierarchical tampering tokens. The general tampering prompts include general tampering characters for capturing common forgery patterns of all manipulated images and specific tampering characters for supplementing forgery patterns other than general features. For example, general tampering characters can capture common forgery patterns and invariants shared by various manipulated facial images, which is crucial for generalization to unseen forgeries. Hierarchical tampering tokens are trainable tokens used to encode fine-grained, image-dependent artifacts such as blurred eyes caused by attribute manipulation and mixed boundaries caused by face swapping. The text encoder is a CLIP text encoder, and when the CLIP text encoder is frozen, the hierarchical tampering tokens are input into its Transformer layers.
[0063] Specifically, the step of generating a tampering segmentation map in step S1 includes:
[0064] S11, input the normal text prompt and the abnormal text prompt combined with the general tampering prompt and the hierarchical tampering token into the CLIP text encoder to obtain the text prompt feature
[0065] S12, the pixel decoder of the continuous deconvolution layer converts the low-resolution image pixel features into high-resolution features
[0066] S13, calculate the similarity score of the high-resolution image features and the text prompt features, the calculation formula is:
[0067]
[0068] Generate a tampering segmentation map through a softmax function, the calculation formula is:
[0069]
[0070] Step S12 converts the low-resolution image pixel features into high-resolution features The purpose is to restore local spatial details.
[0071] To help the two single-modal encoders better exploit the semantic correlation between images and texts, the present application aligns the image and text embeddings through cross-modal contrastive learning. However, some subtle multi-modal manipulations will cause slight semantic inconsistencies between the two modalities, and these are usually difficult to be revealed by ordinary contrastive learning. To emphasize the semantic inconsistencies caused by these manipulations, manipulation-aware contrastive learning is proposed for image and text embeddings. Unlike ordinary cross-modal contrastive learning, which only pushes apart the embeddings of mismatched pairs when bringing the embeddings of original image-text pairs closer, manipulation-aware contrastive learning also pushes apart the embeddings of manipulated pairs, thereby further emphasizing the semantic inconsistencies they produce. Specifically, the image-to-text contrastive learning loss in step S2 is calculated as follows:
[0072]
[0073] where τ represents a temperature parameter, T - represents negative text, T + represents positive text, -E p represents the contrastive learning loss, and K represents the number of samples.
[0074] The text-to-image contrastive learning loss is calculated as follows:
[0075]
[0076] where I - represents a negative sample image, and I + represents a positive sample image.
[0077] Step S3 includes:
[0078] S31, multi-modal feature encoding, using the ImageBind encoder to encode the image or text features, the encoded features of the image X v and the encoded features of the text X t are represented as follows:
[0079] X v = {v cls ,v pat}
[0080] X t = {t cls ,t tok}
[0081] where v cls represents image class encoding, v pat represents image pixel, t cls represents text class encoding, t tok represents text character;
[0082] S32, knowledge injection, generate a search phrase using an LLM, and then use a search engine to search the generated phrase to obtain knowledge E t , E t and X t are input into a bridge adapter, and after fusion, the feature f0 is obtained;
[0083] S33, image and text tampering localization, calculate the attention function through the normalized query Q, key K, and value V features, and the calculation formula is:
[0084]
[0085] Calculate the image-text interaction feature, and the calculation formula is:
[0086] X v→t =Attention(f0,X v ,X v )
[0087] X v→t is input into the BBox decoder D v , and the prediction result of the tampered area of the image is output, and the calculation formula is:
[0088] b=D v ([X v→t ])
[0089] Calculate the text-image interaction feature, and the calculation formula is:
[0090] X t→v =Attention(X v ,X t ,X t )
[0091] X t→v is input into the text character decoder D t , and the text tampering localization result is obtained, and the calculation formula is:
[0092] t=D t ([X t→v ])
[0093] where d represents the feature dimension.
[0094] S34, tampering classification, X t→vThe input is classified into the classifier to obtain the classification result of the multi-modal tampering information, and the calculation formula is:
[0095] Y = MLP(X t→v ).
[0096] In step S31, when the input image contains a human face, the face feature code is extracted using the Resnet50 network and is fused into the coding feature of the image generated by the ImageBind encoder.
[0097] In step S32, the phrase generated by the large language model LLM uses the google search engine. The bridge adapter can provide external knowledge for tampering information detection and positioning.
[0098] Because the lack of complementary information between modalities may hinder cross-modal semantic reasoning. For this purpose, the present application introduces a double-branch cross-attention mechanism in step S33, which can establish close feature interaction between different modalities to guide the interaction between image and text features, so that semantic correlation can be extracted and multi-modal information can be better fused.
[0099] The reasoning process of the large language model in step S4 includes:
[0100] S41, analyze whether there is face manipulation in the image, if there is, record the type and classify the news as false;
[0101] S42, analyze whether there is manipulation in the text, if there is, record the type and manipulation words and classify the news as false;
[0102] S43, if neither the image nor the text is manipulated, the news is classified as true.
[0103] The prompt learning template in step S4 is:
[0104] ##Human:
[0105] <semantictoken> <text> <forgerytoken>< / forgerytoken> < / text> <classification>
[0106] You are a professional assistant specialized in detecting fake news.
[0107] ##Chain-of-Thought (CoT) Reasoning Process:
[0108] 1. First, analyze the image to determine if the face has been manipulated. If manipulated, record the type and classify the news as fake.
[0109] 2. Second, analyze the text for any manipulations. If manipulated, record the type and manipulated words, and classify the news as fake.
[0110] 3. If neither the image nor the text is manipulated, classify the news as real.
[0111] ##Rules:
[0112] Generate a JSON object with four properties: 'Label', 'ImageManipulate', 'Text Manipulate', and 'Manipulated Words'.
[0113] ##Your Task:
[0114] Given a piece of Input Text and Image, your task is to analyze both the image and text to determine the authenticity of the news and provide a JSON-formatted result.
[0115] Input Text:{TEXT}
[0116] ##Your Response:
[0117] Let’s think step by step according to the above Chain-of-Thought,yourresponse:
[0118] In summary, the invention introduces tampering prompt learning, generating distinct text embeddings for different images, and locating tampered image areas under the guidance of prompts. The general tampering prompt can be applied to unknown forgery types, addressing the poor adaptability of traditional methods to new tampering patterns. Furthermore, through hierarchical tampering tokens and pixel-level feature recovery, it can identify pixel-level image modifications and character-level text adjustments. Secondly, using a modality-aware contrastive loss, it learns the semantic differences between images and text, locates text tampering areas, and provides classification results of tampering information. A bridge adapter is also designed to inject the retrieved knowledge into the text modality. The modality-aware contrastive loss and knowledge injection mechanism enhance the semantic difference recognition ability between images and text. Finally, the identified image tampering clues, text tampering clues, image features, and text features are input into the LLM to provide multi-modal tampering information detection and positioning explanations, making the detection conclusions traceable and easy to understand.< / classification> < / semantictoken>
Claims
1. A large model-based fine-grained tampering information detection and positioning method, characterized in that, The method comprises the following steps: S1, tampering prompt learning is performed, general tampering prompts and hierarchical tampering tokens are generated, and the general tampering prompts and the hierarchical tampering tokens are input into a text encoder together with description text of an image to obtain text prompt features, and meanwhile, image pixel features are processed through a pixel decoder, a similarity score of the image features and the text features is calculated, and a tampering segmentation map is generated; S2, modal perception contrast loss of image to text and text to image is constructed based on InfoNCE loss, and semantic difference between the image and the text is learned; S3, the image and the text features are encoded by using a multi-modal encoder, knowledge related to the text is retrieved through a search engine, the retrieved knowledge and the text features are fused by using a bridge adapter, interactive features of the image and the text are extracted through a bidirectional cross-attention mechanism, positioning results of tampered image regions and tampered text regions are obtained by inputting the interactive features into a decoder, and tampering type classification results are obtained by inputting the interactive features into a classifier; S4, the interactive features, the positioning results and the classification results are input into a large language model for reasoning, and explanation results are generated according to a prompt learning template.
2. The large model-based fine-grained tampering information detection and positioning method according to claim 1, characterized in that: The general tampering prompts in the step S1 comprise general tampering characters for capturing common fake modes of all manipulated images and specific tampering characters for supplementing fake modes other than the general features, the hierarchical tampering tokens are trainable tokens and are used for encoding fine-grained and image-dependent artifacts, and the text encoder is a CLIP text encoder, when the CLIP text encoder is frozen, the hierarchical tampering tokens are input into a Transformer layer of the CLIP text encoder.
3. The large model-based fine-grained tampering information detection and positioning method according to claim 1, characterized in that, The step of generating the tampering segmentation map in the step S1 comprises the following steps: S11, input the normal text prompt and the abnormal text prompt into the CLIP text encoder after combining the general tampering prompt and the hierarchical tampering token to obtain text prompt features S12, the pixel decoder of the successive deconvolution layer converts the low-resolution image pixel features into high-resolution features S13, a similarity score of high-resolution image features and text prompt features is calculated, and the calculation formula is as follows: The tampering segmentation map is generated through a softmax function, and the calculation formula is as follows:
4. The large model-based fine-grained tampering information detection and positioning method according to claim 1, characterized in that: The contrast learning loss of image to text in the step S2 is calculated by the following formula: where τ denotes a temperature parameter, T - denotes negative text, T + denotes positive text, -E p denotes a contrastive learning loss, K denotes a number of samples; The contrast learning loss of text to image is calculated by the following formula: where I - represents a negative sample image, I + represents a positive sample image.
5. The large model-based fine-grained tampering information detection and positioning method according to claim 1, characterized in that, The step S3 comprises the following steps: S31, multi-modal feature encoding, encoding image or text features using an ImageBind encoder, the encoded features X v of the text t The representation is: X v = {v cls , v pat} X t = {t cls , t tok} where v cls represents an image class code, v pat represents an image pixel, t cls represents a text class code, t tok represents a text character; S32, knowledge injection, generate a search phrase using LLM, and then use a search engine to search the generated phrase to obtain knowledge E t put E t and X t into the bridge adapter, and after fusion, the feature f0 is obtained; S33, image and text tampering positioning is performed, an attention function is calculated through normalized query Q, key K and value V features, and the calculation formula is as follows: Image-text interactive features are calculated by the following formula: X v→t = Attention(f0, X v , X v ) Put X v→t Input to the BBox decoder D v The predicted result of the tampered region of the output image is calculated by the following formula: b = D v ([X v→t ]) Text-image interactive features are calculated by the following formula: X t→v = Attention(X v , X t , X t ) Put X t→v Input to the text character decoder D t The text tampering positioning result is obtained, and the calculation formula is: t = D t ([X t→v ]) Wherein, d represents a feature dimension. S34, tampering classification, put X t→v into the classifier, get the classification result of the multi-modal tampering information, the calculation formula is: Y = MLP(X t→v ).
6. The large model-based fine-grained tampering information detection and positioning method according to claim 5, characterized in that: In the step S31, when the input image contains a face, a Resnet50 network is used to extract face feature encoding, and the face feature encoding is fused into image encoding features generated by an ImageBind encoder.
7. The large model-based fine-grained tampering information detection and positioning method according to claim 1, characterized in that, The reasoning process of the large language model in the step S4 comprises the following steps: S41, whether a face manipulation exists in the image is analyzed, if yes, the type is recorded, and the news is classified as false; S42, whether a manipulation exists in the text is analyzed, if yes, the type and the manipulated words are recorded, and the news is classified as false; S43, if neither the image nor the text is manipulated, the news is classified as true.