A knowledge enhancement-based multi-modal large model deep fake detection method and system

Abstract

The application discloses a kind of multi-modal big model depth forgery detection method and system based on knowledge enhancement, belong to computer vision technical field.The system includes: data set construction module, for constructing the instance data set containing real image sample and fake image sample;Instance guide module is used to retrieve several instance data from instance data set according to test image, and obtain true and false discrimination logic based on instance data and the label of instance data;Knowledge enhancement module is used to encode fusion to test image, structured authentication priori knowledge and true and false discrimination logic, and obtain knowledge enhancement structured prompt;Model inference module is used to convert knowledge enhancement structured prompt into the input format that multi-modal big model can recognize, then input into multi-modal big model and execute multidimensional inference, obtain the detection result of test image.The application can improve the generalization ability of model to unknown forgery type, and output the detection conclusion with explainability.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and artificial intelligence security technology, and in particular relates to a knowledge-enhanced multimodal large model (MLLM) deepfake detection method and system. Background Technology

[0002] In recent years, with the rapid development of generative artificial intelligence technology, large generative models, represented by diffusion models (DMs) and generative adversarial networks (GANs), have been widely used. AI-generated images possess a high degree of visual realism, making it difficult to directly distinguish them from real images. However, the widespread adoption and development of these synthetic images has also brought serious information security risks. When deepfake images are misused in constructing fake scenarios involving well-known politicians and public figures, or in spreading false information about sensitive social topics, they can easily distort public perception, exacerbate social conflicts, and pose a serious threat to social trust and public stability. Therefore, developing efficient and universal deepfake image detection methods has become an important research direction in the field of artificial intelligence technology, and it has significant practical implications for curbing the spread of false information and ensuring information security.

[0003] To address the need for deepfake image detection, various detection methods have been developed in related technical fields. Early deepfake detection methods were mainly implemented using deep neural networks. These methods often employ lightweight network architectures such as Convolutional Neural Networks (CNNs) and Visual Transformers (ViT), coupled with efficient model optimization strategies, demonstrating good performance in detection accuracy and forgery region localization. The core detection logic of these methods involves manually designing and extracting features from the image (such as noise inconsistencies, lighting anomalies, JPEG compression traces, etc.) or capturing forgery traces in the image based on data-driven mechanisms. Some methods also analyze image features from a single dimension or a fusion of two dimensions in the spatial and frequency domains, while combining deep learning techniques such as data augmentation, supervised learning, and image reconstruction to improve detection accuracy. Some solutions also design customized forensic features specifically for detecting the output results of specific generative models.

[0004] To address the insufficient generalization ability of traditional deep neural network detection methods, researchers have recently begun applying Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) to deepfake detection. Multimodal Large Language Models, based on the Transformer architecture, possess rich general knowledge after pre-training on large-scale datasets. They also boast powerful instruction following capabilities, natural language generation capabilities, and visual-semantic feature alignment capabilities. When applied to deepfake detection, they can analyze and capture subtle forgery traces that are difficult for human vision to detect, demonstrating significant potential in detecting unknown types of forged images. Furthermore, these models can generate detailed text explanations, greatly improving the interpretability of the detection results. There are already several research results on deepfake detection that integrate multimodal large models. For example, by fine-tuning the multimodal large model on a dedicated dataset and incorporating specific modules, the accuracy and interpretability of face forgery detection can be improved; or a multimodal framework can be constructed and dedicated detection and localization modules can be designed to achieve fine-grained analysis of forged images; or the detection and localization capabilities of the model can be enhanced by a mask-aware forgery extractor to achieve accurate localization of tampered areas.

[0005] Although existing deepfake detection methods have made some research progress, various methods still have obvious defects and shortcomings in practical applications, making it difficult to meet the needs of universal, efficient, and low-cost deepfake detection. The shortcomings can be roughly divided into the following points: Traditional deep neural network-based detection methods have several drawbacks. First, these methods can only perform binary classification to determine the authenticity of images, and their detection results lack interpretable reasoning. Second, their detection performance is highly dependent on the size, diversity, and design of the training dataset. The model can only maintain good performance on specific datasets and specific forgery types covered during training, and its generalization ability drops significantly when faced with unknown types of forged images. Finally, some methods design customized forensic features to detect the output results of specific generative models, further leading to poor model adaptability and difficulty in dealing with the ever-evolving image generation and tampering techniques.

[0006] Existing deep forgery detection methods based on large language models and multimodal large models have two main drawbacks. On the one hand, they heavily rely on large-scale, high-quality labeled datasets for task-specific fine-tuning, requiring significant investment in dataset construction and model training. Furthermore, the fine-tuned models lack flexibility and are difficult to implement in practical applications. On the other hand, these methods lack effective guidance from domain-specific forensic knowledge during the detection process, failing to fully leverage the advantages of multimodal large models in visual perception, semantic understanding, and logical reasoning. The detection accuracy and interpretability of the results still have considerable room for improvement. Summary of the Invention

[0007] To address the shortcomings of current deepfake detection methods, such as insufficient generalization ability, heavy reliance on fine-tuning of labeled datasets, and inadequate interpretability of detection results, this invention proposes a knowledge-enhanced multimodal large-scale deepfake detection method and system. The aim is to achieve deepfake detection without training, improve the model's generalization ability to unknown forgery types, and integrate image authenticity determination, forgery generation method inference, and tampered region localization into a unified detection process. It also outputs interpretable detection conclusions and ensures compatibility with mainstream multimodal large-scale models.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following:

[0009] A knowledge-enhanced multimodal large model deepfake detection method, the method comprising: Construct an instance dataset containing real image samples and fake image samples; Based on the test image, several instance data are retrieved from the instance dataset, and the authenticity discrimination logic is obtained based on the instance data and its label; among them, the several instance data include at least one real image sample and at least one deepfake image sample; The test image, structured anti-counterfeiting prior knowledge, and authenticity discrimination logic are encoded and fused to obtain knowledge-enhanced structured prompts. The structured anti-counterfeiting prior knowledge includes low-order forgery features, high-order forgery features, and model-specific features of the forged image. The low-order forgery features are typical visual traces introduced after the image is generated or tampered with, and the high-order forgery features are typical semantic inconsistencies introduced after the image is generated or tampered with. After converting the knowledge-enhanced structured prompt into an input format recognizable by the multimodal large model, it is input into the multimodal large model to perform multi-dimensional reasoning and obtain the detection result of the test image; wherein, the dimensions include anomaly detection, forgery generation method reasoning, and tampered region localization.

[0010] Furthermore, the tags of the instance data include binary tags and fine-grained tags. The binary tags are used to label the authenticity of the image, and the fine-grained tags are used to label the source of the image.

[0011] Furthermore, based on the test images, several instance data points are retrieved from this instance dataset, including: Extract the visual feature vector of the test image; wherein the visual feature vector contains texture information, structural information and semantic information; Construct an instance dataset feature library based on visual feature vectors from instance data; Based on the visual feature vector of the test image, similarity matching is performed in the feature library of the instance dataset to obtain at least one real image sample and at least one deepfake image sample with the highest similarity to the test image.

[0012] Furthermore, based on the instance data and its labels, a logic for determining authenticity is generated, including: The label is embedded in the prompt information in the form of text, and together with the task instruction, they constitute the prompt text; wherein, the task instruction is used to instruct the multimodal large language model to learn the true and false discrimination logic; After fusing and encoding the instance data and prompt text to form an input format recognizable by the multimodal large model, the data is input into the multimodal large model to obtain the authenticity discrimination logic. The authenticity discrimination logic integrates the difference patterns of visual features between real image samples and fake image samples, the feature distribution rules corresponding to different generation or tampering methods, the consistency discrimination criteria between local areas and overall semantics of the image, and the inherent relationship between the image authenticity label and the corresponding features of the instance image sample.

[0013] Furthermore, the low-level forgery features include noise distribution patterns, lighting and shadow angles, texture consistency, boundary blurring, and pixel splicing and synthesis traces; The high-level forgery features include logical contradictions, causal anomalies, and inconsistencies in scene elements; The model-specific features include local smoothness, repetitive texture, pixel sharpness, and / or structural blur residue.

[0014] Furthermore, the reasoning process for the anomaly detection dimension includes: Guided by structured prior knowledge for counterfeit detection, and combined with the logic of authenticity discrimination, this method captures anomalous signals in test images from three dimensions: visual information, semantic information, and cross-modal consistency. Specifically, in the visual information dimension, visual anomalous signals in counterfeit image samples are identified by comparing the differences in texture continuity, noise distribution, and boundary smoothness between real and counterfeit image samples. In the semantic information dimension, semantic anomalous signals in counterfeit image samples are identified by comparing the differences in logical consistency and causal relationships between real and counterfeit scenes. In the cross-modal consistency dimension, consistency anomalous signals are obtained by analyzing the consistency between visual and semantic information. The abnormal signal is marked, and its location and characteristic information are recorded.

[0015] Furthermore, the reasoning process of the forgery generation method's reasoning dimension includes: The characteristic information of the abnormal signals was analyzed; The analysis results are matched with model-specific features, and the feature distribution patterns of different forgery types in the instance data are combined to determine the similarity between the feature patterns of the test images and known forgery mechanisms. Based on this similarity, the reasoning result of the forgery generation method for the test image is obtained; wherein, the reasoning result of the forgery generation method includes whether the test image is a real image or a forged image, and the forgery generation method of the test image when the test image is a forged image.

[0016] Furthermore, the reasoning process for tampering with the regional positioning dimension includes: For test images identified as forged, information on the location of the altered area is generated in natural language based on the location of the abnormal signal.

[0017] Furthermore, the process of generating the detection result includes: The detection information from different dimensions of reasoning is organized in a structured manner to form a structured set of detection evidence; Based on the logic of distinguishing between true and false, consistency analysis and mutual verification are performed on various features in the detection evidence set to obtain cross-validation results; Based on the cross-validation results, high-confidence evidence is obtained from the detection evidence set, and the authenticity of the test image is determined based on this high-confidence evidence. Based on the determination of authenticity, high-confidence evidence is linked and explained to form a complete and interpretable chain of reasoning; The detection results are output according to a preset structured format. The detection results include image authenticity determination, forgery generation method tracing, forgery area location information, and interpretable reasoning chain.

[0018] A knowledge-enhanced multimodal large-scale deepfake detection system, the system comprising: The dataset building module is used to construct instance datasets containing both real image samples and fake image samples; The instance guidance module is used to retrieve several instance data from the instance dataset based on the test image, and to obtain the authenticity judgment logic based on the instance data and its label; wherein, the several instance data include at least one real image sample and at least one deepfake image sample; The knowledge enhancement module is used to encode and fuse the test image, structured anti-counterfeiting prior knowledge, and authenticity discrimination logic to obtain knowledge-enhanced structured prompts. The structured anti-counterfeiting prior knowledge includes low-order forgery features, high-order forgery features, and model-specific features of the forged image. The low-order forgery features are typical visual traces introduced after the image is generated or tampered with, and the high-order forgery features are typical semantic inconsistencies introduced after the image is generated or tampered with. The model reasoning module is used to convert the knowledge-enhanced structured prompts into an input format recognizable by the multimodal large model, and then input them into the multimodal large model to perform multi-dimensional reasoning to obtain the detection results of the test image; wherein, the dimensions include anomaly detection, forgery generation method reasoning, and tampering region localization.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects.

[0020] (1) This invention adopts a training-free detection paradigm, which does not rely on large-scale, high-quality labeled datasets or parameter fine-tuning, greatly reducing the need for dataset construction and model training costs for deepfake detection methods. At the same time, it avoids the problems of insufficient model flexibility and difficulty in practical application after fine-tuning, and adapts to the detection needs of real-world scenarios.

[0021] (2) The present invention designs an instance guidance module, which adaptively retrieves typical real-fake image pairs as reference instances and guides the multimodal large model to learn the image authenticity discrimination logic by combining context-aware prompting strategy. Ablation experiments have confirmed that the instance guidance mechanism can significantly improve the detection accuracy of the model and at the same time greatly reduce its rejection rate, so that the model can stably perform the image authentication task.

[0022] (3) The present invention introduces a knowledge enhancement module, which injects the domain prior knowledge of image forgery detection into the detection process, giving full play to the advantages of multimodal large models in visual perception, semantic understanding and logical reasoning, and can systematically guide it to identify forgery traces from multiple dimensions, thus solving the problem that existing detection methods based on multimodal large models lack effective domain professional knowledge guidance.

[0023] (4) This invention achieves the integrated task objective of deepfake detection. It can not only accurately determine the authenticity of images, but also infer the forgery generation method and locate the tampered area. At the same time, it generates interpretable reasoning basis based on the detected forgery traces. Compared with the existing detection methods that only achieve binary classification, the detection dimensions are more comprehensive and the credibility and reliability of the results are higher.

[0024] (5) This invention has good compatibility and generalization ability. It can be deployed on various mainstream multimodal large models such as Deepseek-VL2, Llama-3.2-Vision, and GPT series. It can maintain stable detection performance on different types of datasets and has a certain generalization ability for unknown types of forged images, thus solving the problem of insufficient generalization ability of traditional deep learning-based detection methods. Attached Figure Description

[0025] Figure 1 This is a framework diagram of a knowledge-enhanced multimodal large-model deep forgery detection method.

[0026] Figure 2 This is a block diagram of the instance bootstrapping module.

[0027] Figure 3 This is a block diagram of the knowledge enhancement module.

[0028] Figure 4 The present invention presents the ROC curves of image authenticity determination experiments conducted on five mainstream multimodal large models: Deepseek-VL2, Llama-3.2-Vision, GPT-4o-mini, GPT-4o, and GPT-4.1. Detailed Implementation

[0029] To make the technical features, advantages, and effects of the present invention clearer and easier to understand, they will be described in detail below with reference to the accompanying drawings.

[0030] This invention specifically provides an instance-guided multimodal large-model deepfake detection method, the overall framework of which is as follows: Figure 1 As shown, this method follows a three-tiered progressive technical paradigm of instance-guided, knowledge-enhanced deep analysis, and interpretable joint reasoning, with its core consisting of two main modules: the instance-guided module and the knowledge-enhanced module. This invention eliminates the need for large-scale data training or parameter fine-tuning of multimodal large-scale models. By adapting typical real-forged reference instance pairs to the images to be detected, it guides the model to autonomously learn the logic for judging image authenticity. Simultaneously, the knowledge-enhanced module embeds prior knowledge from the field of image forgery detection into the detection process, fully leveraging the comprehensive advantages of multimodal large-scale models in visual perception, semantic understanding, and logical reasoning to achieve multi-dimensional and accurate detection of deepfake images. This invention not only performs binary classification of images to determine authenticity but also enables reasoning for forgery generation methods, localization of tampered regions, and generation of interpretable judgment criteria based on detected forgery traces. It also possesses good architectural portability, allowing deployment on various mainstream multimodal large-scale models while maintaining stable detection performance. This effectively solves the problems of insufficient generalization ability, reliance on labeled datasets for fine-tuning, and poor interpretability of detection results in existing deepfake detection methods.

[0031] The technical solution of this invention is based on an instance dataset, with an instance guidance module and a knowledge enhancement module as the core, and a multimodal large model as the reasoning carrier. The overall execution process is divided into several stages, including instance dataset construction, instance-guided feature learning, knowledge enhancement deep analysis, and model reasoning. Each stage is interconnected and works synergistically to ultimately achieve efficient, universal, and interpretable deep forgery detection. The specific implementation details of each stage and core module are described in detail below.

[0032] (a) Dataset construction module.

[0033] The instance dataset is the foundation for the instance-guided module to achieve adaptive retrieval in this invention, and also provides learning samples for MLLM's image authenticity discrimination logic. Its construction principles are instance diversity, representativeness, hierarchical annotation, and scalable types. The specific construction and preprocessing steps are as follows: (1) To improve the detection robustness of the model in real adversarial scenarios, this invention constructs a basic reference image sample by collecting real images, real images captured by cameras, deepfake images generated by generative adversarial models and diffusion models from public datasets. and This enables the model to learn comprehensive and diverse feature distribution patterns; (2) A two-level hierarchical labeling scheme is adopted for the basic image samples. The labeling results correspond one-to-one with the sample images and are used as the core information of the reference instance to input into the multimodal large model. The first level sets binary basic labels, labeling real images as <real> and deepfake images as <fake>, providing the model with the basic basis for distinguishing between real and fake. Among them, when the first level label is <real>, its corresponding second level label is <dataset> or <shot>, indicating that the image is a real image collected from a public dataset or taken by a camera; when the first level label is <fake>, its corresponding second level label is further subdivided into <adversarial-global>, <adversarial-local>, <diffusion-global> and <diffusion-local>, indicating that the image is synthesized as a whole or locally forged by the generative adversarial model or diffusion model. This structure makes the value range of the second level label completely determined by the first level label, thus forming a logically strict one-to-one hierarchical constraint between the two levels of labeling, and the fine-grained label supports expansion to adapt to new image forgery technologies; (3) Perform unified standardized preprocessing on the selected real images and deepfake images, complete the image resolution normalization and format unification operations, eliminate the interference of irrelevant image features on retrieval and detection, and ensure the accuracy of feature matching during the retrieval process of the instance guidance module.

[0034] (ii) Instance guidance module.

[0035] like Figure 2 The instance guidance module is one of the core modules of this invention. Its design is based on the principle of Intermediate Context Learning (ICL). The retrieval process uses the feature similarity with the test image as the core basis, so that the retrieved reference instances are representative and can provide effective discrimination references for multimodal large models. That is, it adaptively retrieves a set of typical reference instances from the instance dataset for the test image. (Two real images + two deepfake images) and by constructing a context-aware prompting strategy, the reference instances and corresponding label information are input into the MLLM to guide the model to autonomously capture and learn the inherent discrimination logic between real images and deepfake images, avoiding problems such as unclear task orientation and refusal to respond due to lack of dedicated task guidance.

[0036] The specific implementation steps for this module are as follows.

[0037] (S1) Input the image to be tested into the instance guidance module. This module extracts the visual features of the test image through a feature extraction network and generates a feature vector containing core information such as texture, structure, and semantics, which serves as the matching basis for instance retrieval.

[0038] (S2) Perform the same feature extraction operation on the samples in the preprocessed instance dataset to generate feature vectors, build an instance dataset feature library, and store the feature vectors in association with the sample images and two-level annotation labels to achieve a one-to-one mapping between features and instance information.

[0039] (S3) Using the feature vector of the test image as a benchmark, perform similarity matching in the feature library of the instance dataset, and retrieve the two real image samples and two deepfake image samples with the highest feature similarity to the test image to form a reference instance pair. Ablation experiments have verified that the "2 true + 2 false" sample configuration can provide sufficient information for comparison between true and false samples while avoiding feature noise and information redundancy, thus achieving a balance between detection accuracy, robustness, and inference efficiency.

[0040] (S4) Construct structured context-aware prompts based on the output reference instances. The reference instances include instance images and their corresponding two-level annotations. The labels are embedded in the prompt information as text, forming the prompt text together with the task instructions; the feature vectors are used for similarity calculation within the instance guidance module. Furthermore, the reference instance images and prompt texts are fused and encoded to form an input format recognizable by the multimodal large model, providing clear task guidance for model feature learning.

[0041] Through the structured prompting process described above, the instance guidance module outputs an implicit logic for authenticity determination. This logic deeply integrates the differences in visual features between real and deepfake images, the feature distribution patterns corresponding to different generation or tampering methods, the consistency criteria between local image regions and overall semantics, and the intrinsic relationship between image authenticity labels and their corresponding features. Based on this, the multimodal large model autonomously extracts and learns the implicit logic from reference instance images and their label information through a context learning mechanism, effectively transferring it to the analysis process of the image to be detected, thereby ultimately achieving accurate image authenticity determination and identification of specific generation methods.

[0042] Furthermore, the instance guidance module is designed based on the principle of contextual learning. The retrieval process takes the feature similarity with the test image as the core basis, so that the retrieved reference instances are representative and can provide effective discrimination references for multimodal large models.

[0043] (III) Knowledge Enhancement Module.

[0044] like Figure 3 The knowledge enhancement module is another core module of this invention. It is a prompt enhancement mechanism based on a multimodal large model, used to structure and guide the input information. This module first fuses and encodes key information such as the image to be detected, structured prior knowledge for authentication, and the authenticity judgment logic output by the instance guidance module, through task-oriented prompts, thereby obtaining knowledge-enhanced structured prompts. P Secondly, an input format recognizable by the multimodal large model is formed, and the entire input is fed into the multimodal large model to perform multidimensional inference. The authenticity discrimination logic is implicitly expressed through reference instance images and their corresponding annotation information, used to characterize the feature difference patterns and their correlations between real images and deepfake images. Its core design idea is to embed prior knowledge of authentication into the deepfake detection process by constructing task-oriented instruction prompts, guiding the MLLM to combine the discrimination logic learned from reference instances to perform visual, semantic, and cross-modal consistency analysis on the test images, comprehensively detecting forgery traces in the images, and accumulating sufficient evidence support for the final inference decision. The specific implementation of this module consists of three steps: construction of prior knowledge in the authentication domain, generation of task instruction prompts, and multidimensional deep analysis, detailed as follows: (P1) Construction of prior knowledge in the field of counterfeit detection.

[0045] Prior knowledge for forgery detection is based on the principles of image forgery, common image tampering traces, and unique identifiers for different generative models. This type of knowledge constructs structured text prompts through task instructions, guiding multimodal large models to fully utilize their visual perception, semantic understanding, and logical reasoning capabilities.

[0046] This embodiment integrates professional knowledge in the field of image forgery detection and constructs a structured prior knowledge system for forgery detection. This system provides professional guidance for MLLM to conduct in-depth analysis and mainly includes three categories of knowledge: 1) Common low-level forgery features: typical visual traces introduced after the image is generated or tampered with, such as noise distribution patterns, lighting and shadow angles, texture consistency, abnormally blurred boundaries, and traces of pixel splicing and synthesis. 2) Common high-order forgery features: typical semantic inconsistencies introduced after the image is generated or tampered with, such as logical contradictions, causal anomalies, and incongruities of scene elements; 3) Generate model-specific features: Due to different generation mechanisms, models (such as diffusion models and generative adversarial networks) will inevitably introduce model-specific features during image generation. These model-specific features include local smoothness, repetitive textures, pixel sharpness, and / or structural blur residues, such as excessive smoothing of local textures in images generated based on diffusion models.

[0047] (P2) Task instruction prompt generation.

[0048] The structured prior knowledge for counterfeit detection is transformed into natural language task prompts that MLLM can understand. These prompts are designed in layers and correspond one-to-one with the subsequent multi-dimensional in-depth analysis steps. The prompts include analysis task requirements, detection and judgment criteria, feature recognition guidance, and result output format. At the same time, the reference instance information output by the instance guidance module is integrated into the prompts, requiring MLLM to combine the reference instances and perform targeted logical reasoning and analysis according to the instructions.

[0049] (iv) Model reasoning module.

[0050] Guided by task instructions, MLLM, combined with the implicit authenticity discrimination logic in the reference example, performs in-depth analysis of the test image from three dimensions: anomaly detection, forgery generation method inference, and tampered region localization. The authenticity discrimination logic characterizes the feature difference patterns between real and deepfake images, providing comparative basis and discrimination constraints at each stage. Simultaneously, the analyses of each dimension are interconnected and progressive, accumulating comprehensive and effective evidence for detection decisions.

[0051] Visual, semantic, and cross-modal anomaly detection: This is a crucial step in deep analysis. Guided by prior knowledge of forgery detection, MLLM constructs an analogical reasoning chain by combining the authenticity judgment logic in the examples. It compares and analyzes the feature differences between the test image and the reference example, capturing anomalous signals in the test image from three dimensions: visual information, semantic content, and cross-modal consistency. In the visual dimension, by comparing the differences between real and fake images in terms of texture continuity, noise distribution, and boundary smoothness, it identifies forgeries such as noise inconsistencies, boundary continuity, pixel splicing traces, and lighting consistency. In the semantic content dimension, by comparing the differences between real and fake scenes in terms of logical consistency and causal relationships, it identifies semantic anomalies such as logical contradictions, causal relationships, and scene element inconsistencies. In the cross-modal consistency dimension, it further analyzes the consistency of visual and semantic information, and marks all detected anomalous signals, recording their location and feature information.

[0052] Forgery generation method reasoning: Based on anomaly detection, the MLLM is guided to sort out the detected abnormal features and match them with the generative model feature analysis in the prior knowledge of forgery detection; by using the feature distribution patterns corresponding to different forgery types in the reference examples, the similarity between the feature patterns of the test image and the known forgery mechanisms is judged, thereby inferring its possible generation or tampering methods and outputting the corresponding reasoning basis.

[0053] Tampered Region Location: For test images determined to be forged, MLLM, constrained by the authenticity discrimination logic and combined with the abnormal signals marked in the detection phase, accurately identifies the tampered regions in the image, and outputs the coordinates, range, and other location information of the tampered regions in natural language. For depth-forged images generated entirely as a whole, they are determined to be entirely generated.

[0054] Interpretable decision generation is the core output of the detection process in this invention, and it is completed collaboratively during the multi-modal large-scale model's multi-dimensional analysis and reasoning. Specifically, this step unifies the true / false discrimination logic obtained in the instance guidance phase with the abnormal features, generation methods, and tampered area information obtained in the deep analysis phase. By constructing a structured reasoning prompting strategy, it guides the multimodal large-scale model to follow a closed-loop detection and reasoning process of "evidence review - cross-validation - logical judgment - basis generation," as detailed below: 1) Evidence Sorting: The detection information from different analysis stages is structured and organized. Specifically, this includes visual, semantic, and cross-modal anomaly features acquired during the anomaly detection stage, the reasoning results of the generation method, and the spatial distribution information of the tampered areas. Simultaneously, combining the implicit truth-or-falseness judgment logic in the reference examples, the above evidence is categorized and its features aligned, mapping evidence from different sources to a unified feature space to form a structured detection evidence set. 2) Cross-validation: Based on the learned truth-discrimination logic, the model performs consistency analysis and cross-validation on various features in the detection evidence set. Specifically, it verifies the similarity between abnormal features and known forgery patterns, the matching degree between the inference results of the generation method and the distribution of abnormal features, and the consistency between the tampered area and the spatial distribution of abnormal signals. Through the mutual constraints between multiple pieces of evidence, inconsistent or low-confidence inference paths are filtered out, improving the reliability of the detection results. 3) Logical Judgment: After verifying the consistency of evidence, the multimodal large model, based on the retained high-confidence evidence, determines the authenticity of the test images and further identifies their possible forgery types and generation methods. This process is reflected in the reasoning mapping from the "feature layer" to the "semantic layer," that is, the deduction from low-level abnormal features to high-level semantic conclusions; 4) Evidence generation: Based on the completed judgment result, the multimodal big model explicitly expresses the reasoning process and explains the key evidence on which the judgment is based, including the source of abnormal features, the basis for the generation method matching, and the supporting information of the tampered area, thereby forming a complete interpretable reasoning chain. 5) Decision Output: Outputs detection results according to a preset structured format, including image authenticity determination, forgery generation method tracing, forgery region location information, and corresponding reasoning basis. This provides both detection conclusions and interpretable expression of the determination process, alleviating the opaque nature of traditional detection methods.

[0055] The invention will be further demonstrated by an experiment below.

[0056] (1) Data set.

[0057] To evaluate the robustness and generalization ability of this invention, five different types of image datasets were selected for experiments: Caltech_WebFaces (real face dataset), NeuralTextures (GAN-based fake face dataset), DiffusionFace (diffusion model-based fake face dataset), HiSD (GAN-based image local manipulation dataset), and Autosplice (diffusion model-based image local manipulation dataset). 1000 images were randomly sampled from each dataset to construct a unified test set. The test set included real images, overall generated images, and locally fake images, basically covering the types of deepfake images in real-world scenarios.

[0058] (2) Evaluation indicators.

[0059] To objectively evaluate the method's performance, the experiment used detection accuracy (Acc) and model rejection rate (Rej) as evaluation metrics, further incorporating ROC curves and AUC values ​​to avoid bias caused by data differences. For localization evaluation, a custom localization accuracy (Acc) was used. Verify the effectiveness of the model's localization: in, For the test image, the first The identification result for each tampered area is 1 for correctly identified and 0 for incorrectly identified. For the first The location accuracy score of each tampered area takes a value in the interval [0,1]. This represents the total number of actually tampered areas in the test image.

[0060] The positioning accuracy is calculated through a multimodal large model evaluation system. Specifically, the test image, the real tampered mask, the tampered area description and location information output by the model are input into the evaluation multimodal large model, which scores the accuracy of the tampered area description and the precision of the spatial location description, and then calculates the final positioning accuracy based on the above formula.

[0061] (3) Comparison methods and transferability verification.

[0062] First, the experiment selected the current mainstream deep neural network-based forgery detection methods FreDect, GramNet, and CNNSpot for performance comparison to verify the performance advantages of the present invention.

[0063] Secondly, transferability tests were conducted on mainstream multimodal large models with different architectures. Specifically, the technical framework of this invention was deployed on five mainstream multimodal large models: Deepseek-VL2, Llama-3.2-Vision, GPT-4o-mini, GPT-4o, and GPT-4.1, to test the model judgment accuracy under different architectures.

[0064] (4) Experimental setup.

[0065] This invention uses GPT-4.1 as the base model and implements model calls through the official API. In transferability testing, Deepseek-VL2 and Llama-3.2-Vision are open-source models, deployed locally according to official guidelines; GPT-4o-mini and GPT-4o are called through their respective official APIs. All experiments maintain a consistent input format, prompting strategy, and evaluation protocol to ensure fairness and reproducibility in experimental comparisons. Furthermore, 20 independent detection results are collected for each test image, and the average value is taken as the final result.

[0066] (5) Experimental results and analysis.

[0067] The experiment successfully completed multi-dimensional testing and verification of detection performance, positioning accuracy, ablation experiments, and migration tests.

[0068] A. Method Performance Comparison: The performance test results are shown in Table 1. The present invention, deployed on GPT-4.1, significantly outperforms traditional deepfake detection methods such as FreDect, GramNet, and CNNSpot on the unified test set in terms of average detection accuracy. Traditional detection methods show low accuracy and insufficient generalization ability on fake image datasets generated by diffusion models; while the present invention achieves image authenticity determination accuracy of 100.0% and 91.2% on the DiffFace and Autosplice datasets, respectively. Experimental results verify that the instance guidance mechanism proposed in this invention can guide the model to fully learn the inherent discrimination logic of images based on contextual understanding. Simultaneously, the knowledge enhancement module integrates prior knowledge from the fakefake detection domain into the detection process, fully utilizing the reasoning ability of MLLM to achieve more accurate image authenticity determination.

[0069] B. Positioning Function Test: To verify the fake area positioning function of this invention, the positioning accuracy was tested under different base models. The results are shown in Table 2. The positioning accuracy of this invention based on GPT-4.1 is ( The localization accuracy reached 67.3% and 66.9% on the HiSD and Autosplice datasets, respectively, significantly higher than the test results of other pedestal models. Because the multimodal large model outputs results in text form, the localization accuracy of this invention is slightly lower than that of pixel-level detection methods in terms of numerical presentation, but it can still achieve basic interpretable localization, achieving a good balance between localization accuracy and interpretability.

[0070] C. Ablation Test of Instance Guidance Mechanism: To verify the effectiveness of the instance guidance mechanism in this invention, two types of ablation experiments were designed: instance composition and instance selection mechanism. The results of the instance composition ablation experiment are shown in Table 3: Without instance guidance, the model rejection rate is as high as 82.3%, making it almost impossible to make stable decisions; after introducing a reference instance pair of "1 true + 1 false", the model rejection rate drops to 0%, and the accuracy increases to 92.0%; however, the instance combination of "3 true + 3 false" introduces noise, causing a slight decrease in detection accuracy. Therefore, this invention ultimately selected the instance combination of "2 true + 2 false" to achieve the optimal balance between model detection cost and accuracy. The results of the instance selection method ablation experiment are shown in Table 4. As can be seen from the experimental results, compared with randomly selected instances, the instance matching mechanism proposed in this invention can achieve an average detection accuracy of over 70% on various datasets when performing detection based on GPT4.1, fully demonstrating the effectiveness of the instance dataset constructed and the instance guidance module designed in this invention.

[0071] D. Transferability Testing: This invention conducted image authenticity determination experiments on five mainstream multimodal large models: Deepseek-VL2, Llama-3.2-Vision, GPT-4o-mini, GPT-4o, and GPT-4.1. The ROC curves are shown below. Figure 4 As shown in the figure, the experimental results show that the AUC value of this invention is significantly higher than 0.5 under all architectures, demonstrating an accuracy superior to random guessing. Specifically, the AUC value reaches 0.903 based on GPT-4.1, which verifies that this invention can fully utilize the capabilities of high-performance multimodal large models to achieve accurate detection of deepfake images.

[0072] Table 1 Comparison of Image Detection Accuracy Results Table 2 Comparison of Positioning Accuracy Table 3. Results of ablation experiments in the example configuration Table 4. Results of ablation experiments using the instance screening mechanism The detection performance of this method is comprehensively evaluated through detection accuracy, rejection rate, ROC curve, and AUC value. To eliminate the influence of randomness in multimodal large model inference, multiple independent detections are performed on each test image, and the average value is taken as the final evaluation data. The above experimental results fully verify the effectiveness of the present invention. The present invention not only significantly outperforms existing methods in terms of detection accuracy and generalization ability for deepfake images, but also enables forgery generation method tracing, tampered region localization, and interpretable inference. Furthermore, it exhibits stable transferability across various architectures of multimodal large models, effectively compensating for the shortcomings of existing deepfake detection methods.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A knowledge-enhanced multimodal large-scale deepfake detection method, characterized in that, The method includes: Construct an instance dataset containing real image samples and fake image samples; Based on the test image, several instance data are retrieved from the instance dataset, and the authenticity discrimination logic is obtained based on the instance data and its label; among them, the several instance data include at least one real image sample and at least one deepfake image sample; The test image, structured anti-counterfeiting prior knowledge, and authenticity discrimination logic are encoded and fused to obtain knowledge-enhanced structured prompts. The structured anti-counterfeiting prior knowledge includes low-order forgery features, high-order forgery features, and model-specific features of the forged image. The low-order forgery features are typical visual traces introduced after the image is generated or tampered with, and the high-order forgery features are typical semantic inconsistencies introduced after the image is generated or tampered with. After converting the knowledge-enhanced structured prompt into an input format recognizable by the multimodal large model, it is input into the multimodal large model to perform multi-dimensional reasoning and obtain the detection result of the test image; wherein, the dimensions include anomaly detection, forgery generation method reasoning, and tampered region localization.

2. The method according to claim 1, characterized in that, The tags for the instance data include binary tags and fine-grained tags. The binary tags are used to label the authenticity of the image, and the fine-grained tags are used to label the source of the image.

3. The method according to claim 1, characterized in that, Based on the test image, several instance data points were retrieved from this instance dataset, including: Extract the visual feature vector of the test image; wherein the visual feature vector contains texture information, structural information and semantic information; Construct an instance dataset feature library based on visual feature vectors from instance data; Based on the visual feature vector of the test image, similarity matching is performed in the feature library of the instance dataset to obtain at least one real image sample and at least one deepfake image sample with the highest similarity to the test image.

4. The method according to claim 1, characterized in that, Based on instance data and its labels, a logic for determining authenticity is generated, including: The label is embedded in the prompt information in the form of text, and together with the task instruction, they constitute the prompt text; wherein, the task instruction is used to instruct the multimodal large language model to learn the true and false discrimination logic; After fusing and encoding the instance data and prompt text to form an input format recognizable by the multimodal large model, the data is input into the multimodal large model to obtain the authenticity discrimination logic. The authenticity discrimination logic integrates the difference patterns of visual features between real image samples and fake image samples, the feature distribution rules corresponding to different generation or tampering methods, the consistency discrimination criteria between local areas and overall semantics of the image, and the inherent relationship between the image authenticity label and the corresponding features of the instance image sample.

5. The method according to claim 1, characterized in that, The low-level forgery features include noise distribution patterns, lighting and shadow angles, texture consistency, boundary blurring, and pixel splicing and synthesis traces; The high-level forgery features include logical contradictions, causal anomalies, and inconsistencies in scene elements; The model-specific features include local smoothness, repetitive texture, pixel sharpness, and / or structural blur residue.

6. The method according to claim 1, characterized in that, The reasoning process for the anomaly detection dimension includes: Guided by structured prior knowledge for counterfeit detection, and combined with the logic of authenticity discrimination, this method captures anomalous signals in test images from three dimensions: visual information, semantic information, and cross-modal consistency. Specifically, in the visual information dimension, visual anomalous signals in counterfeit image samples are identified by comparing the differences in texture continuity, noise distribution, and boundary smoothness between real and counterfeit image samples. In the semantic information dimension, semantic anomalous signals in counterfeit image samples are identified by comparing the differences in logical consistency and causal relationships between real and counterfeit scenes. In the cross-modal consistency dimension, consistency anomalous signals are obtained by analyzing the consistency between visual and semantic information. The abnormal signal is marked, and its location and characteristic information are recorded.

7. The method according to claim 6, characterized in that, The reasoning process of the forgery generation method's reasoning dimension includes: The characteristic information of the abnormal signals was analyzed; The analysis results are matched with model-specific features, and the feature distribution patterns of different forgery types in the instance data are combined to determine the similarity between the feature patterns of the test images and known forgery mechanisms. Based on this similarity, the reasoning result of the forgery generation method for the test image is obtained; wherein, the reasoning result of the forgery generation method includes whether the test image is a real image or a forged image, and the forgery generation method of the test image when the test image is a forged image.

8. The method according to claim 7, characterized in that, The reasoning process for tampering with the regional positioning dimension includes: For test images identified as forged, information on the location of the altered area is generated in natural language based on the location of the abnormal signal.

9. The method according to claim 1, characterized in that, The process of generating the detection results includes: The detection information from different dimensions of reasoning is organized in a structured manner to form a structured set of detection evidence; Based on the logic of distinguishing between true and false, consistency analysis and mutual verification are performed on various features in the detection evidence set to obtain cross-validation results; Based on the cross-validation results, high-confidence evidence is obtained from the detection evidence set, and the authenticity of the test image is determined based on this high-confidence evidence. Based on the determination of authenticity, high-confidence evidence is linked and explained to form a complete and interpretable chain of reasoning; The detection results are output according to a preset structured format. The detection results include image authenticity determination, forgery generation method tracing, forgery area location information, and interpretable reasoning chain.

10. A knowledge-enhanced multimodal large-scale deepfake detection system, characterized in that, The system includes: The dataset building module is used to construct instance datasets containing both real image samples and fake image samples; The instance guidance module is used to retrieve several instance data from the instance dataset based on the test image, and to obtain the authenticity judgment logic based on the instance data and its label; wherein, the several instance data include at least one real image sample and at least one deepfake image sample; The knowledge enhancement module is used to encode and fuse the test image, structured anti-counterfeiting prior knowledge, and authenticity discrimination logic to obtain knowledge-enhanced structured prompts. The structured anti-counterfeiting prior knowledge includes low-order forgery features, high-order forgery features, and model-specific features of the forged image. The low-order forgery features are typical visual traces introduced after the image is generated or tampered with, and the high-order forgery features are typical semantic inconsistencies introduced after the image is generated or tampered with. The model reasoning module is used to convert the knowledge-enhanced structured prompts into an input format recognizable by the multimodal large model, and then input them into the multimodal large model to perform multi-dimensional reasoning to obtain the detection results of the test image; wherein, the dimensions include anomaly detection, forgery generation method reasoning, and tampering region localization.

Images