Multi-modal large model Deepfake detection method based on retrieval enhancement

Abstract

The invention discloses a multi-modal large model Deepfake detection method based on retrieval enhancement, and relates to the frontier crossing field of artificial intelligence security, computer vision and multi-modal information processing, and the method comprises the steps: 1, carrying out two-stage labeling in sequence, constructing an evidence obtaining knowledge database through the first-stage labeling, constructing an evidence obtaining thinking chain data set through the second-stage labeling, and carrying out the second-stage labeling; wherein the evidence obtaining thinking chain data set is used for generating thinking chain data including a cross validation type, an evidence guide correction type and an adversarial sample type according to the validity of initial judgment and retrieval evidences; step 2, for an image to be detected, retrieving text evidences of Top-k similar cases from the evidence obtaining knowledge database by using a dynamic evidence obtaining retriever; and step 3, inputting an image to be detected and the retrieved text evidence into the critical reasoning multi-modal large model, performing multi-modal reasoning analysis, and outputting a detection result containing a reasoning process.

Description

TECHNICAL FIELD

[0001] The present application relates to the cutting-edge intersection of artificial intelligence security, computer vision, and multi-modal information processing, and in particular to a multi-modal large model Deepfake detection method based on retrieval enhancement. BACKGROUND

[0002] With the rapid development of generative artificial intelligence (AIGC) technology, deepfake technologies such as face swapping, facial reenactment, and diffusion model-based image generation have been able to create high-fidelity fake images that are difficult to distinguish from real images. While these technologies have shown great application potential in entertainment, film production, and other fields, they have also brought serious security risks, including privacy invasion, financial fraud, false information dissemination, and challenges to the credibility of judicial evidence. Therefore, as a core defense means, deepfake detection technology is particularly important.

[0003] Traditional deepfake detectors mainly rely on fine-tuning visual backbone networks (such as convolutional neural networks CNN or visual Transformers ViT) and enhancing generalization ability through fine network design. These methods usually aim to mine discriminative fake patterns (such as inconsistent facial textures, abnormal lighting distribution, frequency domain artifacts, etc.) to distinguish real and fake images. However, traditional deepfake detectors are essentially simple binary classifiers, usually built on fine-tuned visual backbone networks (such as CNNs or ViTs), aiming to mine discriminative fake patterns such as inconsistent facial textures or abnormal lighting, etc. However, such traditional methods have two key limitations: 1. "Black box" uninterpretability: These detectors are essentially simple binary classifiers that can only output probability results (such as "0.99 Fake"), but cannot generate interpretable text reasons and cannot answer user interaction needs such as "Why is this picture fake?" This uninterpretability is unacceptable in high-risk scenarios such as judicial forensics and content review; 2. Insufficient generalization ability: Traditional methods are prone to overfitting to limited and monotonous fake patterns. Although improved through data augmentation, performance often drops significantly when facing unknown generation algorithms or cross-domain data.

[0004] To address the above limitations, recent research has begun to introduce multimodal large models (MLLMs) into the deepfake detection task, aiming to leverage the rich pre-training knowledge and powerful semantic understanding and text generation capabilities of MLLMs to improve generalization and provide explanations for detection results. Existing MLLM-based methods can be mainly divided into two categories: 1. Hybrid Frameworks: This type of solution combines MLLMs with small DFD detectors, with the idea that small detectors are responsible for extracting low-level artifacts, while MLLMs are responsible for performing high-level semantic analysis and generating explanatory text. However, this approach results in redundant architecture, and the knowledge interaction between models is often one-way or shallow, and the knowledge obtained by the model after training is still static; 2. Static Knowledge Injection: This type of solution (such as KFD, M2F2d-Det) attempts to inject isolated keywords or attribute lists and other general fake knowledge into LLMs through Prompt Engineering and other methods. However, the injected knowledge is static and of low quality.

[0005] The above MLLM-based methods still have a serious "knowledge gap" problem in the Deepfake detection task, which is manifested in the following aspects: 1. Lack of professional forensic knowledge: general MLLMs lack the ability to identify fine-grained artifacts specific to certain fake traces, and static injected knowledge is often too general and cannot cope with complex and varied fake methods; 2. Hallucination problem is serious: in the absence of dynamic external reference information, MLLMs are prone to produce false descriptions that do not match the image content based on broad keywords or attribute lists; 3. Lack of critical reasoning ability: existing models are difficult to make independent logical judgments in the presence of noise interference or inaccurate reference information, and are easily misled by false external information.

[0006] Therefore, those skilled in the art are committed to developing a new multimodal large model Deepfake detection method to solve the above-mentioned defects in the prior art. SUMMARY

[0007] In view of the above defects of the prior art, the technical problem to be solved by the present application is how to solve the defects of high hallucination, no reference, and static knowledge in the prior art solution, and to realize low hallucination, high interpretability, and strong generalization ability of deepfake detection by constructing a new solution.

[0008] To achieve the above object, the application provides a multi-modal large model Deepfake detection method based on retrieval enhancement, comprising the following steps: Step 1, two-stage labeling is performed in sequence, the first stage labeling constructs a forensic knowledge database, wherein the forensic knowledge database stores true and false image pairs and corresponding structured text labels, the text labels include fake area positioning and fine-grained artifact description; the second stage labeling is to construct a forensic thought chain dataset based on the forensic knowledge database, wherein the forensic thought chain dataset is generated according to initial judgment and retrieval evidence effectiveness, and includes cross-validation type, evidence-guided correction type and adversarial sample type thought chain data; Step 2, for the image to be detected, a dynamic forensic retriever is used to retrieve Top-k similar case text evidence from the forensic knowledge database, wherein the dynamic forensic retriever is based on a deepfake training deepfake feature extraction network to calculate feature similarity; Step 3, the image to be detected and the retrieved text evidence are input into a critical reasoning multi-modal large model for multi-modal reasoning analysis, and a detection result containing a reasoning process is output.

[0009] Further, in the process of constructing the forensic knowledge database in step 1, a multi-modal large model is used as a labeling assistant, and a large-scale face forgery dataset is labeled in a comparative manner based on a predefined forgery guide, wherein a true and false image pair is input simultaneously during the labeling process to enhance the positioning accuracy of the forgery traces.

[0010] Further, the forgery guide covers several mainstream forgery technologies, including but not limited to DeepFakes, Face2Face, FaceSwap and Neural Textures.

[0011] Further, the step 1 of constructing the forensic knowledge database further comprises: Real samples and fake samples are extracted from the face forgery dataset and stored in a balanced manner at a ratio of 1:1, and a vector database is used to manage the correspondence between image features and text labels; The fine-grained artifact description includes descriptions of texture loss, boundary blur and abnormal illumination.

[0012] Further, the step 1 of constructing the forensic thought chain dataset comprises: A plurality of images are sampled, and the text evidence of Top-k similar cases of each sampled image except itself is retrieved from the forensic knowledge database; For each sample image, according to the initial judgment and the effectiveness of the retrieved text evidence, the sample is classified into three types, including the cross-validation type, the evidence-guided correction type and the adversarial sample type; For each type, a structured thinking chain annotation is generated, including preliminary visual analysis, RAG reference information analysis, fusion reasoning and decision making, and complete reasoning path of the final answer.

[0013] Further, the face forgery dataset is FF++ dataset, before starting the step 1, the FF++ dataset is divided by a multi-modal large model, a plurality of videos are randomly taken from the Train subset as training data of stage-1, and the remaining plurality of videos are taken as training data of stage-2 and stage-3, and a stage-1 model is obtained by training in the stage-1 stage, the stage-1 model has basic visual perception ability for forgery task.

[0014] Further, the stage-1 model is trained by lora using the forensic thinking chain dataset to teach the model to imitate the thinking process of experts, and a stage-2 model is obtained, wherein the retriever is the dynamic forensic retriever used in the step 2.

[0015] Further, when performing the lora training, the rank r=128, the scaling factor alpha=256, and the learning rate .

[0016] Further, the stage-2 model is trained by reinforcement learning using the training data of stage-3 to give the model real dynamic decision-making ability, and a stage-3 model, i.e. the critical reasoning multi-modal large model in the step 3, is obtained, and a group relative strategy optimization is adopted to encourage the model to think critically when facing conflicting evidence, and a process perception reward function is set, including: defining A as the correctness of the final answer, and defining C as whether there is a cognitive conflict, and the conflict is defined as: wherein, is the judgment tendency of the stage-1 model, is the label suggested by the strongest evidence retrieved by the stage-2 model.

[0017] Further, the final reward of the critical reasoning multi-modal large model is wherein, is the conflict reward, including critical victory, regular correct, regular error and critical failure; coefficient for format check, coefficient for conflict reward coefficient for conflict reward coefficient for format check coefficient for format check.

[0018] The multi-modal large model Deepfake detection method based on retrieval enhancement provided by the application has at least the following technical effects: 1. The technical solution provided by the application has strong generalization ability and SOTA performance. In cross-dataset tests such as CDF-v1, CDF-v2, FFIW and WildDeepfake (WDF), the scheme of the application achieves the optimal (SOTA) performance, and in particular on the challenging dataset WDF, the performance is significantly improved compared with traditional methods, indicating that it can effectively generalize to the types of forgeries not seen in training.

[0019] 2. The technical solution provided by the application provides a verifiable high-fidelity explanation. Through the RAG mechanism, the reasoning results output by the model are anchored to high-similarity historical expert cases (FKD), effectively suppressing the “illusion” of MLLM, improving the credibility and explainability of the detection results, and meeting the needs of high-demand scenarios such as forensic and content review.

[0020] 3. The technical solution provided by the application has a high-efficiency and flexible architecture. The RAG architecture realizes dynamic updating and scalability of knowledge. When new Deepfake technology appears, only the external FKD knowledge base needs to be updated, and the entire large MLLM base does not need to be retrained, so that the system can realize “lifelong learning”.

[0021] 4. The technical solution provided by the application has innovative critical reasoning ability. Through GRPO reinforcement learning and a customized conflict reward mechanism, the model not only receives evidence, but also has the ability to independently assess the effectiveness of evidence and reject misleading evidence, thereby realizing the paradigm shift from “pattern recognition” to “critical expert analysis”.

[0022] The concept, specific structure and technical effects of the application will be further described below with reference to the accompanying drawings, so as to fully understand the purpose, features and effects of the application. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a comparison diagram of the VRAG-DFD framework of a preferred embodiment of the application and prior art; Figure 2 is a whole system architecture and reasoning flowchart of the VRAG-DFD in a preferred embodiment of the application; Figure 3is a data set construction and annotation process schematic diagram in a preferred embodiment of the present application; Figure 4 is a three-stage training process schematic diagram in a preferred embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application can be embodied in a number of different forms, and the scope of the present application is not limited only to the embodiments mentioned in the text.

[0025] As shown in Figure 1 The embodiment of the present application discards the traditional hybrid architecture and static knowledge injection method, and provides a multi-modal large model Deepfake detection method based on retrieval enhancement (named VRAG-DFD), which reconfigures the Deepfake detection task into a comprehensive cognitive process of "retrieval-verification-reasoning", and is used for multi-modal forensic technology of Deepfake Detection (DFD) and Face Forgery Detection. In particular, the embodiment of the present application provides an innovative detection framework and method, which utilizes multi-modal large language models (MLLMs), retrieval augmented generation (RAG) mechanism and reinforcement learning (RL) technology to solve the limitations of traditional methods in generalization ability and explainability, so as to realize high-fidelity cross-dataset detection and structured forensic reasoning.

[0026] Specifically, the embodiment of the present application constructs a professional forensic knowledge base, dynamically retrieves historical cases highly similar to the to-be-detected image as evidence by using the retrieval augmented generation (RAG) technology, and converts the Deepfake detection task from a simple "classification task" into a "case-based forensic verification task", that is, when judging the authenticity of an image, not only relying on the internal parameter knowledge of the model, but also relying on the historical cases with similar forgery characteristics retrieved from the external professional knowledge base as a reference, and through the design of the progressive three-stage training strategy including basic visual alignment, critical thinking supervision fine-tuning (SFT) and autonomous reasoning reinforcement learning (RL-GRPO), the model learns critical reasoning, prevents the model from being misled by the retrieved false or irrelevant evidence, and enables the model to have autonomous reasoning ability to distinguish the authenticity of the evidence and handle cognitive conflicts, so as to realize low hallucination, high explainability and strong generalization ability of Deepfake detection.

[0027] Embodiment 1 As shown in Figure 2 The embodiment of the present application provides a multi-modal large model Deepfake detection method based on retrieval enhancement, which includes the following steps: Step 1, two-stage labeling is performed in sequence, the first stage labeling constructs a forensic knowledge database (FKD), wherein the forensic knowledge database stores true and false image pairs and their corresponding structured text annotations, the text annotations include forged region positioning and fine-grained artifact description; the second stage labeling is to construct a forensic thinking chain dataset based on the forensic knowledge database, wherein the forensic thinking chain dataset is generated according to the initial judgment and the effectiveness of the retrieved evidence, and includes thinking chain data of cross-validation type, evidence-guided correction type and adversarial sample type; Step 2, for the image to be detected, a dynamic forensic retriever is used to retrieve the text evidence of the top-k similar cases from the forensic knowledge database, wherein the dynamic forensic retriever is based on a deepfake-trained deepfake feature extraction network to calculate feature similarity; Step 3, input the image to be detected and the retrieved text evidence into a critical reasoning multi-modal large model for multi-modal reasoning analysis, and output the detection result containing the reasoning process.

[0028] In particular, as shown in Figure 3 The two-stage labeling of step 1 is completed by a data construction module. First, the FKD is constructed. A multi-modal large model (such as Gemini2.5-Pro or other high-capacity models) is used as a labeling assistant, and based on a pre-defined forgery guide, a large-scale face forgery dataset (FaceForensics++, FF++) is labeled in a comparative manner. During the labeling process, a true and false image pair (Pair) is input simultaneously to enhance the accuracy of the forgery trace positioning. For each picture in the database, input a fake image (Fake) to be labeled and a corresponding original real image (Real) to the labeling model, and input the labeling prompt: “You are a forensic expert. Please compare the two pictures, according to the above forgery guide, accurately point out the manipulated areas (such as eyes, mouth, contour) in the forged image and the specific forged artifact performance (such as high-frequency detail loss, boundary artifact). Please output according to the output format”. Finally, each piece of labeled text data and the corresponding image features are stored in a vector database, and each piece of text annotation includes label, forged region and corresponding forged trace.

[0029] In particular, the final labeling content includes forged region positioning, fine-grained artifact description (such as at least one of texture loss, boundary blur, and light abnormality), forming a structured “image-text evidence” pair, and stored in a vector database.

[0030] In particular, real samples and fake samples are extracted from a large-scale face forgery dataset and stored in a 1:1 ratio, and a vector database is used to manage the correspondence between image features and text labels. Specifically, 3 frames are taken from each fake video in the Train subset and 12 frames are taken from each real video to form a 1:1 ratio database containing 8000 real samples and 8000 fake samples.

[0031] In particular, the forgery guide covers a variety of mainstream forgery techniques such as DeepFakes, Face2Face, FaceSwap, Neural Textures, etc. Combined with literature analysis, the forgery technique characteristics of each of the four forgery types in FF++ are analyzed, and specific forgery trace guides are customized for each type.

[0032] Embodiment 2 Based on embodiment 1, step 1, the construction of the Forensic Chain-of-Thought Dataset (F-CoT) is based on the previously constructed FKD. By having the labeled teacher model simulate the retrieval generation process, training samples containing different types of evidence conflicts are generated, including: Sample multiple images and retrieve the text evidence of the Top-k similar cases of each sampled image from the forensic knowledge database, excluding the image itself; For each sampled image, classify the sample into three types including cross-validation type, evidence-guided correction type, and adversarial sample type according to the effectiveness of the initial judgment and the retrieved text evidence; For each type, generate a structured Chain-of-Thought (CoT) annotation, which includes preliminary visual analysis, RAG reference information analysis, fusion reasoning and decision-making, and the complete reasoning path of the final answer.

[0033] In particular, the cross-validation type specifically refers to the model's initial judgment being correct, and the retrieved evidence mixing correct and interference information; the evidence-guided correction type refers to the model's initial judgment being incorrect, and the retrieved evidence being effective correction information; the adversarial sample type refers to the model's initial judgment being incorrect, and the retrieved evidence being misleading information.

[0034] Step 2 is implemented by a Deepfake detection model architecture module. The Deepfake detection model architecture module includes a dynamic forensic retriever and a training optimization module. Among them, the dynamic forensic retriever does not directly use a general pre-trained model (such as CLIP), but uses a deepfake-trained deepfake feature extraction network as an embedding model to calculate the feature similarity between the query image and the case in the FKD, and retrieve the top-k most relevant text evidence to ensure the similarity of the retrieval results in the "fake mode" rather than just the similarity of the semantic content.

[0035] Embodiment 3 On the basis of Embodiment 1 or 2, as shown in Figure 4 , the FF++ dataset is selected as the training data and the data source of the database, that is, the face fake dataset is the FF++ dataset. Before starting Step 1, use an open-source multi-modal large model such as Qwen2.5-VL or InternVL as a basic model to divide the FF++ dataset. Randomly take a number of videos from the Train subset as training data for stage-1, and take the remaining number of videos as training data for stage-2 and stage-3, and perform stage-1 training to obtain a stage-1 model. The stage-1 model has basic visual perception ability for fake tasks. Specifically, 2500 videos are randomly taken from the Train subset as training data for stage-1, and the remaining 1000 videos are taken as training data for stage-2 and stage-3.

[0036] In particular, all frames of the training videos in stage-1 are used for stage-1 training, in order to enable the model to have a solid visual perception ability. This step only needs to enable the model to "understand" the basic features of the fake image, without the need for complex reasoning. Therefore, the fine-tuning data format is the standard visual question answering (VQA), Prompt: "Is this picture real or fake?" Answer: "Fake / Real". The learning rate of this stage is , and the Batch Size is 512. In this way, a stage-1 model is obtained. The stage-1 model has basic binary classification ability and visual perception ability, but lacks interpretability and anti-interference ability.

[0037] In particular, the stage-1 model is trained with the forensics chain-of-thought dataset to teach the model to mimic the thought process of an expert, resulting in a stage-2 model. This is to make the retriever's representation carry the fake features and retain a large amount of pre-trained knowledge to ensure generalizability. The retriever is the dynamic forensics retriever that will be used in step 2. The training process is to input the image and the retrieved evidence to the model, which is forced to output the complete CoT path. After this stage, the stage-2 model learns the specific output format and how to cite external evidence. When performing lora training, the rank r = 128, the scaling factor alpha = 256, and the learning rate is .

[0038] Example 4 On the basis of example 3, in particular, 1000 videos are randomly sampled from FF++ again, 9 frames are taken from each video, and the dynamic forensics retriever is used to retrieve 5 similar picture corresponding text annotations from the FKD database for each picture. Specifically, the cosine similarity between the query image feature vector and the feature vector of all images in the FKD is calculated using the retriever. For each query picture, the top-k cases with the highest similarity are selected from the FKD database, and their corresponding text annotations (Textual Evidence) are extracted, resulting in As an input of fine-tuning data. Then for these 9000 pictures, first use the stage-1 model trained in the first stage to make predictions to get the initial prediction results, and then analyze the 9000 pictures and the retrieved text labels to analyze whether the external reference information is effective (use majority voting form, for example: if the picture to be detected is Fake, and more than 3 text labels retrieved from Real pictures are external information, it is considered that the reference information is misleading). According to the initial prediction results of the stage-1 model and the effectiveness of the external reference information, the 9000 sample data are divided into three types. Type one: if the stage-1 model initial judgment is correct, and the external reference information is mixed, then construct cross-validation type data. CoT logic example: “Preliminary observation found that the mouth is blurred (Fake). Evidence 1 retrieved supports this view, but evidence 2 calls it real. By comparison, it is found that the lighting conditions described by evidence 2 do not match the picture, so evidence 2 is excluded, and evidence 1 is adopted, maintaining the judgment of forgery.” Type two: if the stage-1 model initial judgment is wrong, but the evidence is effective, then construct error correction type CoT data. CoT logic example: “Preliminary observation found no obvious abnormalities (Real). But high-confidence evidence points out that there is a fake image above the eyebrows. After zooming in and checking again, it is indeed found that there is an unnatural blur at that place. Therefore, the judgment is corrected to be fake.” Type three: if the stage-1 model initial judgment is wrong, and the retrieved evidence is misleading information, then force the model to return to the internal knowledge with higher confidence. CoT logic example: “Preliminary observation found that the philtrum structure is missing (Fake). The evidence retrieved is all “real” cases because they look similar. But this similarity is just a similar appearance, and does not explain the problem of philtrum loss. This is a misleading evidence, I refuse to adopt it, and insist on judging as fake according to visual features.” The output of these three CoT data follows the following XML tag format:, <Preliminary Visual Analysis> first analyzes the image independently < / Preliminary Visual Analysis>. 2、<RAG Reference Information Analysis> analyze external reference information < / RAG Reference Information Analysis>. 3、<Fusion,Reasoning,andDecision> Comprehensive analysis of two information sources < / Fusion,Reasoning,and Decision>. 4、<Answer>Ground_Truth< / Answer>.

[0039] Example 5 On the basis of embodiment 4, the training optimization module includes three stages of optimization. The first stage carries out Foundational Visual Alignment. Using large-scale binary classification data (without RAG and CoT), the MLLM is fine-tuned for the visual question answering (VQA) task, so that it has basic true and false visual perception ability. The second stage carries out Critical Reasoning Supervised Fine-Tuning (SFT). Using the constructed F-CoT dataset, the model is fine-tuned for instructions. The model is trained to mimic the standard forensic reasoning mode, that is, to observe independently first, analyze external evidence, and finally cross-verify and make decisions. The third stage carries out Autonomous Reasoning Reinforcement Learning (RL-GRPO). On unseen training data, the Group Relative Policy Optimization (GRPO) algorithm is used for reinforcement learning. GRPO optimizes the strategy by sampling multiple different outputs (Group) for the same input and calculating the advantage (Advantage) based on the reward function, thus saving the video memory without training an additional value network (Value Network). For each input, the strategy model samples G different reasoning paths and answers. In order to make the model truly learn critical reasoning, the model is encouraged to think critically when faced with conflicting evidence, and a "Process-Aware Reward Function" is designed, including: With Inconsistent, specifically: Wherein, is the judgment tendency of the stage-1 model (or preliminary visual analysis), is the label recommended by the strongest evidence retrieved by the stage-2 model.

[0040] According to the correctness of the final answer of the model and whether the model correctly handles the conflict between "visual intuition" and "external evidence", a differentiated reward signal is given to force the model to learn to accept correct evidence and reject misleading evidence. Specifically, a conflict reward matrix is set: when the model can still make correct judgments in the presence of conflicting evidence, the highest reward (+2.0) is given; when the model is misled by false evidence and makes a wrong judgment, the highest punishment (-2.0) is given; to strengthen the critical thinking of the model.

[0041] In particular, the stage-2 model is trained by reinforcement learning with the training data of stage-3 to solve the problem of "rote memorization" caused by SFT, so that the model learns to dynamically weigh the weights of vision and evidence, gives the model a real dynamic decision-making ability, and obtains a stage-3 model, that is, a critical reasoning multi-modal large model in step 3. The critical reasoning multi-modal large model in step 3 receives the query image, the prompt word, and the top-k text evidence retrieved from the FKD by the dynamic evidence retriever, and outputs the detection result containing the reasoning process.

[0042] In particular, the final reward in the critical reasoning multi-modal large model , wherein, is a conflict reward, including critical victory, regular correct, regular error and critical failure; is a format check, responsible for checking whether the output follows the XML tag format, is a conflict reward coefficient, is the coefficient of the format check .

[0043] The conflict reward matrix is shown in the following table: The learning rate is set to , and the batch size is set to 32. The stage-3 model obtained through this stage no longer simply imitates the language style in the SFT data, but has truly learned to evaluate the credibility of the evidence.

[0044] Embodiment 6 The embodiments of the application describe the software and hardware environment configuration, core component selection and unique three-stage training parameters for realizing autonomous critical thinking of the VRAG-DFD framework.

[0045] The core technology stack and base model are specifically: 1. Base MLLM: Qwen2.5-VL is used as the core of the critical reasoning module.

[0046] 2. Retina architecture: the dynamic evidence retriever is based on the clip-vit-large-patch14 architecture and is fine-tuned by LoRA.

[0047] 3. Efficient training: in order to reduce the consumption of computing resources while maintaining the performance of the model, all training stages are fine-tuned by low-rank adaptation (LoRA) technology, The three-stage training configuration example is as follows: The system training strictly follows the following three-stage progressive pipeline, and the training data all come from the FF++ dataset: The specific model names mentioned in the embodiments of the present application, such as "Gemini 2.5 Pro", "Qwen2.5-VL", etc., are only preferred examples in the embodiments and should not be understood as a limitation on the protection scope of the present application. Those skilled in the art can replace them with other multi-modal large models with corresponding capabilities according to actual conditions, which all belong to the protection scope of the present application.

[0048] The preferred embodiments of the present application are described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the existing technology according to the concept of the present application shall be within the protection scope determined by the claims.

Claims

1. A deepfake detection method for multimodal large models based on retrieval enhancement, characterized in that, The method includes the following steps: Step 1: Perform two-stage annotation. The first stage of annotation constructs an evidence knowledge database, which stores pairs of real and fake images and their corresponding structured text annotations. The text annotations include the location of forgery regions and fine-grained artifact descriptions. The second stage of annotation constructs an evidence thinking chain dataset based on the evidence knowledge database. The evidence thinking chain dataset is generated based on the validity of the initial judgment and retrieved evidence, including cross-validation type, evidence-guided correction type, and adversarial sample type thinking chain data. Step 2: For the image to be detected, use a dynamic forensics retrieval tool to retrieve textual evidence of the top-k similar cases from the forensics knowledge database, wherein the dynamic forensics retrieval tool is based on a deepfake feature extraction network trained on Deepfake to calculate feature similarity. Step 3: Input the image to be detected and the retrieved text evidence into the critical reasoning multimodal big model, perform multimodal reasoning analysis, and output the detection results containing the reasoning process.

2. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 1, characterized in that, In the process of constructing the evidence knowledge database in step 1, a multimodal large model is used as an annotation assistant, and a large-scale face forgery dataset is compared and annotated based on a predefined forgery guide. In the annotation process, real and fake images are input simultaneously to enhance the accuracy of forgery trace location.

3. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 2, characterized in that, The forgery guide covers several mainstream forgery techniques, including but not limited to DeepFakes, Face2Face, FaceSwap, and Neural Textures.

4. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 2, characterized in that, Step 1, which involves constructing the forensic knowledge database, also includes: Real and fake samples are extracted from the face forgery dataset and stored in a balanced ratio of 1:

1. A vector database is used to manage the correspondence between image features and text annotations. The fine-grained artifact description includes descriptions of texture loss, blurred boundaries, and abnormal lighting.

5. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 1, characterized in that, The construction of the evidence collection thought chain dataset in step 1 includes: Multiple images are sampled, and textual evidence of the top-k similar cases (excluding the image itself) for each sampled image is retrieved from the forensic knowledge database. For each sampled image, based on its initial judgment and the validity of the retrieved textual evidence, the samples are classified into three types, including the cross-validation type, the evidence-guided correction type, and the adversarial sample type. For each type, a structured thought chain annotation is generated, which includes a complete reasoning path including preliminary visual analysis, RAG reference information analysis, integrated reasoning and decision-making, and the final answer.

6. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 2, characterized in that, The face spoofing dataset is the FF++ dataset. Before starting step 1, the FF++ dataset is divided using a multimodal large model. Several videos are randomly selected from the Train subset as training data for stage-1, and the remaining videos are used as training data for stage-2 and stage-3. The stage-1 model is obtained by training the stage-1 stage. The stage-1 model has basic visual perception capabilities for spoofing tasks.

7. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 6, characterized in that, The stage-1 model is trained using the aforementioned evidence-gathering thought chain dataset to teach the model to mimic the thought process of an expert, resulting in the stage-2 model, where the retrieval device is the dynamic evidence-gathering retrieval device to be used in step 2.

8. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 7, characterized in that, During the LoRa training, the rank r was set to 128, the scaling factor alpha to 256, and the learning rate to... .

9. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 7, characterized in that, The stage-2 model is trained using the stage-3 training data to perform reinforcement learning, giving the model true dynamic decision-making capabilities, resulting in the stage-3 model, which is the critical reasoning multimodal large model mentioned in step 3. A group-relative strategy is used for optimization to encourage the model to think critically when faced with conflicting evidence. A process-aware reward function is also set, including: defining A as the correctness of the final answer, defining C as the existence of cognitive conflict, and defining conflict as: in, The judgment tendency of the stage-1 model. The label suggested for the strongest evidence retrieved by the stage-2 model.

10. The deepfake detection method for multimodal large models based on retrieval enhancement as described in claim 9, characterized in that, The final reward in the critical reasoning multimodal big model ,in, Rewards for conflict include critical victory, conventional correctness, conventional error, and critical failure; For format checking, it is responsible for checking whether the output conforms to the XML tag format. Conflict Rewards coefficient, For format checking The coefficient.

Images